Data processing method and device, image display method and device, equipment and medium

Abstract

The invention discloses a data processing method and device, an image display method and device, equipment and a medium. The method comprises the steps that first panoramic image information with the resolution ratio being a first resolution ratio and second panoramic image information with the resolution ratio being a second resolution ratio are acquired, and the first resolution ratio is smaller than the second resolution ratio; acquiring first attitude information sent by the first head-mounted display device; determining first visual image information corresponding to the first head-mounted display equipment according to the first attitude information and the second panoramic image information; determining first target image information according to the first visual image information and the first panoramic image information; and sending the first target image information to the first head-mounted display device. According to the technical scheme provided by the embodiments of the invention, the requirements on the bandwidth and the computing power can be reduced while the picture definition and the refresh rate are ensured in the VR live broadcast process, and the effects of saving the cost and improving the user experience are achieved.

Description

Technical Field

[0001] This application belongs to the technical field of VR live broadcast, and particularly relates to a data processing method, an image display method, a device, a device and a medium. Background Technique

[0002] VR panoramic live broadcast is a brand-new live broadcast method, which can bring a subversive sensory experience to users. It is of great significance for teaching, training, scientific research, and product display, and is a live broadcast method with very promising development prospects. VR panoramic live broadcast can push a 3D panoramic video to dozens or even hundreds of VR devices. Each device receiving the push has an immersive 3D space, and users can freely adjust the viewing angle by turning their heads. Before the video stream is pushed to the VR device, it must be decoded as fully as possible. Since the data volume of the panoramic video is extremely large, the requirements for the bandwidth and computing power of the streaming device are extremely high, and the required cost is relatively large. In order to support a larger number of VR devices under the premise of controlling costs, only the picture clarity of the panoramic video within the visible range can be reduced or the picture delay can be increased. However, no matter which method is used, it will greatly affect the user experience, and the user experience is poor. Summary of the Invention

[0003] The embodiments of this application provide an implementation solution different from the related art to solve the technical problem of poor user experience caused by controlling costs by reducing the picture clarity of the panoramic video within the visible range or increasing the picture delay during VR live broadcast.

[0004] In a first aspect, this application provides a data processing method, including:

[0005] Obtain first panoramic image information with a first resolution and second panoramic image information with a second resolution, where the first resolution is less than the second resolution;

[0006] Obtain first attitude information sent by a first head-mounted display device;

[0007] Determine first visible image information corresponding to the first head-mounted display device according to the first attitude information and the second panoramic image information;

[0008] Determine first target image information according to the first visible image information and the first panoramic image information;

[0009] Send the first target image information to the first head-mounted display device.

[0010] In a second aspect, this application provides an image display method for a head-mounted display device, including:

[0011] Obtain the first pose information of the head-mounted display device;

[0012] Obtain the first panoramic image information with a first resolution and the second panoramic image information with a second resolution, where the first resolution is less than the second resolution;

[0013] Determine the first visible image information corresponding to the head-mounted display device according to the first pose information and the second panoramic image information;

[0014] Determine the first target image information according to the first visible image information and the first panoramic image information;

[0015] Display the first target image information.

[0016] In a third aspect, the present application provides a head-mounted display device, including:

[0017] An acquisition unit, configured to obtain the first pose information of the head-mounted display device;

[0018] The acquisition unit is further configured to obtain the first panoramic image information with a first resolution and the second panoramic image information with a second resolution, where the first resolution is less than the second resolution;

[0019] A determination unit, configured to determine the first visible image information corresponding to the head-mounted display device according to the first pose information and the second panoramic image information;

[0020] The determination unit is further configured to determine the first target image information according to the first visible image information and the first panoramic image information;

[0021] A display unit, configured to display the first target image information.

[0022] In a fourth aspect, the present application provides an electronic device, including:

[0023] A processor; and

[0024] A memory, configured to store executable instructions of the processor;

[0025] Wherein, the processor is configured to execute any method in the first aspect, the second aspect, each possible implementation manner of the first aspect, or each possible implementation manner of the second aspect by executing the executable instructions.

[0026] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements any method in the first aspect, the second aspect, each possible implementation manner of the first aspect, or each possible implementation manner of the second aspect.

[0027] The present application provides a method for obtaining first panoramic image information with a first resolution and second panoramic image information with a second resolution, where the first resolution is less than the second resolution; obtaining first pose information sent by a first head-mounted display device; determining first visible image information corresponding to the first head-mounted display device according to the first pose information and the second panoramic image information; determining first target image information according to the first visible image information and the first panoramic image information; and sending the first target image information to the first head-mounted display device. This solution can enable the determination of the final user-visible high-definition image by jointly processing the high-quality image with a higher resolution and the image with a lower resolution. Moreover, the image information sent to the first head-mounted display device also depends on the image information with a lower resolution, saving the data transmission volume, that is, saving the bandwidth cost, and achieving the purpose of ensuring both a lower cost and a higher user experience simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following briefly introduces the drawings required for use in the description of the embodiments or the prior art. Obviously, the following-described drawings are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts. In the drawings:

[0029] Figure 1 is a schematic structural diagram of a VR live broadcast system provided by an embodiment of the present application;

[0030] Figure 2 is a schematic flowchart of a data processing method provided by an embodiment of the present application;

[0031] Figure 3 is a flowchart of a method for obtaining the first coding result and the second coding result by using the same original panoramic image information with different coding methods by a transmitting device according to an exemplary embodiment of the present application;

[0032] Figure 4 is a schematic diagram of a field of view angle region provided by an exemplary embodiment of the present application;

[0033] Figure 5 is a schematic diagram of the generation process of first target image information provided by an exemplary embodiment of the present application;

[0034] Figure 6 is a schematic flowchart of an image display method provided by an exemplary embodiment of the present application;

[0035] Figure 7Schematic diagram of the structure of a head-mounted display device provided by an exemplary embodiment of the present application;

[0036] Figure 8 It is a schematic block diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0037] The embodiments of the present application will be described in detail below. The examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, and should not be construed as a limitation to the present application.

[0038] The terms "first" and "second" etc. in the description, claims and drawings of the embodiments of the present application are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the present solution can be implemented in an order other than the order illustrated or described in the present application. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0039] The solution of the present application can be applied to scenarios such as games and teaching.

[0040] First, some terms in the embodiments of the present application will be explained below to facilitate the understanding of those skilled in the art.

[0041] MTP: Motion-to-Photon Latency, the total time it takes for the user from the start of head movement to the rendered image using the motion parameters to be visible on the screen.

[0042] FOV: Field of View, the angle between the edge of the display device and the line connecting the observation point (glasses). Simply put, it is the content that the user can clearly see and the content that can be glimpsed from the corner of the eye.

[0043] YUV: It is a color encoding method. Y represents luminance, that is, the gray value. U and V are chrominance and concentration, which are used to describe the color and saturation of the image and are used to specify the color of the pixel.

[0044] Among the current VR applications, panoramic live broadcast of VR videos is one of the most promising projects with the best user experience. It can push a 3D panoramic video to dozens or even hundreds of VR devices. Each device receiving the push has an immersive 3D space, and users can adjust the viewing angle at will by turning their heads. This technology is of great significance for teaching, training, scientific research, and product display. However, this technology also faces many challenges. In order to create a more "realistic" effect, ultra-low head movement delay (MTP) must be guaranteed. This requires providing high-definition and high-refresh rate video streams at the technical level. From the hardware level, this means ultra-high bandwidth and computing power.

[0045] For VR devices, as the performance of wearable chips increases year by year, VR devices can bear more and more computing power. More and more computing logic is transferred from the PC side to the VR device side, which effectively improves the real-time performance of VR device data processing. However, for high-definition, high-refresh rate panoramic videos, the computing power provided by VR devices is still difficult to meet the requirements. This requires solving the problem from the streaming end. Before the video stream is pushed to the VR device, it must be fully decoded as much as possible. In addition, VR devices have a unique FOV (field of view) concept. For a 4K panoramic video, the part that VR glasses can see at a certain moment is very limited, and the resolution of this part can only reach 540P, so even with 8K resolution, it is only "enough" for panoramic videos. If the delay is further reduced to meet the user experience, such as providing a refresh rate of more than 90hz, the bandwidth and computing power consumed will inevitably increase further.

[0046] Some technical solutions currently used in the industry simply support more VR devices by reducing the picture clarity or increasing the delay. Either method will greatly affect the user experience. For example, reducing the picture clarity will cause serious distortion of the picture, and increasing the delay will increase the visual fatigue of users and even cause dizziness. However, if high-definition and high-refresh rate video streams are pushed to ensure user experience, the overall cost of the solution will inevitably increase sharply and even become unacceptable.

[0047] Therefore, it is necessary to find a solution suitable for panoramic VR live broadcast, which can reduce the requirements for computing power and bandwidth as much as possible while ensuring clarity and refresh rate, so as to achieve the purpose of saving costs and improving user experience.

[0048] Therefore, to solve the above at least one technical problem, the present application provides a data processing method, an image display method, a device, a device and a medium, which are used to solve the technical problem of poor user experience caused by controlling costs by reducing the picture clarity of the panoramic video within the user's visible range or increasing the picture delay during VR live broadcast.

[0049] The technical solution of the present application and how the technical solution of the present application solves the above technical problems will be described in detail below with specific embodiments. These specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0050] Figure 1 FIG. 1 is a schematic structural diagram of a VR live broadcast system provided by an exemplary embodiment of the present application. The structure includes: a sending end device 10, a server 20, a head-mounted display device 30, etc.

[0051] In some embodiments, an enterprise application suite is deployed in the sending end device 10, that is, the host end device. The application enterprise suite can be used to obtain the original panoramic image information. The original panoramic image information can come from a panoramic video synthesized by a 3D application or a 360° panoramic video captured by a 3D camera. The 3D application can be a game application or a teaching application, etc.

[0052] In some embodiments, the sending end device 10 is specifically configured to:

[0053] Preprocess the original panoramic image information through a first processor to obtain the panoramic image information to be processed in the YUV format;

[0054] Determine the first initial panoramic image information with a resolution of the first resolution according to the panoramic image information to be processed;

[0055] Encode the first initial panoramic image information through a second processor to obtain the first encoding result;

[0056] Slice the panoramic image information to be processed according to a preset slicing rule to obtain a second initial panoramic image information including a plurality of first sub-image information;

[0057] Encode the second initial panoramic image information through the second processor and the plurality of first sub-image information to obtain the second encoding result;

[0058] Send the first encoding result and the second encoding result to the server 20.

[0059] In some embodiments, the server 20 may include an RTC (Real-time Communications) module, and the server 20 may be configured to:

[0060] Obtain a first encoding result and a second encoding result from the sending device 10;

[0061] Decode the first encoding result and the second encoding result to obtain the first panoramic image information and the second panoramic image information respectively;

[0062] Obtain the first panoramic image information with a first resolution and the second panoramic image information with a second resolution, where the first resolution is less than the second resolution;

[0063] Obtain first pose information sent by the first head-mounted display device;

[0064] Determine first visible image information corresponding to the first head-mounted display device according to the first pose information and the second panoramic image information;

[0065] Determine first target image information according to the first visible image information and the first panoramic image information;

[0066] Send the first target image information to the first head-mounted display device for the first head-mounted display device to display the first target image information.

[0067] The sending device 10 may be a terminal, which may be a device such as a tablet computer or a notebook computer.

[0068] The server 20 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms.

[0069] For the execution principles and interaction processes of the constituent units in the embodiments of this system, reference may be made to the descriptions of the following method embodiments. It should be noted that the description order of the following embodiments does not limit the priority order of the embodiments.

[0070] Figure 2 It is a schematic flowchart of a data processing method provided by an exemplary embodiment of this application. This method is applicable to the server 20 and at least includes the following S11-S15:

[0071] S11. Obtain first panoramic image information with a first resolution and second panoramic image information with a second resolution, where the first resolution is less than the second resolution;

[0072] In some embodiments, the first resolution may be 1080P, and the corresponding image may be referred to as a low-definition image. The second resolution may be 8K, and the corresponding image may be referred to as a high-definition image. The magnitudes of the first resolution and the second resolution may be reasonably set by relevant personnel, and this application does not make any limitations thereto.

[0073] In some embodiments, the first panoramic image information and the second panoramic image information are one panoramic frame from the same panoramic video, and the panoramic video supports multiple video sources.

[0074] In the environment of enterprise applications, the video sources for VR panoramic live broadcasts are diverse. First, for some panoramic content, the VR live broadcast SDK can be used. The solution has a simple deployment, high flexibility, and supports real-time switching of multiple image qualities. This solution uses the method of texture sharing, which can share the panoramic texture in the game with the enterprise application suite, and then the enterprise application suite pushes it to the RTC server for live broadcast. Second, if the customer's application development is based on the streaming solution of StreamVR, the stereo images of the stream can be extracted through the SDK of StreamVR, and after being spliced, scaled, and de-distorted by the enterprise application suite, it is finally pushed to the RTC server for live broadcast. The advantage of this solution is strong versatility. Since StreamVR is the world's largest VR content distribution platform, a large number of VR content creators create on it. Third, for VR live broadcast content that does not require interaction, such as 360° panoramic videos or pictures, the enterprise application suite will parse the file content and perform panoramic texturing in the same way as general streaming media, and then hand it over to the RTC server for live broadcast.

[0075] In some embodiments, the above video sources and video processing methods are shown in Table 1:

[0076] Table 1 Corresponding relationship table of video sources and video processing methods

[0077] Video source Processing method Enterprise-customized panoramic content Push stream using VR SDK Content created based on StreamVR Extract streaming images using StreamVR SDK 360° panoramic video Open and map textures in the general streaming media way

[0078] In some embodiments, before performing the above S11, the method further includes S110 - S111:

[0079] S110. Obtain a first encoding result and a second encoding result from a sending device;

[0080] Among them, the first coding result and the second coding result are obtained by the sending-end device encoding the same original panoramic image information through different coding methods.

[0081] Figure 3 The flowchart for a sending-end device provided by an exemplary embodiment of the present application to obtain the first coding result and the second coding result by encoding the same original panoramic image information through different coding methods is shown in Figure 3 , and at least includes the following S01 - S05:

[0082] S01. Preprocess the original panoramic image information through a first processor to obtain the panoramic image information to be processed in the YUV format;

[0083] In some embodiments, the first processor may include a consumer-grade graphics card or an integrated graphics card, and the original panoramic image information may be a panoramic picture in the RGB texture format.

[0084] In some embodiments, the resolution of the original panoramic image information and the panoramic image information to be processed may be the same, specifically 8K.

[0085] S02. Determine the first initial panoramic image information with the first resolution according to the panoramic image information to be processed;

[0086] Specifically, the panoramic image information to be processed can be downsampled to obtain the first initial panoramic image information.

[0087] In some embodiments, the first resolution is lower than the resolution of the panoramic image information to be processed. For example, the first resolution can be 1080P.

[0088] S03. Encode the first initial panoramic image information through a second processor to obtain the first coding result;

[0089] In some embodiments, the data processing capability of the second processor is higher than that of the first processor.

[0090] In some embodiments, the second processor may include a professional-grade graphics card.

[0091] S04. Slice the panoramic image information to be processed according to a preset slicing rule to obtain a second initial panoramic image information including a plurality of first sub-image information;

[0092] In some embodiments, the preset slicing rule may be to evenly partition the panoramic image information to be processed to obtain several partitions with equal areas. Similar to the square "tiles" in architecture, these partitions can be named tiles.

[0093] The purpose of partitioning the regions is to enable independent encoding and decoding in the subsequent process. Therefore, attention should be paid to the partitioning intensity. If the partitions are too large, the encoding and data transmission control will not be precise enough, and the optimization effect will not be obvious. If the partitions are too small, there will be too many partitions for parallel processing, leading to frequent switching of the encoder, which will also reduce the computing efficiency. Generally, it is appropriate that the number of partitions covered within the FOV region of the user in the panoramic image is 12 - 15.

[0094] For example, the panoramic image information to be processed can be divided into 10 * 5 partitions, and the image information in the 10 * 5 partitions is the 10 * 5 pieces of the first sub-image information. The number of partitions in the preset slicing rule can be reasonably set by relevant personnel, and the application does not limit this.

[0095] S05. Encode the second initial panoramic image information through the second processor and the multiple pieces of the first sub-image information to obtain the second encoding result.

[0096] Among them, the second initial panoramic image information is the multiple pieces of the first sub-image information obtained after slicing the panoramic image information to be processed.

[0097] Specifically, based on the second processor, the sending-end device parallelly encodes the multiple pieces of the first sub-image information included in the second initial panoramic image information through multiple encoders to obtain the second encoding result, where one encoder encodes one piece of the first sub-image information.

[0098] Among them, the second processor has the multi-instance encoding ability and includes multiple encoders, and the second encoding result includes the multiple encoding results corresponding to the multiple pieces of the first sub-image information.

[0099] After obtaining the first encoding result and the second encoding result, the sending-end device sends the first encoding result and the second encoding result to the server.

[0100] In some embodiments, when the sending-end device uses the same original panoramic image information to obtain the first encoding result and the second encoding result through different encoding methods, it can also parallelly encode multiple panoramic image information with multiple third processors having the same performance as the second processor. After the multiple original panoramic image information parallelly encoded by the multiple third processors are recombined according to the frame numbers, they are sent to the server.

[0101] S111. Decode the first encoding result and the second encoding result to obtain the first panoramic image information and the second panoramic image information respectively.

[0102] In some embodiments, the first panoramic image information is a low-definition panoramic picture, and the second panoramic image information is a plurality of second sub-image information obtained by encoding and then decoding the plurality of first sub-image information.

[0103] S12. Obtain first pose information sent by a first head-mounted display device;

[0104] Optionally, the first pose information sent by the first head-mounted display device is the first pose information of the first head-mounted display device itself.

[0105] Figure 4 FIG. is a schematic diagram of a field of view angle region provided by an exemplary embodiment of the present application. Specifically, after the to-be-processed panoramic image information is sliced according to the preset slicing rule, a vector representing the positive direction can be defined by relevant personnel, which can be called the 0 vector, and the 0 vector can be at any angle. Among them, for the 0 vector, the center point of the connection line of the user's two eyeballs can be selected as the starting point, and the center of the second panoramic image information can be selected as the end point. The vector with the center point of the connection line of the user's two eyeballs as the starting point and the center of the second panoramic image information as the end point is used as the 0 vector, and the 0 vector remains fixed after being determined.

[0106] Furthermore, the center of the picture within the user's field of view, that is, the center of the visible image range of the user, is used as the end point. The vector with the center point of the connection line of the user's two eyeballs as the starting point and the center of the picture within the user's visible image range as the end point is used as the FOV vector. After the live broadcast starts, as the user continuously turns the head, the offset vector between the 0 vector and the FOV vector continuously changes. In some embodiments, the head-mounted display device continuously records the user's current FOV vector at different times, and uses the offset vector between the user's current 0 vector and the FOV vector as the first pose information and sends it to the server. Among them, the offset vector can refer to the difference between the FOV vector and the 0 vector.

[0107] In some embodiments, the first pose information can also be the offset coordinate information of the center of the picture within the user's visible image range relative to the center of the second panoramic image information.

[0108] S13. Determine first visible image information corresponding to the first head-mounted display device according to the first pose information and the second panoramic image information;

[0109] In some embodiments, determining the first visible image information corresponding to the first head-mounted display device according to the first pose information and the second panoramic image information includes the following S131-S132:

[0110] S131. Determine the first region information within the field of view corresponding to the first pose information from the second panoramic image information;

[0111] In some embodiments, the first region information is region information composed of multiple coordinate position information determined in the second panoramic image information according to the first pose information. There is a specific correspondence between the first pose information and the first region information. Refer to Figure 4 For example, in Figure 4 , the first pose information is the offset vector between the vector corresponding to the solid dot and the vector corresponding to the dashed dot, and the first region information is the region information composed of multiple black squares in the figure.

[0112] S132. Use the image information within the first region information in the second panoramic image information as the first visual image information corresponding to the first head-mounted display device.

[0113] In some embodiments, determine multiple third sub-image information corresponding to the first region information from the second panoramic image information, and use the multiple third sub-image information as the first visual image information. For example, in Figure 4 , six black squares, that is, six third sub-image information, are used as the first visual image information. Among them, the multiple third sub-image information is included in the aforementioned multiple second sub-image information.

[0114] In some embodiments, the multiple third sub-image information corresponding to the first region information is part of the second sub-image information within the region corresponding to the first region information among the aforementioned multiple second sub-image information.

[0115] S14. Determine the first target image information according to the first visual image information and the first panoramic image information;

[0116] In some embodiments, the determination of the first target image information according to the first visual image information and the first panoramic image information includes at least the following S141 - S143:

[0117] S141. Adjust the resolution of the first panoramic image information from the first resolution to the second resolution to obtain the third panoramic image information;

[0118] S142. Obtain the second region information corresponding to the first region information in the third panoramic image information;

[0119] In some embodiments, in the third panoramic image information, the coordinate position information included in the second region information is the same as the coordinate position information included in the first region information in the second panoramic image information.

[0120] In some embodiments, in the third panoramic image information, the coordinate position information included in the second region information corresponds to the coordinate position information included in the first region information in the second panoramic image information.

[0121] S143. Replace the original image information within the second region information based on the first visual image information to obtain first target image information.

[0122] In some embodiments, replacing the original image information within the second region information based on the first visual image information to obtain first target image information includes: using the first visual image information to replace the original image information within the second region information to obtain first target image information. Figure 5 FIG. is a schematic diagram of a generation process of first target image information provided by an exemplary embodiment of the present application.

[0123] S15. Send the first target image information to the first head-mounted display device.

[0124] In some embodiments, before sending the first target image information to the first head-mounted display device, the first target image information may be converted from a rectangular screen to a spherical screen using a preset algorithm, and the preset algorithm can be set by relevant personnel, and the present application does not limit this.

[0125] In some embodiments, after sending the first target image information to the first head-mounted display device, the first head-mounted display device may use a preset algorithm to convert the first target image information from a rectangular screen to a spherical screen, and display corresponding content to the user based on the converted spherical screen.

[0126] In some embodiments, the aforementioned server may be connected to multiple head-mounted display devices for determining the corresponding target image information of each head-mounted display device. Based on this, the above method further includes:

[0127] Obtain second pose information sent by a second head-mounted display device; optionally, the second pose information sent by the second head-mounted display device is the second pose information of the second head-mounted display device itself.

[0128] Determine second visual image information corresponding to the second head-mounted display device according to the second pose information and the second panoramic image information;

[0129] Determine second target image information according to the second visual image information and the second panoramic image information;

[0130] Send the second target image information to the second head-mounted display device.

[0131] Among them, the second head-mounted display device is a VR head-mounted display device that is jointly connected to the server with the first head-mounted display device.

[0132] Specifically, the manner in which the second head-mounted display device determines the second pose information is the same as the manner in which the first head-mounted display device determines the first pose information.

[0133] For the manner of determining the second visible image information corresponding to the second head-mounted display device according to the second pose information and the second panoramic image information, reference may be made to the manner of determining the first visible image information corresponding to the first head-mounted display device according to the first pose information and the first panoramic image information described above.

[0134] For determining the second target image information according to the second visible image information and the second panoramic image information, reference may be made to the manner of determining the first target image information according to the first visible image information and the first panoramic image information.

[0135] Compared with the related technologies, the solution of this application saves 50% of the downlink bandwidth and reduces the requirements for the GPU decoding ability of VR panoramic live broadcast on the premise of achieving the same clarity. In the VR scenario, videos of content such as cloud games, large-scale exhibitions, and events have high requirements for resolution. General desktop GPUs are difficult to meet the decoding requirements of 8K+120fps level. This application reduces the requirements for GPU decoding of panoramic live broadcast by dividing the decoding area and only decoding the FOV-related partitions. This application greatly reduces the requirements for bandwidth of VR panoramic live broadcast by transmitting low-resolution panoramas and high-definition shard images. This also means that under the same network conditions, the live broadcast solution can support twice as many VR devices online. During the actual experience process, based on the distribution solution of high-definition shard images, even if a large amount of bandwidth is saved, the expressiveness of the picture details is still better than that of the traditional live broadcast solution. When the network bandwidth is sufficient, the conversion of the picture from low definition to high definition is hardly noticeable to users, improving the user experience.

[0136] The solution provided by this application also solves the problem of bandwidth waste existing in the process of traditional VR panoramic live broadcast. The solution provided by this application also solves the problem of dizziness caused by high latency in the process of VR live broadcast. While pushing high-definition sharded images (i.e., the first visible image information), this application also pushes a low-definition panoramic image (the first panoramic image information) as the background. When the rotation speed of the VR headset is slow, what the user can see are all high-definition images, and the user will not feel the change in image clarity. When the headset rotates at a high speed, the user may be out of the high-definition area for a short time. However, due to the existence of the low-definition background, there will be no problem of image loss within the field of view. At most, the user will see some blurred images for a short time, ensuring that the user can still see an effective picture during the process of quickly rotating the headset. As the high-definition sharded images newly requested within the FOV area are gradually updated, all the images within the user's field of view will become high-definition again. This solves the problem of user dizziness caused by image loss in the related art.

[0137] Figure 6 FIG. 4 is a schematic flowchart of an image display method provided for an exemplary embodiment of this application. This image display method can be used in a head-mounted display device and at least includes the following S61 - S64:

[0138] S61. Obtain the first pose information of the head-mounted display device;

[0139] S62. Send the first pose information to the server, so that the server obtains the first panoramic image information with a first resolution and the second panoramic image information with a second resolution, where the first resolution is less than the second resolution; determine the first visible image information corresponding to the head-mounted display device according to the first pose information and the second panoramic image information; determine the first target image information according to the first visible image information and the first panoramic image information;

[0140] S63. Receive the first target image information;

[0141] S64. Display the first target image information.

[0142] An exemplary embodiment of this application also provides an image display method for a head-mounted display device, including:

[0143] Obtain the first pose information of the head-mounted display device;

[0144] Obtain the first panoramic image information with a first resolution and the second panoramic image information with a second resolution, where the first resolution is less than the second resolution;

[0145] Determine the first visible image information corresponding to the head-mounted display device according to the first pose information and the second panoramic image information;

[0146] Determine the first target image information according to the first visible image information and the first panoramic image information;

[0147] Display the first target image information.

[0148] For the specific implementation manners in this embodiment, reference may be made to the foregoing content, which will not be elaborated herein.

[0149] This application further provides a head-mounted display device, including:

[0150] An acquisition unit, configured to acquire the first pose information of the head-mounted display device;

[0151] The acquisition unit is further configured to acquire the first panoramic image information with a first resolution and the second panoramic image information with a second resolution, where the first resolution is less than the second resolution;

[0152] A determination unit, configured to determine the first visible image information corresponding to the head-mounted display device according to the first pose information and the second panoramic image information;

[0153] The determination unit is further configured to determine the first target image information according to the first visible image information and the first panoramic image information;

[0154] A display unit, configured to display the first target image information.

[0155] For the specific implementation manners in this embodiment, reference may be made to the foregoing content, which will not be elaborated herein.

[0156] An exemplary embodiment of this application further provides a data processing device, and the data processing device includes:

[0157] An acquisition unit, configured to acquire the first panoramic image information with a first resolution and the second panoramic image information with a second resolution, where the first resolution is less than the second resolution; the acquisition unit is further configured to acquire the first pose information sent by the first head-mounted display device;

[0158] A determination unit, configured to determine the first visible image information corresponding to the first head-mounted display device according to the first pose information and the second panoramic image information; the determination unit is further configured to determine the first target image information according to the first visible image information and the first panoramic image information;

[0159] A sending unit, configured to send the first target image information to the first head-mounted display device.

[0160] In some embodiments, when the data processing device is used to determine the first visible image information corresponding to the first head-mounted display device according to the first pose information and the second panoramic image information, it is specifically used for:

[0161] Determine first region information within the field of view corresponding to the first pose information from the second panoramic image information according to the first pose information;

[0162] Use the image information within the first region information in the second panoramic image information as the first visible image information corresponding to the first head-mounted display device.

[0163] In some embodiments, when the data processing device is used to determine first target image information according to the first visible image information and the first panoramic image information, it is specifically used for:

[0164] Adjust the resolution of the first panoramic image information from the first resolution to the second resolution to obtain third panoramic image information;

[0165] Obtain second region information corresponding to the first region information in the third panoramic image information;

[0166] Replace the original image information within the second region information based on the first visible image information to obtain first target image information.

[0167] In some embodiments, the data processing device is further used for:

[0168] Obtain a first encoding result and a second encoding result from a sending device;

[0169] Decode the first encoding result and the second encoding result to respectively obtain the first panoramic image information and the second panoramic image information.

[0170] In some embodiments, the data processing device is further used for:

[0171] Obtain second pose information sent by a second head-mounted display device;

[0172] Determine second visible image information corresponding to the second head-mounted display device according to the second pose information and the second panoramic image information;

[0173] Determine second target image information according to the second visible image information and the second panoramic image information;

[0174] Send the second target image information to the second head-mounted display device.

[0175] For the specific content corresponding to this embodiment, reference may be made to the foregoing content, which will not be elaborated here.

[0176] Figure 7 FIG. is a schematic structural diagram of a head-mounted display device provided for an exemplary embodiment of the present application. The head-mounted display device includes:

[0177] An acquisition unit 71, configured to acquire first attitude information of the head-mounted display device;

[0178] A sending unit 72, configured to send the first attitude information to a server, so that the server acquires first panoramic image information with a first resolution and second panoramic image information with a second resolution, where the first resolution is less than the second resolution; determine first visible image information corresponding to the head-mounted display device according to the first attitude information and the second panoramic image information; and determine first target image information according to the first visible image information and the first panoramic image information;

[0179] A receiving unit 73, configured to receive the first target image information;

[0180] A display unit 74, configured to display the first target image information.

[0181] It should be understood that the device embodiments and the method embodiments can correspond to each other, and similar descriptions can refer to the method embodiments. To avoid repetition, it will not be elaborated here. Specifically, the device can execute the above method embodiments, and the foregoing and other operations and / or functions of each module in the device respectively correspond to the corresponding processes in each method in the above method embodiments. For the sake of brevity, it will not be elaborated here.

[0182] In the foregoing, the device of the embodiment of the present application has been described from the perspective of functional modules in combination with the accompanying drawings. It should be understood that the functional modules can be implemented in the form of hardware, or can be implemented by instructions in software form, or can be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiments in the present application can be completed by the integrated logic circuit in the hardware in the processor and / or instructions in software form. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by the hardware decoding processor, or can be executed and completed by a combination of the hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps in the above method embodiments.

[0183] Figure 8 FIG. is a schematic block diagram of an electronic device provided for an embodiment of the present application. The electronic device may include:

[0184] A memory 801 and a processor 802, where the memory 801 is used to store a computer program and transmit the program code to the processor 802. In other words, the processor 802 can call and run the computer program from the memory 801 to implement the method in the embodiments of the present application.

[0185] For example, the processor 802 can be used to execute the above method embodiments according to the instructions in the computer program.

[0186] In some embodiments of the present application, the processor 802 may include, but is not limited to:

[0187] A general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and so on.

[0188] In some embodiments of the present application, the memory 801 includes, but is not limited to:

[0189] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be Read-Only Memory (ROM), Programmable ROM (PROM), Erasable PROM (EPROM), Electrically Erasable PROM (EEPROM), or flash memory. The volatile memory can be Random Access Memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synch Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0190] In some embodiments of the present application, the computer program can be divided into one or more modules, and the one or more modules are stored in the memory 801 and executed by the processor 802 to complete the method provided by the present application. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0191] As Figure 8 shown, the electronic device may further include:

[0192] A transceiver 803, which can be connected to the processor 802 or the memory 801.

[0193] Among them, the processor 802 can control the transceiver 803 to communicate with other devices. Specifically, it can send information or data to other devices, or receive information or data sent by other devices. The transceiver 803 can include a transmitter and a receiver. The transceiver 803 may further include an antenna, and the number of antennas can be one or more.

[0194] It should be understood that each component in the electronic device is connected through a bus system. Among them, the bus system includes not only a data bus, but also a power bus, a control bus, and a status signal bus.

[0195] The present application also provides a computer storage medium, on which a computer program is stored. When the computer program is executed by a computer, the computer can execute the methods in the above method embodiments. Or rather, the embodiments of the present application also provide a computer program product containing instructions. When the instructions are executed by a computer, the computer executes the methods in the above method embodiments.

[0196] When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from a website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that the computer can access or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a digital video disc (DVD)), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0197] According to one or more embodiments of the present application, a data processing method is provided, including:

[0198] Obtain first panoramic image information with a first resolution and second panoramic image information with a second resolution, where the first resolution is less than the second resolution;

[0199] Obtain first attitude information sent by a first head-mounted display device;

[0200] Determine first visible image information corresponding to the first head-mounted display device according to the first attitude information and the second panoramic image information;

[0201] Determine first target image information based on the first visual image information and the first panoramic image information;

[0202] Send the first target image information to the first head-mounted display device.

[0203] According to one or more embodiments of the present application, the determining the first visual image information corresponding to the first head-mounted display device according to the first pose information and the second panoramic image information includes:

[0204] Determine first region information within the field of view corresponding to the first pose information from the second panoramic image information according to the first pose information;

[0205] Use the image information within the first region information in the second panoramic image information as the first visual image information corresponding to the first head-mounted display device.

[0206] According to one or more embodiments of the present application, the determining first target image information based on the first visual image information and the first panoramic image information includes:

[0207] Adjust the resolution of the first panoramic image information from the first resolution to the second resolution to obtain third panoramic image information;

[0208] Obtain second region information corresponding to the first region information in the third panoramic image information;

[0209] Replace the original image information within the second region information based on the first visual image information to obtain first target image information.

[0210] According to one or more embodiments of the present application, the method further includes:

[0211] Obtain a first coding result and a second coding result from a sending-end device;

[0212] Decode the first coding result and the second coding result to obtain the first panoramic image information and the second panoramic image information respectively.

[0213] According to one or more embodiments of the present application, the sending-end device is configured to:

[0214] Preprocess the original panoramic image information through a first processor to obtain preprocessed panoramic image information in the YUV format;

[0215] Determine first initial panoramic image information with the resolution of the first resolution according to the preprocessed panoramic image information;

[0216] Encoding the first initial panoramic image information through a second processor to obtain the first encoding result;

[0217] Fragmenting the panoramic image information to be processed according to a preset fragmentation rule to obtain a second initial panoramic image information including a plurality of first sub-image information;

[0218] Encoding the second initial panoramic image information through the second processor and the plurality of first sub-image information to obtain the second encoding result.

[0219] According to one or more embodiments of the present application, when the sending device is used to encode the second initial panoramic image information through the second processor and the plurality of first sub-image information to obtain the second encoding result, it is specifically used for:

[0220] Based on the second processor, parallelly encoding the plurality of first sub-image information included in the second initial panoramic image information through a plurality of encoders to obtain the second encoding result.

[0221] According to one or more embodiments of the present application, the method further includes:

[0222] Obtaining current second pose information sent by a second head-mounted display device;

[0223] Determining second visible image information corresponding to the second head-mounted display device according to the second pose information and the second panoramic image information;

[0224] Determining second target image information according to the second visible image information and the second panoramic image information;

[0225] Sending the second target image information to the second head-mounted display device.

[0226] According to one or more embodiments of the present application, there is provided an image display method for a head-mounted display device, including:

[0227] Obtaining first pose information of the head-mounted display device;

[0228] Obtaining first panoramic image information with a first resolution and second panoramic image information with a second resolution, where the first resolution is less than the second resolution;

[0229] Determining first visible image information corresponding to the head-mounted display device according to the first pose information and the second panoramic image information;

[0230] Determining first target image information according to the first visible image information and the first panoramic image information;

[0231] Display the first target image information.

[0232] According to one or more embodiments of the present application, a head-mounted display device is provided, including:

[0233] An acquisition unit, configured to acquire first attitude information of the head-mounted display device;

[0234] The acquisition unit is further configured to acquire first panoramic image information with a first resolution and second panoramic image information with a second resolution, where the first resolution is less than the second resolution;

[0235] A determination unit, configured to determine first visible image information corresponding to the head-mounted display device according to the first attitude information and the second panoramic image information;

[0236] The determination unit is further configured to determine first target image information according to the first visible image information and the first panoramic image information;

[0237] A display unit, configured to display the first target image information.

[0238] According to one or more embodiments of the present application, an electronic device is provided, including:

[0239] A processor; and

[0240] A memory, configured to store executable instructions of the processor;

[0241] Wherein, the processor is configured to implement the above method by executing the executable instructions.

[0242] According to one or more embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented.

[0243] Those of ordinary skill in the art can realize that the modules and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the present application.

[0244] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or module can be in electrical, mechanical, or other forms.

[0245] The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical modules, that is, they can be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. For example, in each embodiment of the present application, the various functional modules can be integrated into one processing module, or each module can exist physically alone, or two or more modules can be integrated into one module.

[0246] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed in the present application can easily think of changes or substitutions, which should all be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data processing method, characterized in that, Including: Obtaining first panoramic image information with a first resolution and second panoramic image information with a second resolution, where the first resolution is less than the second resolution; Obtaining first pose information sent by a first head-mounted display device; Determining first visible image information corresponding to the first head-mounted display device according to the first pose information and the second panoramic image information; Determining first target image information according to the first visible image information and the first panoramic image information; Sending the first target image information to the first head-mounted display device.

2. The method according to claim 1, characterized in that, Determining first visible image information corresponding to the first head-mounted display device according to the first pose information and the second panoramic image information includes: Determining first region information within a field of view corresponding to the first pose information from the second panoramic image information according to the first pose information; Taking the image information within the first region information in the second panoramic image information as the first visible image information corresponding to the first head-mounted display device.

3. The method according to claim 2, wherein Determining first target image information according to the first visible image information and the first panoramic image information includes: Adjusting the resolution of the first panoramic image information from the first resolution to the second resolution to obtain third panoramic image information; Obtaining second region information corresponding to the first region information in the third panoramic image information; Replacing the original image information within the second region information based on the first visible image information to obtain first target image information.

4. The method according to claim 3, wherein The method further includes: Obtaining a first encoding result and a second encoding result from a sending-end device; Decoding the first encoding result and the second encoding result to obtain the first panoramic image information and the second panoramic image information respectively.

5. The method according to claim 4, characterized in that, The sending-end device is configured to: Preprocess original panoramic image information through a first processor to obtain to-be-processed panoramic image information in YUV format; Determine first initial panoramic image information with the first resolution according to the to-be-processed panoramic image information; Encode the first initial panoramic image information through a second processor to obtain the first encoding result; Slice the to-be-processed panoramic image information according to a preset slicing rule to obtain second initial panoramic image information including a plurality of first sub-image information; Encode the second initial panoramic image information through the second processor and the plurality of first sub-image information to obtain the second encoding result.

6. The method according to claim 5, wherein When the sending-end device is configured to encode the second initial panoramic image information through the second processor and the plurality of first sub-image information to obtain the second encoding result, it is specifically configured to: Based on the second processor, parallelly encode the plurality of first sub-image information included in the second initial panoramic image information through a plurality of encoders to obtain the second encoding result.

7. The method according to claim 1, wherein The method further includes: Obtaining second pose information sent by a second head-mounted display device; Determining second visible image information corresponding to the second head-mounted display device according to the second pose information and the second panoramic image information; Determine second target image information based on the second visual image information and the second panoramic image information; Send the second target image information to the second head-mounted display device.

8. An image display method for a head-mounted display device, characterized in that, Comprising: Obtain first attitude information of the head-mounted display device; Obtain first panoramic image information with a first resolution and second panoramic image information with a second resolution, wherein the first resolution is less than the second resolution; Determine first visual image information corresponding to the head-mounted display device according to the first attitude information and the second panoramic image information; Determine first target image information according to the first visual image information and the first panoramic image information; Display the first target image information.

9. A head-mounted display device, characterized in that, Comprising: An obtaining unit, configured to obtain first attitude information of the head-mounted display device; The obtaining unit is further configured to obtain first panoramic image information with a first resolution and second panoramic image information with a second resolution, wherein the first resolution is less than the second resolution; A determining unit, configured to determine first visual image information corresponding to the head-mounted display device according to the first attitude information and the second panoramic image information; The determining unit is further configured to determine first target image information according to the first visual image information and the first panoramic image information; A displaying unit, configured to display the first target image information.

10. An electronic device, characterized in that, Comprising: A processor; And A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the method according to any one of claims 1-8 by executing the executable instructions.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the method according to any one of claims 1-8 is implemented.

Images