QUIC protocol-based panoramic video transmission method, apparatus and device, and storage medium

Abstract

The invention discloses a panoramic video transmission method, device and equipment based on a QUIC protocol and a storage medium, and relates to the field of video transmission, the method comprises the following steps: receiving a target transmission tile fragment obtained by a client according to a current view angle range request, the target transmission tile fragment comprising a corresponding first tile fragment in the current view angle range, the first tile fragment is obtained by splitting the panoramic video to be transmitted by the server according to a first preset mode, and the first tile fragment comprises coded data of each frame of picture of the panoramic video from video start to video end; based on a QUIC protocol, a plurality of data streams are established between the server and the client, and each data stream corresponds to one first tile fragment; and transmitting the corresponding first tile fragment in the current view angle range to the client through the corresponding data stream, so that the client decodes and displays the first tile fragment. The transmission efficiency is improved, and the playing delay and the bandwidth resource waste are reduced.

Description

Technical Field

[0001] The present invention relates to the field of video transmission, and in particular, to a panoramic video transmission method, device, equipment and storage medium based on the QUIC protocol. Background Art

[0002] Traditional panoramic video transmission requires stitching and projective transformation of the original panoramic video material collected by a panoramic camera and storing it as a video file in a high-resolution ordinary encapsulation format. The client needs to obtain the entire video file through the network for decoding and rendering, which undoubtedly causes waste of bandwidth and storage resources.

[0003] To solve this problem, existing panoramic video transmission methods usually split the panoramic video into multiple tile shards for decoding and rendering. However, to ensure smooth playback, it is necessary to concurrently transmit the data of multiple tile shards within the viewport range, which requires establishing an independent thread and connection for each tile shard between the client and the server through the TCP protocol to transmit the corresponding data stream. However, as the number of tile shards increases, too many threads and connections will seriously consume performance resources; if the number of threads and connections is restricted, due to the head-of-line blocking of the TCP protocol, the transmission efficiency will be relatively low, and there will be playback latency and waste of bandwidth resources. Summary of the Invention

[0004] Embodiments of the present invention provide a panoramic video transmission method, device, equipment and storage medium based on the QUIC protocol to solve the problems of relatively low transmission efficiency caused by the head-of-line blocking of the existing TCP protocol, as well as playback latency and waste of bandwidth resources.

[0005] In a first aspect, a panoramic video transmission method based on the QUIC protocol is provided, which is applied to a server. The method includes: Receiving a target transmission tile shard requested by a client according to a current viewing range. The target transmission tile shard includes a first tile shard corresponding to the current viewing range. The first tile shard is obtained by splitting the panoramic video to be transmitted by the server in a first preset manner. The first tile shard includes the encoded data of each frame of the panoramic video from the start to the end of the video; Based on the QUIC protocol, establishing multiple data streams between the server and the client, and each data stream corresponds to one of the first tile shards; Transmitting the first tile shard corresponding to the current viewing range to the client through the corresponding data stream, so that the client decodes and displays the first tile shard.

[0006] In one embodiment, the target transmission tile shard further includes a second tile shard, which is obtained by splitting the compressed panoramic video according to a second preset method, and the compressed panoramic video is obtained by compressing the panoramic video; Transmitting the first tile shard corresponding to the current viewing range to the client through a corresponding data stream, so that the client decodes and displays the first tile shard, includes: Transmitting the first tile shard and the second tile shard to the client through a corresponding data stream, so that the client decodes and displays the first tile shard and the second tile shard.

[0007] In one embodiment, transmitting the first tile shard and the second tile shard to the client through a corresponding data stream, includes: Dividing the first tile shard according to a first preset number of frames to obtain a plurality of first tile shard segments; Dividing the second tile shard according to a second preset number of frames to obtain a plurality of second tile shard segments; Transmitting the first tile shard segments of the first tile shard and the second tile shard segments of the second tile shard to the client in sequence through a corresponding data stream.

[0008] In a second aspect, a panoramic video transmission method based on the QUIC protocol is provided, which is applied to a client. The method includes: According to the current viewing range, determining the target transmission tile shard to be requested from the server, where the target transmission tile shard includes the first tile shard corresponding to the current viewing range, and the first tile shard is obtained by splitting the panoramic video to be transmitted by the server according to a first preset method, and the first tile shard includes the encoded data of each frame of the panoramic video from the start to the end of the video; Based on the QUIC protocol, establishing a plurality of data streams between the client and the server, and each data stream corresponds to one of the first tile shards; Receiving the first tile shard corresponding to the current viewing range transmitted by the server through a corresponding data stream, and decoding and displaying the first tile shard.

[0009] In one embodiment, the determining the target transmission tile shard to be requested from the server according to the current viewing range, includes: Determining the target boundary of the current viewing range; Determining the first display area of the current viewing range according to the target boundary; Determine the target transmission tile shards to be requested from the server according to the Euler angle direction of the perspective center point within the first display area.

[0010] In one embodiment, the determining the target transmission tile shards to be requested from the server according to the Euler angle direction of the perspective center point within the first display area includes: Determine the first tile shards to be requested from the server according to the Euler angle direction of the perspective center point within the first display area; Determine the second tile shards to be requested from the server according to the area of the panoramic video; Use the first tile shards and the second tile shards as the target transmission tile shards to be requested from the server.

[0011] In one embodiment, the receiving the first tile shards corresponding to the current perspective range transmitted by the server through the corresponding data stream and decoding and displaying the first tile shards includes: Receive the first tile shard segments of the first tile shards corresponding to the current perspective range transmitted by the server through the corresponding data stream, and store the first tile shard segments in a pre-established data buffer; When the number of the first tile shard segments is greater than or equal to the first preset number of segments, stop receiving the first tile shard segments of the first tile shards transmitted by the server through the corresponding data stream; When the number of the first tile shard segments is less than or equal to the second preset number of segments, continue to receive the first tile shard segments of the first tile shards transmitted by the server through the corresponding data stream.

[0012] In a third aspect, a panoramic video transmission system based on the QUIC protocol is provided. The panoramic video transmission system includes a server and a client, and the server performs the following steps: Receive the target transmission tile shards requested by the client according to the current perspective range. The target transmission tile shards include the first tile shards corresponding to the current perspective range. The first tile shards are obtained by splitting the panoramic video to be transmitted by the server in a first preset manner. The first tile shards include the encoded data of each frame of the panoramic video from the start to the end of the video; Based on the QUIC protocol, establish multiple data streams between the server and the client, and each data stream corresponds to one of the first tile shards; Transmit the first tile shards corresponding to the current perspective range to the client through the corresponding data streams, so that the client decodes and displays the first tile shards.

[0013] In a fourth aspect, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for panoramic video transmission based on the QUIC protocol described in the first aspect above is implemented, or when the processor executes the computer program, the method for panoramic video transmission based on the QUIC protocol described in the second aspect above is implemented.

[0014] In a fifth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the method for panoramic video transmission based on the QUIC protocol described in the first aspect above is implemented, or when the computer program is executed by the processor, the method for panoramic video transmission based on the QUIC protocol described in the second aspect above is implemented.

[0015] In a solution implemented by the above method, device, equipment, and storage medium for panoramic video transmission based on the QUIC protocol, a target transmission tile slice requested and obtained by a receiving client according to a current viewing range is received. The target transmission tile slice includes a first tile slice corresponding to the current viewing range. The first tile slice is obtained by splitting the panoramic video to be transmitted by a server according to a first preset method. The first tile slice includes encoded data of each frame of the panoramic video from the start to the end of the video; based on the QUIC protocol, multiple data streams are established between the server and the client, and each data stream corresponds to one of the first tile slices; the first tile slice corresponding to the current viewing range is transmitted to the client through the corresponding data stream, so that the client decodes and displays the first tile slice. In this embodiment, based on the QUIC protocol, a connection is established between the server and the client. Only one connection establishment is required to achieve parallel and interleaved transmission of multiple data streams, avoiding head-of-line blocking of the TCP protocol, greatly improving the transmission efficiency, enabling the first tile slice to be quickly and stably transmitted to the client through the corresponding data stream; the client can receive the first tile slice in time and perform decoding and display, effectively reducing the playback delay, improving the smoothness of the user's viewing of the panoramic video, while reducing the waste of bandwidth resources and optimizing the overall performance of panoramic video transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0017] Figure 1 It is a schematic diagram of a panoramic video transmission system based on the QUIC protocol in an embodiment of the present invention; Figure 2 It is a flowchart of a panoramic video transmission method based on the QUIC protocol in an embodiment of the present invention; Figure 3 It is a corresponding schematic diagram of the current viewing angle range of a panoramic video in an embodiment of the present invention; Figure 4 It is a corresponding schematic diagram after the change of the current viewing angle range of a panoramic video in an embodiment of the present invention; Figure 5 It is a schematic diagram of an electronic device in an embodiment of the present invention. Detailed implementation manners

[0018] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0019] The panoramic video transmission method provided by the embodiments of the present invention can be applied in the application environment as Figure 1 shown. Specifically, the panoramic video transmission method can be applied in a panoramic video transmission system based on the QUIC protocol. The panoramic video transmission system includes a client and a server as Figure 1 shown. The two communicate through a network, and are used to solve the problems of low transmission efficiency caused by head-of-line blocking of the existing TCP protocol, as well as playback delay and waste of bandwidth resources; it is applicable to application scenarios such as panoramic video with large amounts of data, low latency, and high bandwidth requirements, such as virtual reality, live broadcast, and remote medical treatment. In the panoramic video transmission system, the client is responsible for receiving, decoding, and displaying the target transmission tile shards transmitted, while the server performs target transmission tile shard transmission with the client through multiple parallel data streams. The client includes but is not limited to various personal computers, smartphones, tablets, televisions, VR glasses, or wearable devices, etc.; the server can be implemented by an independent server or a server cluster composed of multiple servers.

[0020] For the convenience of understanding the embodiments of the present invention, the nouns involved in the embodiments of the present invention are introduced as follows: QUIC protocol: The QUIC protocol is an enhancement of UDP. It introduces features of the TCP protocol, such as packet retransmission mechanism, congestion control algorithm, and multiplexing function, while effectively avoiding the head-of-line blocking problem in the TCP protocol. QUIC allows multiple data streams (Streams) to be transmitted in parallel on the same connection. These data streams can transmit data in an interleaved and efficient manner, thereby significantly improving the efficiency and speed of data transmission. In addition, the QUIC protocol also ensures that data packets within a single data stream can be delivered in order, which improves the overall transmission efficiency while strictly ensuring the reliability of the data.

[0021] Tile: A tile is a unit obtained by splitting a panoramic video into multiple areas according to certain rules.

[0022] The following is a detailed description of the panoramic video transmission method based on the QUIC protocol provided by an embodiment of the present invention. It should be understood that the following description is only an implementation method of the present invention and does not constitute a limitation of the present invention.

[0023] First, as Figure 2 As shown, a panoramic video transmission method based on the QUIC protocol is provided, and the method is applied in Figure 1 The server in the example is used as an example to illustrate the following steps: S101, receiving target transmission tile segments obtained by the client according to the current viewing angle range request, wherein the target transmission tile segments include first tile segments corresponding to the current viewing angle range, wherein the first tile segments are obtained by splitting the panoramic video to be transmitted by the server according to a first preset method, and the first tile segments include encoded data of each frame of the panoramic video from the beginning to the end of the video.

[0024] In one embodiment of the present invention, the target transmission tile segment includes the first tile segment corresponding to the current viewing angle range, wherein the first tile segment includes multiple tiles, which can be determined according to the current viewing angle range and the specific splitting rules, and are not limited here. Figure 3 As shown, for example, the current viewing angle range requested by the client is a 180-degree area in a 360-degree panoramic video. At this time, the first magnet slice includes all the tile slices obtained by splitting the 180-degree area.

[0025] As an example, Figure 3As shown in the figure, if the original resolution of the panoramic video is 8000x4000 pixels and the panoramic video is split into 128 tile slices according to the rule of 16x8, the size of each tile is 500x500 pixels. When the current viewing range requested by the client is the first display area, the first tile slices corresponding to the current viewing range are all the first tile slices split from the first display area. For example, the number of the first tile slices corresponding to the first display area is 32, and these 32 tile slices are all the first tile slices split from the first display area.

[0026] The first tile slices are obtained by splitting the panoramic video to be transmitted by the server according to a first preset method. As an example, HEVC Tile independent decoding can be used to split the panoramic video according to the first preset method, so as to obtain multiple independently encoded first tile slices. For example, splitting according to a region of a first preset size, multiple independently encoded first tile slices are obtained, such as splitting according to a region of 500x500 size.

[0027] The first tile slices include the encoded data of each frame of the panoramic video from the start to the end of the video. By way of a simple analogy, imagine that the panoramic video is a huge picture album composed of many pages (frames), and the first tile slices are like small parts cut from each page of the picture album. Each small part of the tile slice has its own complete set of "small page numbers" (frames), and from the first page of the picture album (the start of the video) to the last page (the end of the video) represents the encoded data of each frame of the panoramic video from the start to the end of the video. It should be understood that this analogy is only for facilitating the understanding of the related embodiments of the present invention and does not constitute a limitation.

[0028] Receiving the target transmission tile slices requested by the client according to the current viewing range can be understood as follows: when a user views digital content (such as a map, a game scene or a virtual environment) through an electronic device (such as a smart phone, a tablet computer or a VR glasses), the electronic device will intelligently request the required data segments, that is, the target transmission tile slices, from the server according to the user's current viewing angle or viewing range. It can also be understood as sending a acquisition signal for acquiring the target transmission tile slices to the server. In this way, the electronic device can efficiently manage the loading and rendering of data, ensure that the user only sees the content they currently need to view, and thus improve the performance of the application and the user experience.

[0029] S102. Based on the QUIC protocol, establish multiple data streams between the server and the client, and each data stream corresponds to one of the first tile slices.

[0030] In an embodiment of the present invention, when the client requests the target tile shards within the current viewing range, the server uses the QUIC protocol to establish multiple parallel data streams between the server and the client according to the client's request. Each data stream is assigned to a first tile shard, which can ensure that each first tile shard can be transmitted independently and in parallel.

[0031] It can be understood that by establishing a connection through the QUIC protocol, these parallel data streams can be efficiently managed, avoiding the head-of-line blocking problem in the traditional TCP protocol, improving the transmission efficiency of panoramic videos, reducing latency, and ensuring the stable transmission of the first tile shards.

[0032] For example, if the current viewing range requested by the client involves the first display area of the panoramic video, the server will assign a corresponding data stream to each first tile shard in the first display area; if the first display area corresponds to 4 first tile shards, then 4 parallel data streams will be corresponding, that is, the 4 first tile shards to be transmitted can be respectively corresponding to different data stream identifiers. For example, in the order of the numbers of the first tile shards, unique data stream IDs are assigned in sequence, so as to ensure that each data stream transmits the encoded data of the corresponding first tile shard respectively.

[0033] It should be understood that when establishing multiple data streams between the server and the client through the QUIC protocol, version negotiation of encryption algorithms and merging of the transport layer handshake link can be adopted to realize the connection between the two, which is not limited here.

[0034] S103. Transmit the first tile shards corresponding to the current viewing range to the client through the corresponding data streams, so that the client decodes and displays the first tile shards.

[0035] In an embodiment of the present invention, after the server establishes a connection with the client through the QUIC protocol, further, the first tile shards corresponding to the current viewing range will be transmitted to the client through the corresponding data streams. After receiving the first tile shards corresponding to the current viewing range, the client can use the HEVC Tile encoding technology to splice and combine each received first tile shard into a complete video data, and then perform unified decoding processing. Further, the decoded video data is rendered, and finally the video content required by the user is displayed on the client screen, effectively ensuring the smoothness and clarity of video playback and improving the user's viewing experience.

[0036] In summary, in one solution provided by the embodiments of the present invention, based on the QUIC protocol, a connection is established between the server and the client. Only one connection establishment is required to enable parallel and interleaved transmission of multiple data streams, avoiding the head-of-line blocking of the TCP protocol, greatly improving the transmission efficiency, and enabling the first tile slice to be quickly and stably transmitted to the client through the corresponding data stream. The client can receive the first tile slice in a timely manner, decode and display it, effectively reducing the playback latency, improving the smoothness of the user's viewing of the panoramic video, while reducing the waste of bandwidth resources and optimizing the overall performance of the panoramic video transmission.

[0037] In one embodiment, that is, in step S103, the target transmission tile slice further includes a second tile slice; That is, in the step of transmitting the first tile slice corresponding to the current viewing range to the client through the corresponding data stream so that the client decodes and displays the first tile slice, the following steps are included: S1031. Transmit the first tile slice and the second tile slice to the client through the corresponding data stream so that the client decodes and displays the first tile slice and the second tile slice.

[0038] In this embodiment, the target transmission tile slice not only includes the first tile slice corresponding to the current viewing range, but also includes a second tile slice. The second tile slice includes the encoded data of each frame of the compressed panoramic video from the start to the end of the video.

[0039] Among them, the second tile slice is obtained through the following steps: Compress the panoramic video to obtain a compressed panoramic video; Split the compressed panoramic video according to a second preset method to obtain the second tile slice.

[0040] In this embodiment, the server first compresses the panoramic video, that is, compresses the original panoramic video to be transmitted to obtain a compressed panoramic video, that is, the background video, to reduce the data volume. The compressed panoramic video retains the main visual information of the original panoramic video, but the resolution is reduced to meet the requirements of the transmission bandwidth and the decoding ability of the client. Then, the server splits the compressed panoramic video according to a second preset method to obtain the second tile slice.

[0041] As an example, such as Figure 3As shown, assume that the resolution of the original panoramic video is 8000x4000. First, the server compresses the original panoramic video to obtain a panoramic video with a resolution of 2000x1000. Then, the server splits the compressed panoramic video into multiple regions to obtain the second tile shards. For example, assume that the original panoramic video is compressed by a factor of 4 and split into 4x2, resulting in 8 second tile shards.

[0042] It should be understood that here each data stream still corresponds to a second tile shard, that is, each first tile shard and second tile shard respectively correspond to a data stream. Among them, the first tile shard is high-precision video data, and the second tile shard is low-precision video data.

[0043] In an embodiment of the present invention, according to the request sent by the client to obtain the target transmission tile shard, the server transmits the first tile shard and the second tile shard to the client through multiple parallel data streams respectively. After receiving these transmission data, the client can use the HEVC Tile decoding technology to decode the encoded data of the transmitted first tile shard to present a clear video picture. At the same time, in order to avoid a blank screen after the viewing angle is rotated, the client also uses the HEVC Tile decoding technology to decode the encoded data of the transmitted second tile shard, which can fill the screen when the viewing angle is switched, so as to ensure that when the viewing angle of the client changes, the video playback is kept smooth and the delay caused by the switch is reduced, and the blank and delay caused by the viewing angle switch are reduced, improving the user experience.

[0044] In one embodiment, that is, in step S1031, that is, the step of transmitting the first tile shard and the second tile shard to the client through the corresponding data stream includes the following steps: S1031A. Divide the first tile shard according to the first preset number of frames to obtain a number of first tile shard segments; S1031B. Divide the second tile shard according to the second preset number of frames to obtain a number of second tile shard segments; S1031C. Transmit the first tile shard segments of the first tile shard and the second tile shard segments of the second tile shard to the client through the corresponding data streams in sequence.

[0045] In an embodiment of the present invention, the server can divide the first tile shard according to a first preset number of frames, thereby splitting the first tile shard into several first tile shard segments. Each of the first tile shard segments may include one frame of data or multiple frames of data, which is related to the first preset number of frames for division here and does not constitute a limitation to the present invention. Similarly, the server also divides the second tile shard according to a second preset number of frames, thereby splitting the second tile shard into several second tile shard segments. Each of the second tile shard segments may include one frame of data or multiple frames of data, which is related to the second preset number of frames for division here and also does not constitute a limitation to the present invention. Then, the server transmits the divided first tile shard segments and second tile shard segments to the client in sequence through corresponding data streams, so that the client can decode and display them one by one.

[0046] For example, if the first preset number of frames is 1 frame and the second preset number of frames is 1 frame, the server will divide the first tile shard into several first tile shard segments at intervals of 1 frame and transmit them to the client in sequence through the corresponding data stream. At the same time, the second tile shard can also be divided into 1 frame and transmitted to the client through the corresponding data stream. After receiving the first tile shard segments and the second tile shard segments, the client splices and combines them into a complete H265 video data for unified decoding and display.

[0047] It should be understood that the above first preset number of frames and the second preset number of frames may be the same or different, and can be preset according to the actual situation. For example, the first preset number of frames can be 1 frame, 5 frames, 10 frames, 15 frames or 20 frames, etc.; the first preset number of frames can also be 1 frame, 5 frames, 10 frames, 15 frames or 20 frames, etc., which do not constitute a limitation to the present invention here.

[0048] Second, as Figure 2 shown, a panoramic video transmission method based on the QUIC protocol is provided. Taking the method applied to the Figure 1 client as an example, the method includes the following steps: S201. Determine the target transmission tile shard to be requested from the server according to the current viewing range. The target transmission tile shard includes the corresponding first tile shard within the current viewing range. The first tile shard is obtained by the server splitting the panoramic video to be transmitted according to a first preset method. The first tile shard includes the encoded data of each frame of the panoramic video from the start to the end of the video.

[0049] In an embodiment of the present invention, the client first determines the target transmission tile shards that need to be requested and obtained according to the viewing range of the current user, and then sends a request to the server to obtain the target transmission tile shards. Specifically, the description of the target transmission tile shards can refer to the description of the embodiment of the foregoing step S101. To avoid repetition, it will not be elaborated here.

[0050] S202. Based on the QUIC protocol, establish multiple data streams between the client and the server, and each data stream corresponds to one of the first tile shards.

[0051] In an embodiment of the present invention, for the specific embodiment of establishing multiple data streams between the client and the server based on the QUIC protocol, with each data stream corresponding to one of the first tile shards, reference can be made to the description of the embodiment of the foregoing step S102. To avoid repetition, it will not be elaborated here either.

[0052] S203. Receive the first tile shards corresponding to the current viewing range transmitted by the server through the corresponding data streams, and decode and display the first tile shards.

[0053] In an embodiment of the present invention, after the client receives the first tile shards transmitted by the server through parallel data streams, the HEVC Tile decoding technology can be used to decode the transmitted first tile shards and display the corresponding pictures in real time, so as to present a high-resolution panoramic image matching the current viewing range of the user. Through decoding optimization and parallel processing, it is ensured that the picture display is smooth and without delay.

[0054] In summary, in a solution provided by the embodiment of the present invention, based on the QUIC protocol, a connection is established between the client and the server. With only one connection establishment, parallel and interleaved transmission of multiple data streams can be achieved, avoiding the head-of-line blocking of the TCP protocol, greatly improving the transmission efficiency, enabling the first tile shards to be quickly and stably transmitted to the client through the corresponding data streams; the client can receive the first tile shards in a timely manner and perform decoding and display, effectively reducing the playback delay, improving the smoothness of the user's viewing of the panoramic video, while reducing the waste of bandwidth resources and optimizing the overall performance of the panoramic video transmission.

[0055] In one embodiment, that is, in step S201, that is, in the step of determining the target transmission tile shards that need to be requested from the server according to the current viewing range, the following steps are included: S2011. Determine the target boundary of the current viewing range; S2012. Determine the first display area of the current viewing range according to the target boundary; S2013. Determine the target transmission tile shards to be requested from the server according to the Euler angle direction of the viewing center point in the first display area.

[0056] In an embodiment of the present invention, a coordinate system can be established with the center of the screen as the origin. According to the viewing boundary angles in the horizontal and vertical directions, the corresponding coordinate ranges are converted to determine the target boundary, and then the area of the current viewing range is determined as the first display area according to the target boundary. For example, as Figure 3 shown in the rectangular area; then, the following steps are executed: S2013A. Determine the first tile shard to be requested from the server according to the Euler angle direction of the viewing center point in the first display area; S2013B. Determine the second tile shard to be requested from the server according to the area of the panoramic video; S2013C. Use the first tile shard and the second tile shard as the target transmission tile shards to be requested from the server.

[0057] In an embodiment of the present invention, after determining the first display area, the first tile shard to be requested from the server can be determined according to the Euler angle direction of the viewing center point in the first display area. For example, the user is watching a 360-degree panoramic concert video. At this time, the Euler angle of the viewing center point in the first display area is (40°, 50°). According to the preset algorithm, the first tile shard covering the performance area of the singer in the center of the stage can be located. Then, according to the area of the panoramic video, the second tile shard is determined to supplement the surrounding content involved, such as the audience seats, the venue environment, etc. Finally, the above-determined first tile shard and the second tile shard are integrated together as the target transmission tile shards to be requested from the server.

[0058] In an embodiment, that is, in step S203, that is, in the step of receiving the first tile shard corresponding to the current viewing range transmitted by the server through the corresponding data stream and decoding and displaying the first tile shard, the following steps are included: S2031. Receive the first tile shard segment of the first tile shard corresponding to the current viewing range transmitted by the server through the corresponding data stream, and store the first tile shard segment in a pre-established data buffer; S2032. When the number of the first tile shard segments is greater than or equal to the first preset number of segments, stop receiving the first tile shard segments of the first tile shard transmitted by the server through the corresponding data stream; S2033. When the number of the first tile shard segments is less than or equal to the second preset number of segments, continue to receive the second tile shard segments of the second tile shard transmitted by the server through the corresponding data stream.

[0059] In an embodiment of the present invention, after the client receives the first tile shard segments transmitted by the server through the data stream, the received first tile shard segments are stored in a pre-established data buffer; when the number of the first tile shard segments stored in the data buffer is greater than or equal to the first preset number of segments (for example, the number of frames in a GOP (Group Of Pictures), that is, the frame data set between two I frames, such as 20 frame segments), the client stops receiving the first tile shard segments transmitted by the server to avoid exceeding the capacity of the data buffer and over-occupying the memory of the electronic device; on the contrary, when the number of the first tile shard segments stored in the data buffer is less than or equal to the second preset number of segments (for example, (for example, half of the number of frames in a GOP (Group Of Pictures), such as 10 frame segments), the client continues to receive the first tile shard segments to ensure the smoothness of the subsequent decoded and displayed data. This way of dynamically controlling data reception can ensure the playback quality while avoiding resource waste or stuttering.

[0060] In an embodiment, as Figure 4 shown, that is, after step S203, that is, after receiving the first tile shard transmitted by the server through the data stream and decoding and displaying the first tile shard, the following steps are included: S204. When the current viewing range changes, determine a new target transmission tile shard to be requested from the server according to the new current viewing range.

[0061] In an embodiment of the present invention, when it is detected that the current viewing range changes, first, according to the new current viewing range, recalculate the new target tile shard to be requested from the server; then, based on the difference between the new target transmission tile shard and the target transmission tile shard of the client, clean up the irrelevant first tile shards in the data buffer, and reuse the first tile shards in the data buffer related to the new target transmission tile shard, thereby saving bandwidth. Furthermore, obtain the new first tile shards not stored in the data buffer, so as to ensure the smoothness of the user's viewing angle switching and the real-time nature of the panoramic video content, and improve the adaptability of the panoramic video transmission and the user experience.

[0062] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution is prior or subsequent, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0063] In a third aspect, a panoramic video transmission system based on the QUIC protocol is provided, and the panoramic video transmission system based on the QUIC protocol corresponds one-to-one with the panoramic video transmission method based on the QUIC protocol in the above embodiments. As Figure 1 shown, the panoramic video transmission system based on the QUIC protocol includes a server and a client; The server performs the following steps: Receiving a target transmission tile shard requested by the client according to the current viewing range, where the target transmission tile shard includes a first tile shard corresponding to the current viewing range, and the first tile shard is obtained by splitting the panoramic video to be transmitted by the server in a first preset manner, and the first tile shard includes encoded data of each frame of the panoramic video from the start to the end of the video; Based on the QUIC protocol, establishing multiple data streams between the server and the client, and each data stream corresponds to one of the first tile shards; Transmitting the first tile shard corresponding to the current viewing range to the client through the corresponding data stream, so that the client decodes and displays the first tile shard.

[0064] In an embodiment, the target transmission tile shard further includes a second tile shard, and the second tile shard is obtained by splitting the compressed panoramic video in a second preset manner, and the compressed panoramic video is obtained by compressing the panoramic video; The step of transmitting the first tile shard corresponding to the current viewing range to the client through the corresponding data stream, so that the client decodes and displays the first tile shard, includes: Transmitting the first tile shard and the second tile shard to the client through the corresponding data stream, so that the client decodes and displays the first tile shard and the second tile shard.

[0065] In an embodiment, the step of transmitting the first tile shard and the second tile shard to the client through the corresponding data stream includes: Dividing the first tile shard according to a first preset number of frames to obtain a plurality of first tile shard segments; Dividing the second tile shard according to a second preset number of frames to obtain a plurality of second tile shard segments; Sequentially transmitting the first tile shard segments of the first tile shard and the second tile shard segments of the second tile shard to the client through the corresponding data stream.

[0066] The client performs the following steps: Determine the target transmission tile shards to be requested from the server according to the current view range, where the target transmission tile shards include the first tile shards corresponding to the current view range, and the first tile shards are obtained by splitting the panoramic video to be transmitted by the server in accordance with a first preset method, and the first tile shards include the encoded data of each frame of the panoramic video from the start to the end of the video; Based on the QUIC protocol, establish multiple data streams between the client and the server, and each data stream corresponds to one of the first tile shards; Receive the first tile shards corresponding to the current view range transmitted by the server through the corresponding data stream, and decode and display the first tile shards.

[0067] In one embodiment, the determining the target transmission tile shards to be requested from the server according to the current view range includes: Determine the target boundary of the current view range; Determine the first display area of the current view range according to the target boundary; Determine the target transmission tile shards to be requested from the server according to the Euler angle direction of the view center point within the first display area.

[0068] In one embodiment, the determining the target transmission tile shards to be requested from the server according to the Euler angle direction of the view center point within the first display area includes: Determine the first tile shards to be requested from the server according to the Euler angle direction of the view center point within the first display area; Determine the second tile shards to be requested from the server according to the area of the panoramic video; Use the first tile shards and the second tile shards as the target transmission tile shards to be requested from the server.

[0069] In one embodiment, the receiving the first tile shards corresponding to the current view range transmitted by the server through the corresponding data stream, and decoding and displaying the first tile shards includes: Receive the first tile shard segments of the first tile shards corresponding to the current view range transmitted by the server through the corresponding data stream, and store the first tile shard segments in a pre-established data buffer; When the number of the first tile shard segments is greater than or equal to a first preset number of segments, stop receiving the first tile shard segments of the first tile shards transmitted by the server through the corresponding data stream; When the number of the first tile shard segments is less than or equal to a second preset number of segments, continue to receive the first tile shard segments of the first tile shard transmitted by the server through a corresponding data stream.

[0070] For the specific limitations of the panoramic video transmission system based on the QUIC protocol, reference can be made to the limitations of the panoramic video transmission method based on the QUIC protocol in the foregoing text, which will not be elaborated herein.

[0071] In one embodiment, an electronic device is provided. The electronic device may be a server, and its internal structure diagram may be as Figure 5 shown. The electronic device includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store data required for executing the panoramic video transmission method. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements the panoramic video transmission method based on the QUIC protocol described in the first aspect, or when the computer program is executed by the processor, it implements the panoramic video transmission method based on the QUIC protocol described in the second aspect.

[0072] In one embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented: Receive a target transmission tile shard requested by a client according to a current viewing range. The target transmission tile shard includes a corresponding first tile shard within the current viewing range. The first tile shard is obtained by splitting a panoramic video to be transmitted by the server in a first preset manner. The first tile shard includes encoded data of each frame of the panoramic video from the start of the video to the end of the video; Based on the QUIC protocol, establish a plurality of data streams between the server and the client, and each data stream corresponds to one of the first tile shards; Transmit the first tile shard corresponding to the current viewing range to the client through the corresponding data stream, so that the client decodes and displays the first tile shard; Or, Determine the target transmission tile shards to be requested from the server according to the current viewing range. The target transmission tile shards include the first tile shards corresponding to the current viewing range. The first tile shards are obtained by splitting the panoramic video to be transmitted by the server in accordance with a first preset method. The first tile shards include the encoded data of each frame of the panoramic video from the start to the end of the video. Based on the QUIC protocol, establish multiple data streams between the client and the server, with each data stream corresponding to one of the first tile shards. Receive the first tile shards corresponding to the current viewing range transmitted by the server through the corresponding data streams, and decode and display the first tile shards.

[0073] In one embodiment, a computer-readable storage medium is provided. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the following steps are implemented: Receive the target transmission tile shards requested by the client according to the current viewing range. The target transmission tile shards include the first tile shards corresponding to the current viewing range. The first tile shards are obtained by splitting the panoramic video to be transmitted by the server in accordance with a first preset method. The first tile shards include the encoded data of each frame of the panoramic video from the start to the end of the video. Based on the QUIC protocol, establish multiple data streams between the server and the client, with each data stream corresponding to one of the first tile shards. Transmit the first tile shards corresponding to the current viewing range to the client through the corresponding data streams, so that the client decodes and displays the first tile shards. Or, Determine the target transmission tile shards to be requested from the server according to the current viewing range. The target transmission tile shards include the first tile shards corresponding to the current viewing range. The first tile shards are obtained by splitting the panoramic video to be transmitted by the server in accordance with a first preset method. The first tile shards include the encoded data of each frame of the panoramic video from the start to the end of the video. Based on the QUIC protocol, establish multiple data streams between the client and the server, with each data stream corresponding to one of the first tile shards. Receive the first tile shards corresponding to the current viewing range transmitted by the server through the corresponding data streams, and decode and display the first tile shards.

[0074] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0075] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0076] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A panoramic video transmission method based on the QUIC protocol, characterized in that: Applied to the server, the method includes: Receive target transmission tile segments acquired by the client according to the current viewing angle range request, wherein the target transmission tile segments include first tile segments corresponding to the current viewing angle range, the first tile segments are obtained by the server by splitting the panoramic video to be transmitted according to a first preset method, and the first tile segments include encoded data of each frame of the panoramic video from the beginning to the end of the video; Based on the QUIC protocol, multiple data streams are established between the server and the client, each of the data streams corresponding to one of the first tile shards; The first magnetic tile segment corresponding to the current viewing angle is transmitted to the client through the corresponding data stream, so that the client decodes and displays the first magnetic tile segment.

2. The panoramic video transmission method based on the QUIC protocol according to claim 1, characterized in that: The target transmission tile fragments further include second tile fragments, the second tile fragments are obtained by splitting the compressed panoramic video according to a second preset method, and the compressed panoramic video is obtained by compressing the panoramic video; The transmitting the first magnetic tile slice corresponding to the current viewing angle range to the client through the corresponding data stream so that the client decodes and displays the first magnetic tile slice, includes: The first tile slice and the second tile slice are transmitted to the client through corresponding data streams, so that the client decodes and displays the first tile slice and the second tile slice.

3. The panoramic video transmission method based on the QUIC protocol according to claim 2, characterized in that: The transmitting the first tile slice and the second tile slice to the client through corresponding data streams includes: Dividing the first magnetic tile slice according to a first preset frame number to obtain a plurality of first magnetic tile slice segments; Dividing the second magnetic tile slice according to a second preset frame number to obtain a plurality of second magnetic tile slice segments; The first tile slice segment of the first tile slice and the second tile slice segment of the second tile slice are sequentially transmitted to the client through corresponding data streams.

4. A panoramic video transmission method based on the QUIC protocol, characterized in that: Applied to a client, the method comprises: Determine, according to the current viewing angle range, a target transmission tile slice that needs to be requested from the server, wherein the target transmission tile slice includes a first tile slice corresponding to the current viewing angle range, wherein the first tile slice is obtained by splitting the panoramic video to be transmitted by the server according to a first preset method, and the first tile slice includes encoded data of each frame of the panoramic video from the beginning to the end of the video; Based on the QUIC protocol, multiple data streams are established between the client and the server, each of the data streams corresponding to one of the first tile shards; Receive the first magnetic tile fragment corresponding to the current viewing angle range transmitted by the server through the corresponding data stream, and decode and display the first magnetic tile fragment.

5. The panoramic video transmission method based on the QUIC protocol as claimed in claim 4, characterized in that: The step of determining the target transmission tile slices to be requested from the server according to the current viewing angle range includes: Determining a target boundary of the current viewing angle range; Determining a first display area of ​​the current viewing angle range according to the target boundary; According to the Euler angle direction of the viewing angle center point in the first display area, it is determined that a target transmission tile slice needs to be requested to the server.

6. The panoramic video transmission method based on the QUIC protocol according to claim 5, characterized in that: The step of determining the target transmission tile fragments to be requested from the server according to the Euler angle direction of the viewing angle center point in the first display area includes: Determine a first tile slice that needs to be requested from the server according to the Euler angle direction of the viewing angle center point in the first display area; Determining, according to the area of ​​the panoramic video, a second tile segment that needs to be requested from the server; The first tile slice and the second tile slice are used as target transmission tile slices that need to be requested to the server.

7. The panoramic video transmission method based on the QUIC protocol according to claim 4, characterized in that: The receiving the first tile slice corresponding to the current viewing angle range transmitted by the server through the corresponding data stream, and decoding and displaying the first tile slice, includes: Receiving a first tile slice segment corresponding to a first tile slice within the current viewing angle range transmitted by the server through a corresponding data stream, and storing the first tile slice segment in a pre-established data buffer; When the number of the first tile shard fragments is greater than or equal to a first preset number of fragments, stop receiving the first tile shard fragment of the first tile shard transmitted by the server through the corresponding data stream; When the number of the first tile shard fragments is less than or equal to the second preset number of fragments, the first tile shard fragments of the first tile shard transmitted by the server through the corresponding data stream continue to be received.

8. A panoramic video transmission system based on the QUIC protocol, characterized in that: The panoramic video transmission system includes a server and a client, and the server performs the following steps: Receive target transmission tile segments acquired by the client according to the current viewing angle range request, wherein the target transmission tile segments include first tile segments corresponding to the current viewing angle range, the first tile segments are obtained by the server by splitting the panoramic video to be transmitted according to a first preset method, and the first tile segments include encoded data of each frame of the panoramic video from the beginning to the end of the video; Based on the QUIC protocol, multiple data streams are established between the server and the client, each of the data streams corresponding to one of the first tile shards; The first magnetic tile segment corresponding to the current viewing angle is transmitted to the client through the corresponding data stream, so that the client decodes and displays the first magnetic tile segment.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the panoramic video transmission method based on the QUIC protocol as described in any one of claims 1 to 3 is implemented, or, when the processor executes the computer program, the panoramic video transmission method based on the QUIC protocol as described in any one of claims 4 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the panoramic video transmission method based on the QUIC protocol as described in any one of claims 1 to 3, or, when the computer program is executed by the processor, it implements the panoramic video transmission method based on the QUIC protocol as described in any one of claims 4 to 7.

Images