Video data processing method, device and network side equipment

Abstract

The present invention provides a method, device and network side equipment for processing video data, the method comprising: decoding and copying a low-resolution video stream to obtain a first low-resolution video stream and a second low-resolution video stream; performing resolution enhancement processing on the second low-resolution video stream to obtain a high-resolution video stream; performing tile division processing on the high-resolution video stream to obtain a plurality of high-resolution video tiles; using the first low-resolution video stream as the background layer of a panoramic video, using the high-resolution video tile as the enhancement layer of the panoramic video, and splicing to generate a panoramic video stream; in an embodiment of the present invention, a panoramic video acquisition terminal uploads the low-resolution video stream to a network side equipment, and the network side equipment uses a video resolution enhancement technology to perform video resolution enhancement processing on the network side equipment, and cooperates with the tile division transmission technology, so as to obtain a relatively high-resolution panoramic video, thereby ensuring the reasonable use of uplink bandwidth and downlink bandwidth.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a method and apparatus for processing video data and network-side equipment. Background Art

[0002] As an emerging hot spot of video services, panoramic video consists of 360-degree all-round video images. Users can choose any angle to watch the content of interest and get an immersive experience. Panoramic video has broad application prospects in real-life guided tours of scenic spots and urban blocks, panoramic displays of corporate appearances, real-life virtual tours of museums and exhibitions, and on-site records of major events, cultural performances, and sports events. At the same time, in real-time video services, panoramic live broadcasts and panoramic video calls have also become current research hotspots.

[0003] However, ordinary resolution in 360-degree video cannot achieve the clarity presented by traditional video, which is easy to produce screen door effect, seriously affecting the user experience. Therefore, panoramic videos generally have ultra-high resolution (4K and above), and the amount of data is huge compared to traditional videos. Considering the limitations of end-to-end hardware conditions, network bandwidth, and computer resources, how to transmit panoramic videos with huge data processing volume in a limited network environment has become a key issue.

[0004] The end-to-end network transmission solution of panoramic video can be divided into a series of processes: acquisition, editing, storage, transfer, and broadcast. Since 360-degree panoramic video has the characteristics of ultra-high resolution, the huge amount of data often requires a high network bandwidth during the transmission process. The adaptive transmission method using terminal FOV (field of view) feedback in the prior art is especially the tile transmission method, such as Figure 1 As shown in the figure, ultra-high-definition panoramic videos such as 8K are collected and uploaded, and decoded and mapped with the cloud server. ① is the original ultra-high-definition video stream, which is tiled and encoded as an enhancement layer, and ② is downsampled to reduce the resolution to 2K and encoded as a background layer. The two panoramic video streams are downloaded adaptively based on the FOV feedback of the viewing terminal, which can improve the rational use of the bandwidth of the video distribution network during the panoramic video transmission process. However, the problem that still exists is that the Tile-based panoramic video transmission solution can improve the use of the network downlink bandwidth and save the video distribution network bandwidth, but for services such as panoramic live broadcast and panoramic video calls that have high requirements for the network uplink bandwidth, its effect is relatively limited. In other words, the existing solution only saves the network downlink bandwidth, and the congestion of the uplink bandwidth has not improved. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a method, an apparatus and a network-side device for processing video data, so as to solve the problem that the existing panoramic video transmission solution cannot improve the uplink bandwidth congestion.

[0006] In order to solve the above problem, an embodiment of the present invention provides a method for processing video data, which is executed by a network side device and includes:

[0007] Receive at least one low-resolution video stream;

[0008] Decoding and copying the low-resolution video stream to obtain a first low-resolution video stream and a second low-resolution video stream;

[0009] Performing resolution enhancement processing on the second low-resolution video stream to obtain a high-resolution video stream;

[0010] Performing tile division processing on the high-resolution video stream to obtain a plurality of high-resolution video tiles;

[0011] The first low-resolution video stream is used as the background layer of the panoramic video, and the high-resolution video tile is used as the enhancement layer of the panoramic video, and the panoramic video stream is generated by stitching;

[0012] The panoramic video stream is sent to a panoramic video playback terminal.

[0013] Wherein, the method further comprises:

[0014] Determine the FOV area according to the FOV information fed back by the panoramic video playback terminal;

[0015] The method of using the first low-resolution video stream as a background layer of a panoramic video and using the high-resolution video tile as an enhancement layer of the panoramic video to generate a panoramic video stream by stitching includes:

[0016] The first low-resolution video stream is used as the background layer of the panoramic video, and the high-resolution video tiles in the FOV area are used as the enhancement layer of the panoramic video, and the FOV-based panoramic video stream is generated by splicing.

[0017] The step of performing resolution enhancement processing on the second low-resolution video stream to obtain a high-resolution video stream includes:

[0018] Obtaining a target resolution corresponding to an enhancement layer of a panoramic video stream;

[0019] The second low-resolution video stream is subjected to resolution enhancement processing to obtain a high-resolution video stream having a target resolution.

[0020] The step of using the first low-resolution video stream as a background layer of the panoramic video includes:

[0021] The first low-resolution video stream is subjected to equirectangular projection ERP and encoding to form a background layer of the panoramic video.

[0022] The high-resolution video tile in the FOV area is used as an enhancement layer of the panoramic video, including:

[0023] ERP and encoding are performed on the high-resolution video tiles within the FOV area to form an enhanced layer of the panoramic video.

[0024] Wherein, in the case where the network side device is an IP multimedia subsystem network side device, the panoramic video playback terminal is a caller of a panoramic video call, and the method includes:

[0025] receiving a panoramic video request sent by the calling party, and forwarding the panoramic video request to the called party of the panoramic video call;

[0026] The receiving at least one low-resolution video stream comprises:

[0027] In a case where the called party supports a panoramic video function, at least one low-resolution video sent by the called party is received.

[0028] The embodiment of the present invention further provides a video data processing device, which is applied to a network side device, including:

[0029] A first receiving module, used for receiving at least one low-resolution video stream;

[0030] A first processing module decodes and copies the low-resolution video stream to obtain a first low-resolution video stream and a second low-resolution video stream;

[0031] A second processing module is used to perform resolution enhancement processing on the second low-resolution video stream to obtain a high-resolution video stream;

[0032] A third processing module is used to perform tile division processing on the high-resolution video stream to obtain a plurality of high-resolution video tiles;

[0033] A generation module, configured to use the first low-resolution video stream as a background layer of a panoramic video and the high-resolution video tile as an enhancement layer of the panoramic video to generate a panoramic video stream by stitching;

[0034] The sending module is used to send the panoramic video stream to the panoramic video playback terminal.

[0035] An embodiment of the present invention further provides a network side device, including a processor and a transceiver, wherein the transceiver receives and sends data under the control of the processor, and the processor is used to perform the following operations:

[0036] Receive at least one low-resolution video stream;

[0037] Decoding and copying the low-resolution video stream to obtain a first low-resolution video stream and a second low-resolution video stream;

[0038] Performing resolution enhancement processing on the second low-resolution video stream to obtain a high-resolution video stream;

[0039] Performing tile division processing on the high-resolution video stream to obtain a plurality of high-resolution video tiles;

[0040] The first low-resolution video stream is used as the background layer of the panoramic video, and the high-resolution video tile is used as the enhancement layer of the panoramic video, and the panoramic video stream is generated by stitching;

[0041] The panoramic video stream is sent to a panoramic video playback terminal.

[0042] An embodiment of the present invention further provides a network side device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned video data processing method when executing the program.

[0043] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the method for processing video data as described above are implemented.

[0044] The above technical solution of the present invention has at least the following beneficial effects:

[0045] In the video data processing method, device and network side equipment of the embodiments of the present invention, the panoramic video acquisition terminal uploads a low-resolution video stream to the network side equipment, and the network side equipment uses video resolution enhancement technology to perform video resolution enhancement processing on the network side equipment, and cooperates with tile partitioning transmission technology to obtain a higher-resolution panoramic video, which saves both network uplink bandwidth and network downlink bandwidth, and ensures the rational use of uplink bandwidth and downlink bandwidth. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram showing a tile transmission scheme for panoramic video in the prior art;

[0047] Figure 2 A flowchart showing the steps of a method for processing video data provided by an embodiment of the present invention;

[0048] Figure 3 A schematic diagram showing a tile transmission scheme for panoramic video in a method for processing video data provided by an embodiment of the present invention;

[0049] Figure 4 A schematic diagram showing a structure of a network side device provided in an embodiment of the present invention;

[0050] Figure 5 A schematic diagram showing the principle of Example 1 provided by an embodiment of the present invention;

[0051] Figure 6 A flowchart showing the steps of Example 1 provided by an embodiment of the present invention;

[0052] Figure 7 A schematic diagram showing the principle of Example 2 provided by an embodiment of the present invention;

[0053] Figure 8 A flowchart showing the steps of Example 2 provided by an embodiment of the present invention;

[0054] Fig. 9 A schematic diagram showing the structure of a video data processing device provided by an embodiment of the present invention;

[0055] Fig.10 The second structural diagram of the network side device provided in the embodiment of the present invention is shown. DETAILED DESCRIPTION

[0056] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0057] like Figure 2 and Figure 3 As shown, an embodiment of the present invention provides a method for processing video data, which is executed by a network side device and includes:

[0058] Step 201, receiving at least one low-resolution video stream; the at least one low-resolution video stream is collected and reported by a panoramic video collection terminal.

[0059] The low-resolution video stream may be one low-resolution panoramic video stream or multiple low-resolution 2D video streams, which is not specifically limited here.

[0060] Step 202, decoding and copying the low-resolution video stream to obtain a first low-resolution video stream and a second low-resolution video stream;

[0061] Step 203, performing resolution enhancement processing on the second low-resolution video stream to obtain a high-resolution video stream;

[0062] Step 204, performing tile division processing on the high-resolution video stream to obtain a plurality of high-resolution video tiles;

[0063] Step 205, using the first low-resolution video stream as the background layer of the panoramic video and the high-resolution video tile as the enhancement layer of the panoramic video, and splicing them to generate a panoramic video stream;

[0064] Step 206: Send the panoramic video stream to the panoramic video playback terminal.

[0065] It should be noted that the network side device can be a cloud server of the general Internet or a cloud server of the IMS network center; in other words, the video data processing method can be used for panoramic live broadcast based on the general Internet, and can also be used for panoramic video calls based on the IMS network, and it is necessary to deploy cloud servers in different communication network centers to realize the functions of panoramic video super-resolution enhancement processing and tile division processing.

[0066] On the one hand, in order to solve the problem of uplink bandwidth congestion in panoramic video live broadcast, the embodiment of the present invention reduces the video capture resolution requirement of the panoramic video capture terminal (which can be an ordinary smart terminal or a professional panoramic video capture device), uploads a lower resolution video stream, and coordinates with tile transmission on the network side to perform resolution video enhancement processing, and then combines the tile division transmission scheme to distribute the panoramic video stream, thereby forming end-to-end panoramic video transmission.

[0067] On the other hand, in order to solve the problem that the uplink network bandwidth of panoramic video calls cannot fully meet the ultra-high-resolution data transmission, the embodiments of the present invention can ensure the quality and experience of the video call while ensuring the rational use of the uplink and downlink bandwidths of the call network by super-resolution enhancement processing of the low-resolution video stream sent by one party of the call on the cloud server in the network center and then send it to the other party of the call through the tile transmission scheme.

[0068] As an optional embodiment, the method further includes:

[0069] Determine the FOV area according to the FOV information fed back by the panoramic video playback terminal;

[0070] Accordingly, step 205 includes:

[0071] The first low-resolution video stream is used as the background layer of the panoramic video, and the high-resolution video tiles in the FOV area are used as the enhancement layer of the panoramic video, and the FOV-based panoramic video stream is generated by splicing.

[0072] The embodiment of the present invention deploys panoramic video super-resolution enhancement processing capabilities in a network center and cooperates with a tile transmission method to improve the utilization of uplink network bandwidth during panoramic video tile transmission.

[0073] The step of performing resolution enhancement processing on the second low-resolution video stream to obtain a high-resolution video stream includes:

[0074] Obtaining a target resolution corresponding to an enhancement layer of a panoramic video stream;

[0075] The second low-resolution video stream is subjected to resolution enhancement processing to obtain a high-resolution video stream having a target resolution.

[0076] For example, the network-side device determines the target resolution of the panoramic video super-resolution enhancement processing based on the resolution of the enhancement layer in the tile transmission scheme, such as 8K.

[0077] For another example, by constructing a multi-frame video super-resolution algorithm model, such as combining generative adversarial networks (GANs) and deep recurrent convolutional networks (DRCN), a deep learning method based on external and internal samples is used to obtain a nonlinear mapping relationship between low-resolution video frames and high-resolution video frames, thereby further determining the target resolution.

[0078] It should be noted that if the second low-resolution video stream is a multi-channel low-resolution 2D video stream, a parallel GPU (Graphics Processing Unit) can be used to perform super-resolution enhancement processing.

[0079] The step of using the first low-resolution video stream as a background layer of the panoramic video includes:

[0080] The first low-resolution video stream is subjected to equirectangular projection ERP and encoding to form a background layer of the panoramic video. That is, the first low-resolution video stream is directly subjected to panoramic video ERP splicing and generated, and is encoded as the background layer using H.265.

[0081] The high-resolution video tile in the FOV area is used as an enhancement layer of the panoramic video, including:

[0082] ERP and encoding are performed on the high-resolution video tiles within the FOV area to form an enhanced layer of the panoramic video.

[0083] like Figure 2As shown in the figure, one or more low-resolution video streams (such as 2K resolution) encoded and uploaded by the acquisition end can be obtained. The H.264 / H.265 encoding method can be used. After decoding, the YUV format video stream is copied into two channels. Channel ① is to directly splice the low-resolution video stream to generate panoramic video ERP, and encode it using H.265 as the background layer. Channel ② is to perform super-resolution enhancement processing on the low-resolution video stream, and splice it to generate 8K high-resolution panoramic video ERP as a tile transmission enhancement layer. Then the high-resolution panoramic video ERP is divided by Tile and encoded by H.265. And according to the FOV information fed back by the panoramic video playback terminal, ① is used as the background stream and the high-resolution tiles in the ②FOV area are combined and rendered in the GPU to generate a panoramic video based on FOV for network distribution.

[0084] It should be noted that the network side device provided in the embodiment of the present invention is deployed on a general Internet and / or IMS network center cloud server to implement panoramic video resolution enhancement processing and tile division. Figure 3 As shown, the network side device includes a communication unit, a transcoding unit, a processing unit and a partitioning unit. The communication unit includes the video stream acquisition and distribution functions, and supports network transmission protocols RTP (Real-time Transport Protocol) / DASH (Adaptive Streaming Protocol) / HLS (Streaming Network Transmission Protocol), etc.; the transcoding unit includes the uploaded video stream decoding and FOV adaptive panoramic video stream encoding functions, and needs to support H.264 / H.265 encoding methods; the processing unit includes the video super-resolution enhancement processing algorithm model and panoramic video stitching and ERP projection; the partitioning unit includes the parameter definition of tile transmission and the grid partitioning function, such as the number of partitions, etc., which can be coordinated with the processing unit.

[0085] As an optional embodiment, when the network side device is an IP multimedia subsystem network side device, the panoramic video playback terminal is a caller of a panoramic video call, and the method includes:

[0086] receiving a panoramic video request sent by the calling party, and forwarding the panoramic video request to the called party of the panoramic video call;

[0087] The receiving at least one low-resolution video stream comprises:

[0088] In a case where the called party supports a panoramic video function, at least one low-resolution video sent by the called party is received.

[0089] Furthermore, the method further comprises:

[0090] Receiving a 2D video stream sent by the calling party;

[0091] The 2D video stream is sent to the called party.

[0092] It should be noted that, during a panoramic video call, the caller requests the panoramic video of the called party. Normally, the called party can only play the 2D video of the caller.

[0093] In order to more clearly describe the method for processing video data provided by the embodiment of the present invention, two examples are provided below for illustration.

[0094] Example 1: The processing method of panoramic live broadcast in a general Internet environment, such as Figure 5 and Figure 6 As shown:

[0095] Step 1: A 360-degree panoramic camera or multiple 2D cameras captures images of the live broadcast site, which can be encoded using encoders such as H.264 / H.265.

[0096] Step 2: A panoramic video stream or multiple 2D video streams are uploaded to the network location where the cloud server of the Internet device is located via WiFi / 5G / dedicated line.

[0097] Step 3: The low-resolution panoramic video stream or multiple 2D video streams are processed in the Internet device to obtain the target resolution fed back by the segmentation unit, and a high-resolution video is obtained by using a video super-resolution method such as NVIDIA's PG-GAN, which generates adversarial network learning. The high-resolution video and the original low-resolution video are spliced ​​to form the background layer and enhancement layer of the panoramic video respectively. This step can be implemented using a general-purpose GPU device.

[0098] Step 4: The high-resolution panoramic video is processed through tile division. According to the FOV parameters fed back by the panoramic video acquisition terminal, the original low-resolution panoramic video is used as the background stream. The high-resolution video tiles in the FOV area are combined and rendered in the GPU to generate a FOV-based panoramic video, which is distributed over the network through the communication unit.

[0099] Step 5: Transmit the enhanced Tile-based FOV panoramic video stream to the panoramic video playback terminal for playback through WiFi / 5G / dedicated line and other connection methods, and the encoding method used is H.265.

[0100] In this example, the panoramic video acquisition terminal acquires, encodes, compresses and uploads one or more lower-resolution video streams, obtains high-resolution panoramic video through the panoramic video resolution enhancement processing module deployed by the network center, and forms a panoramic video stream based on the FOV parameters fed back by the playback terminal, and distributes and transmits it in conjunction with the tile network. This example combines the network's own media control function to ensure collaboration and anti-packet loss, and distributes it to the playback terminal for viewing. In panoramic video transmission application scenarios based on the general Internet, such as panoramic live broadcasting, this example can effectively alleviate the congestion that the upload network may encounter in the transmission link.

[0101] Example 2: A data processing method for a panoramic video call in an IMS network environment, such as Figure 7 and Figure 8 As shown:

[0102] Step 6, the two parties of the call are A and B, where B is the caller and A is the called party. B initiates a request for a panoramic video call (i.e., SDP offer) to A, requesting the panoramic video media function and the viewport processing function. The IMS network cloud server forwards the SDPoffer to the called user A, requesting the panoramic video function. The called user A generates a response (i.e., SDP answer), indicating whether the panoramic video function is supported. The cloud server responds to user B, confirming the panoramic video media function and accepting the viewport processing function.

[0103] Step 7: A uses a 360-degree panoramic camera to collect low-resolution panoramic images, and B uses a traditional camera to collect its 2D images, which can be encoded using encoders such as H.264 / H.265 after media negotiation. A's panoramic video stream and B's 2D video stream are carried and controlled by IMS, and the media data stream is uploaded to the network location where the IMS center cloud server is located, and transmitted through RTP media stream.

[0104] Step 8: A sends a low-resolution panoramic video stream in Figure 4 The network side device described in the description decodes and negotiates with the division unit the tile transmission enhancement layer target resolution. The high-resolution video stream and the original low-resolution video stream are spliced ​​to form the enhancement layer and background layer of the panoramic video respectively. This step can be implemented using a general-purpose GPU device. At this time, the data streams of B and A need to maintain clock synchronization.

[0105] Step 9: The processed high-resolution video and the original low-resolution panoramic video sent by A are divided by tiles and based on the FOV parameters fed back by B, the original low-resolution panoramic video is used as the background stream, the high-resolution video tiles in the FOV area are combined and rendered in the GPU to generate a FOV-based panoramic video, and distributed through the communication unit. User B can feedback FOV information to the cloud server through RTCP, and the cloud server transmits the high-resolution FOV video through the RTP media stream. User B can receive the enhanced tile-based FOV panoramic video stream and transmit it to the playback terminal, which uses H.265 decoding for playback. User A can receive B's 2D video and use / H.264 / H.265 decoding for playback.

[0106] Step 10: Either A or B exits the call, and the call ends.

[0107] In this example, since the operator mainly follows the business model of data services when actually deploying wireless resources (i.e., the uplink traffic volume is much smaller than the downlink traffic volume). In other words, the uplink and downlink resources are asymmetric. However, in the audio and video communication model of the bit rate and resolution negotiated based on SDP offer / answer, they are symmetric. In this asymmetric business model, panoramic video transmission is prone to the problem of insufficient network resources, especially in the uplink, where video calls may be affected and even audio calls may be rolled back. Therefore, an embodiment of the present invention deploys panoramic video resolution enhancement processing capabilities in the IMS network center business capability layer to reduce the demand for uplink network resources. The panoramic video acquisition party can send a lower resolution media stream, and the uplink media stream can transmit a higher resolution media stream through super-resolution processing in the IMS network center to ensure the call experience of the panoramic video callee.

[0108] In this example, during media negotiation, the SDP offer is sent by user B to request the panoramic video media function and the viewport processing function. The IMS network cloud server forwards the SDP offer to the called user A to request the panoramic video function or the 2D video function. The SDP answer is generated by the called user A to respond to whether the cloud server supports the panoramic video function. The cloud server responds to the remote user B to confirm the panoramic video media function and accept the viewport processing function. The calling user B can feedback the FOV information to the cloud server through RTCP. The cloud server processes the FOV panoramic video stream and transmits it through the RTP media stream. User B can receive high-resolution FOV video.

[0109] In summary, the embodiments of the present invention implement the network-centric super-resolution enhancement processing function during the end-to-end network tile transmission of 360-degree panoramic videos, which reduces the requirements for acquisition equipment for current panoramic videos, saves uplink bandwidth of the end-to-end transmission network, improves the efficiency of panoramic video upload, and ensures the quality and viewing experience of panoramic videos. Taking the uploading of 2K lower-resolution panoramic videos mentioned in the embodiments as an example, the target resolution of 8K is negotiated with the Tile transmission and distribution solution, and super-resolution processing is performed. While ensuring the viewing experience, the bit rate of the uploaded panoramic video is reduced from 120Mbps of 8K to 4Mbps of 2K, and the corresponding demand for network uplink bandwidth is also greatly reduced. Combined with the Tile transmission and distribution solution, the network downlink bandwidth is saved by 60% to 80%, and the network bandwidth utilization of the end-to-end network transmission of panoramic videos, including uplink and downlink, is significantly improved.

[0110] Compared with the existing technical solutions, this technical solution is implemented in the form of a system unit, realizes two-way negotiation of video super-resolution processing and tile partitioning transmission, optimizes the tile transmission scheme, and improves the flexibility and versatility of the functions.

[0111] like Fig. 9 As shown, an embodiment of the present invention further provides a video data processing device, which is applied to a network side device, including:

[0112] A first receiving module 901 is used to receive at least one low-resolution video stream;

[0113] A first processing module 902 decodes and copies the low-resolution video stream to obtain a first low-resolution video stream and a second low-resolution video stream;

[0114] The second processing module 903 is used to perform resolution enhancement processing on the second low-resolution video stream to obtain a high-resolution video stream;

[0115] A third processing module 904 is used to perform tile division processing on the high-resolution video stream to obtain a plurality of high-resolution video tiles;

[0116] A generating module 905 is used to use the first low-resolution video stream as a background layer of the panoramic video and the high-resolution video tile as an enhancement layer of the panoramic video to generate a panoramic video stream by splicing;

[0117] The sending module 906 is used to send the panoramic video stream to the panoramic video playback terminal.

[0118] As an optional embodiment, the device further includes:

[0119] An area determination module, used to determine the FOV area according to the FOV information fed back by the panoramic video playback terminal;

[0120] The generation module comprises:

[0121] The generation submodule is used to use the first low-resolution video stream as the background layer of the panoramic video, and the high-resolution video tiles in the FOV area as the enhancement layer of the panoramic video, and stitch them together to generate a panoramic video stream based on the FOV.

[0122] As an optional embodiment, the second processing module includes:

[0123] The first unit is used to obtain a target resolution corresponding to an enhancement layer of a panoramic video stream;

[0124] The second unit is used to perform resolution enhancement processing on the second low-resolution video stream to obtain a high-resolution video stream with a resolution equal to the target resolution.

[0125] As an optional embodiment, the third processing module includes:

[0126] The third unit is used to perform equirectangular projection ERP and encoding on the first low-resolution video stream to form a background layer of the panoramic video.

[0127] As an optional embodiment, the generating submodule includes:

[0128] The fourth unit is used to perform ERP and encoding on the high-resolution video tiles in the FOV area to form an enhancement layer of the panoramic video.

[0129] As an optional embodiment, when the network side device is an IP multimedia subsystem network side device, the panoramic video playback terminal is a caller of a panoramic video call, and the apparatus includes:

[0130] A request receiving module, used for receiving a panoramic video request sent by the calling party, and forwarding the panoramic video request to the called party of the panoramic video call;

[0131] The first receiving module comprises:

[0132] The first receiving submodule is used to receive at least one low-resolution video sent by the called party when the called party supports the panoramic video function.

[0133] The embodiments of the present invention implement a network-centric super-resolution enhancement processing function during the end-to-end network tile transmission of 360-degree panoramic videos, thereby reducing the requirements for acquisition equipment for current panoramic videos, saving uplink bandwidth of the end-to-end transmission network, improving the efficiency of panoramic video uploads, and ensuring the quality of panoramic videos and viewing experience; and realizing two-way negotiation of video super-resolution processing and tile partitioning transmission, optimizing the tile transmission scheme, and improving the flexibility and versatility of the functions.

[0134] It should be noted that the video data processing device provided in the embodiment of the present invention is a device capable of executing the above-mentioned video data processing method. All embodiments of the above-mentioned video data processing method are applicable to the device and can achieve the same or similar beneficial effects.

[0135] like Fig.10 As shown, an embodiment of the present invention further provides a network side device, including a processor 100 and a transceiver 110, wherein the transceiver 110 receives and sends data under the control of the processor 100, and the processor 100 is used to perform the following operations:

[0136] Receive at least one low-resolution video stream;

[0137] Decoding and copying the low-resolution video stream to obtain a first low-resolution video stream and a second low-resolution video stream;

[0138] Performing resolution enhancement processing on the second low-resolution video stream to obtain a high-resolution video stream;

[0139] Performing tile division processing on the high-resolution video stream to obtain a plurality of high-resolution video tiles;

[0140] The first low-resolution video stream is used as the background layer of the panoramic video, and the high-resolution video tile is used as the enhancement layer of the panoramic video, and the panoramic video stream is generated by stitching;

[0141] The panoramic video stream is sent to a panoramic video playback terminal.

[0142] As an optional embodiment, the processor is further configured to perform the following operations:

[0143] Determine the FOV area according to the FOV information fed back by the panoramic video playback terminal;

[0144] The first low-resolution video stream is used as the background layer of the panoramic video, and the high-resolution video tiles in the FOV area are used as the enhancement layer of the panoramic video, and the FOV-based panoramic video stream is generated by splicing.

[0145] As an optional embodiment, the processor is further configured to perform the following operations:

[0146] Obtaining a target resolution corresponding to an enhancement layer of a panoramic video stream;

[0147] The second low-resolution video stream is subjected to resolution enhancement processing to obtain a high-resolution video stream having a target resolution.

[0148] As an optional embodiment, the processor is further configured to perform the following operations:

[0149] The first low-resolution video stream is subjected to equirectangular projection ERP and encoding to form a background layer of the panoramic video.

[0150] As an optional embodiment, the processor is further configured to perform the following operations:

[0151] ERP and encoding are performed on the high-resolution video tiles within the FOV area to form an enhanced layer of the panoramic video.

[0152] As an optional embodiment, when the network side device is an IP multimedia subsystem network side device, the panoramic video playback terminal is a caller of a panoramic video call, and the processor is further configured to perform the following operations:

[0153] receiving a panoramic video request sent by the calling party, and forwarding the panoramic video request to the called party of the panoramic video call;

[0154] In a case where the called party supports a panoramic video function, at least one low-resolution video sent by the called party is received.

[0155] The embodiments of the present invention implement a network-centric super-resolution enhancement processing function during the end-to-end network tile transmission of 360-degree panoramic videos, thereby reducing the requirements for acquisition equipment for current panoramic videos, saving uplink bandwidth of the end-to-end transmission network, improving the efficiency of panoramic video uploads, and ensuring the quality of panoramic videos and viewing experience; and realizing two-way negotiation of video super-resolution processing and tile partitioning transmission, optimizing the tile transmission scheme, and improving the flexibility and versatility of the functions.

[0156] It should be noted that the network side device provided in the embodiment of the present invention is a network side device capable of executing the above-mentioned video data processing method. All embodiments of the above-mentioned video data processing method are applicable to the network side device and can achieve the same or similar beneficial effects.

[0157] An embodiment of the present invention also provides a network side device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, each process in the embodiment of the video data processing method described above is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0158] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, each process in the above-mentioned video data processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0159] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0160] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A device that specifies functions in one or more processes and / or one or more blocks.

[0161] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable storage medium produce a paper product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0162] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that the computer or other programmable device executes a series of operating steps to produce a computer-implemented process, thereby providing instructions executed on the computer or other programmable device for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0163] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for processing video data, executed by a network side device, characterized in that: include: Receiving at least one low-resolution video stream; wherein the low-resolution video stream is a low-resolution panoramic video stream, or the low-resolution video stream is multiple low-resolution 2D video streams; Decoding and copying the low-resolution video stream to obtain a first low-resolution video stream and a second low-resolution video stream; Performing resolution enhancement processing on the second low-resolution video stream to obtain a high-resolution video stream; Performing tile division processing on the high-resolution video stream to obtain a plurality of high-resolution video tiles; The first low-resolution video stream is used as the background layer of the panoramic video, and the high-resolution video tile is used as the enhancement layer of the panoramic video, and the panoramic video stream is generated by stitching; The panoramic video stream is sent to a panoramic video playback terminal.

2. The method according to claim 1, characterized in that The method further comprises: Determine the FOV area according to the FOV information fed back by the panoramic video playback terminal; The method of using the first low-resolution video stream as a background layer of a panoramic video and using the high-resolution video tile as an enhancement layer of the panoramic video to generate a panoramic video stream by stitching includes: The first low-resolution video stream is used as the background layer of the panoramic video, and the high-resolution video tiles in the FOV area are used as the enhancement layer of the panoramic video, and the FOV-based panoramic video stream is generated by splicing.

3. The method according to claim 1, characterized in that The step of performing resolution enhancement processing on the second low-resolution video stream to obtain a high-resolution video stream includes: Obtaining a target resolution corresponding to an enhancement layer of a panoramic video stream; The second low-resolution video stream is subjected to resolution enhancement processing to obtain a high-resolution video stream having a target resolution.

4. The method according to claim 1, characterized in that: The step of using the first low-resolution video stream as a background layer of the panoramic video includes: The first low-resolution video stream is subjected to equirectangular projection ERP and encoding to form a background layer of the panoramic video.

5. The method according to claim 2, characterized in that: The high-resolution video tile in the FOV area is used as an enhancement layer of the panoramic video, including: ERP and encoding are performed on the high-resolution video tiles within the FOV area to form an enhanced layer of the panoramic video.

6. The method according to claim 1, characterized in that In the case where the network side device is an IP multimedia subsystem network side device, and the panoramic video playback terminal is a caller of a panoramic video call, the method includes: receiving a panoramic video request sent by the calling party, and forwarding the panoramic video request to the called party of the panoramic video call; The receiving at least one low-resolution video stream comprises: In a case where the called party supports a panoramic video function, at least one low-resolution video sent by the called party is received.

7. A video data processing device, applied to a network side device, characterized in that: include: A first receiving module is configured to receive at least one low-resolution video stream; wherein the low-resolution video stream is a low-resolution panoramic video stream, or the low-resolution video stream is multiple low-resolution 2D video streams; A first processing module decodes and copies the low-resolution video stream to obtain a first low-resolution video stream and a second low-resolution video stream; A second processing module is used to perform resolution enhancement processing on the second low-resolution video stream to obtain a high-resolution video stream; A third processing module is used to perform tile division processing on the high-resolution video stream to obtain a plurality of high-resolution video tiles; A generation module, configured to use the first low-resolution video stream as a background layer of a panoramic video and the high-resolution video tile as an enhancement layer of the panoramic video to generate a panoramic video stream by stitching; The sending module is used to send the panoramic video stream to the panoramic video playback terminal.

8. A network side device, comprising a processor and a transceiver, wherein the transceiver receives and sends data under the control of the processor, characterized in that: The processor is configured to perform the following operations: Receiving at least one low-resolution video stream; wherein the low-resolution video stream is a low-resolution panoramic video stream, or the low-resolution video stream is multiple low-resolution 2D video streams; Decoding and copying the low-resolution video stream to obtain a first low-resolution video stream and a second low-resolution video stream; Performing resolution enhancement processing on the second low-resolution video stream to obtain a high-resolution video stream; Performing tile division processing on the high-resolution video stream to obtain a plurality of high-resolution video tiles; The first low-resolution video stream is used as the background layer of the panoramic video, and the high-resolution video tile is used as the enhancement layer of the panoramic video, and the panoramic video stream is generated by stitching; The panoramic video stream is sent to a panoramic video playback terminal.

9. A network side device, comprising a memory, a processor and a program stored in the memory and executable on the processor; characterized in that: When the processor executes the program, the method for processing video data according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the method for processing video data as described in any one of claims 1 to 6 are implemented.

Images