An interactive video improvement method, computer device and storage medium

Abstract

The application discloses an interactive video improvement method, in particular to an interactive video improvement method, computer equipment and storage medium which can be used for a cloud terminal, wherein B frames are not contained in a video stream of an interactive video service, and a service timestamp is inserted in a DTS or PTS field according to a value of a PTSDTS flag bit in a PES optional header in a sending process of the video stream. By using the method, the computer equipment and the storage medium, not only the delay can be reduced, but also the use experience of a user in the interactive video service can be improved.

Description

TECHNICAL FIELD

[0001] The application relates to an interactive video improvement method, a computer device and a storage medium, in particular to an interactive video improvement method, a computer device and a storage medium applicable to a cloud terminal. BACKGROUND

[0002] An interactive video service refers to a service in which a user performs operations according to a played video. Cloud application is a typical interactive video service, which places the operation part in the cloud rather than on the local machine, encodes the output of the cloud, such as audio and video output, and transmits the output to the user end for display and play, and the user performs corresponding operations according to the image seen. Cloud application usually includes cloud game, cloud terminal and the like. Cloud application usually has a very high requirement for low latency, otherwise the experience will be greatly affected, so the video compression method with as low decoding delay as possible is usually used in cloud application.

[0003] Video compression usually includes three frames I, P and B. The I frame is an internal encoding frame (also known as a key frame), the P frame is a forward prediction frame (forward reference frame), and the B frame is a bidirectional interpolation frame (bidirectional reference frame). The B frame needs to be combined with the content of the frame after the cloud encoding and terminal decoding, and the existence of the B frame will inevitably increase the time delay by one frame of video time, accounting for about 1 / 3-1 / 4 of the total experience time. Therefore, the video compression used in cloud application will avoid using B frames.

[0004] The PES Optional Header (optional header) of the HLS protocol includes two time stamps DTS and PTS. DTS (Decoding Time Stamp) is a decoding time stamp, which tells the player when to decode the data of this frame. PTS (Presentation Time Stamp) is a display time stamp, which tells the player when to display the data of this frame.

[0005] When the video stream contains B frames, the decoder needs to skip I and P frames first, decode the B frame behind first, and then return to decode the I and P frames in front. When playing, the I and P frames in front need to be played first, and then the B frame behind is played. At this time, the decoding order and the playing order are different, so DTS and PTS need to be identified respectively. Figure 1 The prior art is shown.

[0006] When the video stream does not contain B frames, DTS is equal to PTS. SUMMARY

[0007] The HLS protocol is a protocol suitable for Internet transmission of streaming media, and has wide applicability. However, in the interactive video service, some fields are not actually useful.

[0008] To solve the above problems, the application provides an interactive video improvement method.

[0009] An interactive video improvement method, characterized in that no B frame is contained in a video stream of an interactive video service, and a service timestamp is inserted in a DTS or PTS field according to a value of a PTSDTS flag bit in a PES (Packet Elemental Stream) optional header during transmission of the video stream.

[0010] In the method, the PES optional header can contain 13 bytes, the first three bytes are identifiers, the PTSDTS flag bit containing two bits is used to identify whether to use DTS and PTS, and in the last 8 bytes, 4 bytes are PTS and 4 bytes are DTS.

[0011] In the method, the PTSDTS flag bit can be set to 01, DTS is a video timestamp, and the service timestamp is written into the 4 bytes of PTS.

[0012] In the method, the PTSDTS flag bit can be set to 10, PTS is a video timestamp, and the DTS field is used for the service timestamp.

[0013] In the method, the interactive video service can be a cloud application.

[0014] In the method, the interactive video service can be a video interactive game, the PTSDTS flag bit is set to 01, the service timestamp is taken out by a user-side device during decoding, the service timestamp is updated once every frame output, when a user performs an operation, user input and the service timestamp are sent to a cloud end, and the cloud end performs service processing according to a picture at the time of the user operation.

[0015] In the method, the interactive video service can be a live broadcast type service with multi-party video, the PTSDTS flag bit is set to 01, a receiving end decodes multiple videos after receiving video streams of a main conference and a sub-conference respectively, at this time, the DTS of the main conference is used as a reference, the sub-conference uses the DTS of the main conference currently received as a current service timestamp, and the receiving end aligns the DTS of the main conference with a PTS time axis of the sub-conference to perform synchronous display.

[0016] A computer device, characterized in that it comprises a memory and a processor, the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the interactive video improvement method.

[0017] A computer readable storage medium having stored thereon a computer program, wherein when the computer program is executed by a processor, the processor executes the interactive video improvement method.

[0018] By using the interactive video improvement method of the present application, in the scene where the video stream does not contain B frames, by using the characteristics that only one of DTS and PTS is required in video coding, the other time stamp is used for service time stamp identification, not only the delay can be reduced, but also the user experience in the interactive video service can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0019] The specific embodiments of the present application will be described below with reference to the accompanying drawings, which are exemplary but not limiting.

[0020] Figure 1 Show the prior art;

[0021] Figure 2 Show the flow chart according to an example of an embodiment of the present application;

[0022] Figure 3 Show the flow chart according to another example of an embodiment of the present application; and

[0023] Figure 4 The schematic diagram of the computer device of the embodiment of the present application. DETAILED DESCRIPTION

[0024] The present application aims at the low latency service scene of cloud application, and there is no B frame in video coding, so only DTS is necessary in HLS structure. In the cloud application service scene, the service itself has strong relevance and interactivity with the used video, so we can use PTS in HLS to identify the service time stamp.

[0025] The PES Optional Header contains 13 bytes, of which the first three bytes are identifiers, and the PTSDTSFlags (PTS DTS flag) containing two bits is used to identify whether to use DTS and PTS; in the last 8 bytes, 4 bytes are PTS, and 4 bytes are DTS. The present application can set PTSDTSFlags to 01, at this time DTS is the video time stamp, and the service time stamp is written into the 4 bytes of PTS. The value of PTS will not affect the video decoding, so the decoding method does not need to be changed.

[0026] Similarly, PTSDTSFlags can also be set to 10, at this time PTS is the video time stamp, and the DTS field is used for service time stamp.

[0027] The following example will be described taking PTSDTSFlags with value 01 as an example.

[0028] Figure 2 A flow chart according to an example of an embodiment of the application is shown.

[0029] The example is a video interactive game, such as a cloud game with body sensing or a network video game service. The user side device takes out the service timestamp when decoding, and updates the service timestamp once for each output frame. When the user performs an operation, the user input and the service timestamp are sent to the cloud. The cloud performs service processing according to the picture at the time of the user operation, rather than according to the current cloud encoded picture.

[0030] In this example, the video encoding part of the cloud sends a video stream containing DTS information to the terminal device of the user side, which caches the latest DTS information. The camera of the user side sends the original image to the terminal device. The terminal device sends a video stream to the cloud application, in which the PTS DTS Flags value is 01, the DTS is for video decoding, and the PTS is for the cloud video DTS at the time of user operation. The cloud application performs corresponding service processing.

[0031] Figure 3 A flow chart according to another example of an embodiment of the application is shown.

[0032] The example is a live service with multi-party video, such as multi-party video conference, remote interview, etc. The receiving end needs to decode multiple videos respectively. At this time, the DTS of the main conference is taken as the reference, and the DTS of the main conference received by the branch conference is taken as the PTS of the current encoding. The receiving end aligns the DTS of the main conference and the PTS time axis of the branch conference, and performs synchronous display.

[0033] In this example, there are two branch conferences, but in the present application, the number of branch conferences is not limited to two, but can be more than two. The main conference sends audio and video streams to branch conference 1, branch conference 2 and the receiving end respectively. Branch conference 1 and branch conference 2 perform the following operations respectively: extracting DTS; the branch conference performs local shooting, sets the PTS DTS Flags to 01, and writes the DTS of the main conference into the PTS of the branch conference shooting; then the branch conference sends the audio and video stream containing the service timestamp to the receiving end, in which the PTS DTS Flags value is 01. The receiving end decodes the multiple video streams, aligns the DTS of the main conference and the PTS of the branch conference on the time axis, and plays the multiple videos synchronously according to the aligned time axis.

[0034] Figure 4Fig. 1 is a schematic diagram of a computer device according to an embodiment of the present application. As shown in the figure, the computer device 110 of the present application comprises a memory 111 and a processor 112. The memory 111 stores a computer program 113, and when the processor 112 executes the computer program 113 stored in the memory 111, the processor 112 performs the interactive video improvement method of the present application.

[0035] In addition, the present application also discloses a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor performs the interactive video improvement method of the present application.

[0036] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and appropriate combinations thereof. In the hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component can have multiple functions, or one function or step can be performed by several physical components in cooperation. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on a computer readable medium, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, as known to those skilled in the art, communication media generally includes computer readable instructions, data structures, program modules or other data in modulated data signals such as carrier waves or other transport mechanisms, and can include any information delivery medium.

[0037] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An interactive video improvement method characterized by, The B frame is not included in the video stream of the interactive video service, and during transmission of the video stream, a service timestamp is inserted in a DTS or PTS field according to a value of a PTSDTS flag bit in a PES optional header; The PES optional header includes 13 bytes, the first three bytes are identifiers, the PTSDTS flag bit including two bits is used to identify whether to use the DTS and the PTS, and in the last 8 bytes, 4 bytes are the PTS and 4 bytes are the DTS. The PTSDTS flag bit is set to 01, the DTS is the video timestamp, and the service timestamp is written into the 4 bytes of the PTS. The PTSDTS flag bit is set to 10, the PTS is the video timestamp, and the DTS field is used for the service timestamp.

2. The method of claim 1, wherein the interactive video service is a cloud application.

3. The method of claim 1, wherein the interactive video service is a video interactive game, the PTSDTS flag bit is set to 01, the service timestamp is taken out by a user-side device during decoding, the service timestamp is updated once per frame output, and when a user performs an operation, the user input and the service timestamp are sent to a cloud end, and the cloud end performs service processing according to a picture at the time of the user operation.

4. The method of claim 1, wherein the interactive video service is a live broadcast service with multiple videos, the PTSDTS flag bit is set to 01, a receiving end decodes multiple videos after receiving video streams of a main conference and a sub-conference respectively, at this time, the DTS of the main conference is used as a reference, the sub-conference uses the DTS of the main conference currently received as a current encoded service timestamp, the receiving end aligns the DTS of the main conference with the PTS time axis of the sub-conference, and synchronous display is performed. A computer program product including a memory and a processor, the memory storing the computer program, and when the processor executes the computer program stored in the memory, the processor executes the interactive video improvement method according to any one of claims 1 to 4. When the computer program is executed by the processor, the processor executes the interactive video improvement method according to any one of claims 1 to 4. ​ 5. A computer device, comprising: ​ 6. A computer-readable storage medium having stored thereon a computer program, characterized in that, ​

Images