Video processing method and apparatus, and electronic device and storage medium
By generating IP frame group video streams with interval settings, the problems of live broadcast delay and encoding resource consumption are solved, and the video code rate and real-time reduction are achieved, which improves the live broadcast experience.
Patent Information
- Application Number
- PCT/CN2024/096467
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2024-05-30
- Publication Date
- 2024-12-05
AI Technical Summary
In live broadcast scenarios, the longer GOP length causes the delay in the live broadcast screen when the audience enters the live broadcast room. The prior art may increase the consumption of encoding resources and video code rate through two-channel video stream push, affecting the audience's live broadcast experience.
Generate a first video stream and a second video stream, wherein the second video stream sets an intra-encoded image frame and a predicted encoded image frame IP frame group at intervals of a specific number of video frames, each GOP of the first video stream and a second One or more IP frame groups of the video stream correspond to each IP frame group, each IP frame group includes an intra-encoded image frame I frame and a predicted encoded image frame P frame. In this way, the encoding resource consumption and video code rate are reduced and the live broadcast delay is reduced. .
Through this method, the video code rate and live broadcast delay are reduced, the live broadcast real-time performance and the viewer's viewing experience are improved, and the correct decoding and playback of video frames is ensured.
Smart Images

Figure CN2024096467_05122024_PF_FP_ABST
Abstract
Description
Video processing method, device, electronic device and storage medium
[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 31, 2023, with application number 202310633415.1 and invention name “Video processing method, device, electronic device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The present disclosure relates to the field of communications, and in particular to a video processing method, device, electronic device, and storage medium. Background Art
[0003] In a live broadcast scenario, when a viewer enters the live broadcast room, the server pushes the live image to the viewer's terminal device. The video stream containing the live image includes one or more GOPs (Group of Pictures), and the first frame of each GOP is an I-frame (Intra Coded Picture).
[0004] When pushing, in order to ensure that the viewer's terminal device can decode and play the live picture normally, the server needs to push the live picture to the terminal device starting from the first frame of the GOP (i.e., I frame) where the current live picture is located when the viewer enters the live room. When the GOP length is relatively large, pushing the live picture to the terminal device starting from the first frame of the GOP where the current live picture is located when the viewer enters the live room will cause a large delay in the live picture. For example, the GOP length is 50 frames, and the time when the viewer enters the live room corresponds to the 40th video frame of the current GOP. However, in order to ensure that the terminal device can decode normally, the server needs to push the live picture from the 1st video frame to the 40th video frame in the current GOP to the terminal device, resulting in a delay of 39 frames of video frames. Based on this, the existing technology proposes that two video streams can be encoded, each of which has a complete GOP structure and can be decoded and played to obtain the live picture. The GOP length of the first video stream is greater than the GOP length of the second video stream. When the audience enters the live broadcast room, the second video stream is pushed to the audience first, using the shorter GOP length to reduce the delay of the live broadcast image, and then switching to the first video stream to continue pushing the live broadcast image to the audience.
[0005] However, when pushing live images to viewers through two video streams, encoding the second video stream with a complete GOP structure may result in greater encoding resource consumption, increase the video bit rate when sending it down, increase the live broadcast delay, and affect the viewer's experience of watching the live broadcast.
[0006] Summary of the Invention
[0007] The embodiments of the present disclosure provide a video processing method, apparatus, electronic device, and storage medium to solve one or more of the above technical problems.
[0008] In a first aspect, an embodiment of the present disclosure provides a video processing method, including:
[0009] Based on a video captured by a first terminal device, a first video stream and a second video stream are generated by the first terminal device, wherein the first video stream includes a plurality of consecutive GOPs, and the second video stream includes a plurality of intra-frame coded image frames and predictive coded image frame IP frame groups arranged at intervals of a specific number of video frames; each GOP of the first video stream corresponds to one or more IP frame groups of the second video stream; each IP frame group includes an intra-frame coded image frame I frame and a predictive coded image frame P frame following the I frame; each video frame in each IP frame group has the same picture content as a video frame having the same decoding timestamp DTS in the first video stream; and the number of P frames included in each IP frame group is the same as the number of reference frames of the P frames in the first video stream;
[0010] The first video stream and the second video stream are sent to a server.
[0011] In a second aspect, an embodiment of the present disclosure provides a video processing method, including:
[0012] Based on the video captured by the first terminal device, a first video stream and a second video stream are generated, wherein the first video stream includes multiple continuous GOPs, and the second video stream includes multiple intra-frame coded image frames and predictive coded image frame IP frame groups arranged at intervals of a specific number of video frames; each GOP of the first video stream corresponds to one or more IP frame groups of the second video stream; each IP frame group includes an intra-frame coded image frame I frame and a predictive coded image frame P frame after the I frame; each video frame in each IP frame group has the same picture content as the video frame with the same decoding timestamp DTS in the first video stream; the number of P frames included in each IP frame group is the same as the number of reference frames of the P frames in the first video stream.
[0013] In a third aspect, an embodiment of the present disclosure provides a video processing method, including:
[0014] Send live data push request to the server;
[0015] receiving a target IP frame group and a target playback stream from the server, wherein the target IP frame group is an IP frame group in the second video stream whose decoding timestamp DTS is before the reception time of the live data push request and the time interval between the DTS and the reception time is within a preset time length; and the target playback stream is a video frame in the first video stream whose DTS is after the last predictively coded image frame P frame in the target IP frame group and starts from the P frame;
[0016] The target IP frame group and the target playback stream are decoded.
[0017] In a fourth aspect, an embodiment of the present disclosure provides a video processing device, including:
[0018] A first generating unit is configured to generate, by the first terminal device, a first video stream and a second video stream based on a video captured by a first terminal device, wherein the first video stream includes a plurality of consecutive GOPs, and the second video stream includes a plurality of intra-frame coded image frames and predictive coded image frame IP frame groups arranged at intervals of a specific number of video frames; each GOP of the first video stream corresponds to one or more IP frame groups of the second video stream; each IP frame group includes an intra-frame coded image frame I frame and a predictive coded image frame P frame following the I frame; each video frame in each IP frame group has the same picture content as a video frame having the same decoding timestamp DTS in the first video stream; and the number of P frames included in each IP frame group is the same as the number of reference frames of the P frames in the first video stream;
[0019] The first sending unit is configured to send the first video stream and the second video stream to a server.
[0020] In a fifth aspect, an embodiment of the present disclosure provides a video processing device, including:
[0021] The second generating unit is used to generate a first video stream and a second video stream based on the video captured by the first terminal device, wherein the first video stream includes multiple continuous GOPs, and the second video stream includes multiple intra-frame coded image frames and predictive coded image frame IP frame groups arranged at intervals of a specific number of video frames; each GOP of the first video stream corresponds to one or more IP frame groups of the second video stream; each IP frame group includes an intra-frame coded image frame I frame and a predictive coded image frame P frame after the I frame; each video frame in each IP frame group has the same picture content as the video frame with the same decoding timestamp DTS in the first video stream; the number of P frames included in each IP frame group is the same as the number of reference frames of the P frames in the first video stream.
[0022] In a sixth aspect, an embodiment of the present disclosure provides a video processing device, including:
[0023] A request sending unit, used to send a live data push request to the server;
[0024] a data receiving unit, configured to receive a target IP frame group and a target playback stream from the server, wherein the target IP frame group is an IP frame group in the second video stream whose decoding timestamp DTS is before the reception time of the live data push request and whose time interval between the DTS and the reception time is within a preset time length; and the target playback stream is a video frame in the first video stream whose DTS is after the last predictively coded image frame P frame in the target IP frame group and starts from the P frame;
[0025] A data decoding unit is used to decode the target IP frame group and the target playback stream.
[0026] In a seventh aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, enable the processor to implement the steps of the method described in the first aspect above.
[0027] In an eighth aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, enable the processor to implement the steps of the method described in the second aspect above.
[0028] In a ninth aspect, an embodiment of the present disclosure provides an electronic device comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, enable the processor to implement the steps of the method described in the third aspect above.
[0029] In a tenth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, which is used to store computer-executable instructions. When the computer-executable instructions are executed by a processor, the steps of the method described in the first aspect are implemented.
[0030] In an eleventh aspect, an embodiment of the present disclosure provides a computer-readable storage medium for storing computer-executable instructions, which implement the steps of the method described in the second aspect when executed by a processor.
[0031] In a twelfth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, which is used to store computer-executable instructions. When the computer-executable instructions are executed by a processor, the steps of the method described in the third aspect are implemented.
[0032] In one or more embodiments of the present disclosure, a first video stream and a second video stream can be generated, wherein the first video stream includes multiple consecutive GOPs, and the second video stream includes multiple intra-frame coded image frames and predictive coded image frame IP frame groups spaced at intervals of a specific number of video frames. Each GOP of the first video stream corresponds to one or more IP frame groups of the second video stream, each IP frame group includes an intra-frame coded image frame (I frame) and a predictive coded image frame (P frame) following the I frame. Each video frame in each IP frame group has the same picture content as a video frame with the same decoding timestamp in the first video stream, and the number of P frames included in each IP frame group is the same as the number of reference frames of the P frames in the first video stream. Since the IP frame groups in the second video stream are spaced at intervals of a specific number of video frames, the second video stream is not a complete video stream. Therefore, during video distribution, encoding and sending the second video stream can reduce encoding resource consumption, reduce the bit rate of the sent video, and reduce the delay in pushing the live broadcast image to the terminal device, thereby ensuring the real-time performance of the live broadcast. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate one or more embodiments of the present disclosure or technical solutions in the prior art, the following briefly introduces the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings described below are only some embodiments described in the present disclosure. Those skilled in the art can derive other drawings based on these drawings without inventive effort.
[0034] FIG1 is a schematic diagram of the prior art provided by an embodiment of the present disclosure;
[0035] FIG2 is a schematic diagram of a scenario of a video processing method provided by an embodiment of the present disclosure;
[0036] FIG3 is a flow chart of a video processing method according to an embodiment of the present disclosure;
[0037] FIG4a is a schematic diagram of a first video stream and a second video stream provided by an embodiment of the present disclosure;
[0038] FIG4 b is a schematic diagram of a first video stream and a second video stream provided by another embodiment of the present disclosure;
[0039] FIG4c is a schematic diagram of a first video stream and a second video stream provided by another embodiment of the present disclosure;
[0040] FIG4 d is a schematic diagram of a first video stream and a second video stream provided by another embodiment of the present disclosure;
[0041] FIG5 is a flow chart of a video processing method according to another embodiment of the present disclosure;
[0042] FIG6 is a flow chart of a video processing method provided by yet another embodiment of the present disclosure;
[0043] FIG7a is a schematic diagram of pushing a first video stream and a second video stream according to an embodiment of the present disclosure;
[0044] FIG7 b is a schematic diagram of pushing a first video stream and a second video stream provided by another embodiment of the present disclosure;
[0045] FIG7c is a schematic diagram of pushing a first video stream and a second video stream according to another embodiment of the present disclosure;
[0046] FIG7 d is a schematic diagram of pushing a first video stream and a second video stream according to another embodiment of the present disclosure;
[0047] FIG8 is a flow chart of a video processing method provided by yet another embodiment of the present disclosure;
[0048] FIG9 is a schematic structural diagram of a video processing device provided by an embodiment of the present disclosure;
[0049] FIG10 is a schematic structural diagram of a video processing device provided by another embodiment of the present disclosure;
[0050] FIG11 is a schematic structural diagram of a video processing device provided by another embodiment of the present disclosure;
[0051] FIG12 is a schematic structural diagram of an electronic device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION
[0052] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of the present disclosure, the technical solutions in one or more embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in one or more embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on one or more embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present disclosure.
[0053] The embodiments of the present disclosure provide a video processing method. To facilitate understanding of the various embodiments of the present disclosure, the technical terms involved in the various embodiments of the present disclosure are first explained here.
[0054] In video codecs, video frames are usually divided into three types: I, P, and B:
[0055] An I-frame (Intra Coded Picture) is an independent frame that carries all its own information and can be decoded independently without referring to other video frames.
[0056] P-frames (Predictive Coded Pictures) require reference to the previous I-frame to be decoded. They encode the difference / residual between the current video frame and the previously referenced I-frame (the number of reference frames is greater than or equal to 1). During decoding, the previously cached video frames are superimposed on the difference / residual of the current frame to generate the final image. Compared with I-frames, P-frames have a smaller data size.
[0057] B-frames (Bidirectionally Predictive Coded Pictures) are bidirectionally predictively coded frames. This means they record the difference (or residual) between the current frame and the preceding and following frames. Decoding a B-frame requires not only retrieving the previously cached frame but also decoding the following frame. The final image is generated by overlaying the data from the preceding and following frames with the current frame's data. B-frames have the smallest data size but place high demands on decoding performance.
[0058] A Group of Pictures (GOP) is a set of consecutive video frames consisting of I / P / B frames. In a GOP, the first frame must be an I-frame to ensure that subsequent frames can be decoded and played. The GOP length refers to the number of frames between two adjacent I-frames. During video encoding, GOP length is a critical parameter that affects the bitrate of the encoded video. Increasing the GOP length reduces the number of I-frames in the video, improving the video encoding compression ratio and reducing the bitrate of the encoded video.
[0059] DTS (Decode Time Stamp) refers to the decoding timestamp, which is used to indicate the time when the player decodes the video frame.
[0060] PTS (Presentation Time Stamp) refers to the display timestamp, which is used to indicate the time when the player plays the video frame.
[0061] When there are no B frames in the video stream, the DTS and PTS of each frame are equal, that is, the decoding order is consistent with the display order. When there are B frames in the video stream, because B frames need to refer to the previous and next frames for decoding, the DTS and PTS of each frame may be different.
[0062] For example, for a group of video streams with a display order of I, B, B, P, B, B, P, the B frames need to refer to the previous and next I frames and P frames, or refer to the previous P frames and the next P frames when decoding. Therefore, the decoding order and playback order of this group of video streams may be shown in Table 1 below.
[0063] Table 1
[0064] FIG1 is a schematic diagram of the prior art provided by an embodiment of the present disclosure. In FIG1 , n is a positive integer greater than or equal to 0, and each video frame is arranged in the order of DTS. As shown in FIG1 , in a live broadcast scenario, the GOP length of a group of video streams is 10, and a GOP includes an I frame and 9 P frames. When the server receives the live broadcast data push request sent by the terminal device at a time corresponding to the P frame with the frame number n+8, since the P frame cannot be decoded independently, the live broadcast image is pushed to the terminal device starting from the I frame with the frame number n, thereby introducing a multi-frame live broadcast image delay. In addition, the solution of pushing live broadcast images based on two video streams with a complete GOP structure in the prior art will bring about a large consumption of encoding resources, increase the video bit rate when the video frame is sent, increase the live broadcast delay, and affect the audience's experience of watching the live broadcast.
[0065] Based on this, the embodiments of the present disclosure provide a video processing method to solve one or more of the above technical problems.
[0066] FIG2 is a schematic diagram of a scenario of a video processing method provided by an embodiment of the present disclosure. As shown in FIG2 , the scenario includes a terminal device 101 on the host side, a server 102, and a terminal device 103 on the audience side. The terminal device on the host side interacts with the server during the live broadcast and pushes the live image to the terminal device on the audience side. In this embodiment, the terminal device on the host side and the terminal device on the audience side can include electronic devices such as mobile phones, computers, tablet computers, car computers, wearable devices, etc. that have the function of displaying live images. The server can be a single server or a server cluster.
[0067] FIG3 is a flow chart of a video processing method provided by an embodiment of the present disclosure. The method may be executed by a first terminal device, which may be a terminal device on the broadcast side. As shown in FIG3 , the flow includes:
[0068] Step S302: Based on the video captured by the first terminal device, the first terminal device generates a first video stream and a second video stream, wherein the first video stream includes a plurality of consecutive GOPs, and the second video stream includes a plurality of IP frame groups spaced at intervals of a specific number of video frames; each GOP of the first video stream corresponds to one or more IP frame groups of the second video stream; each IP frame group includes an I frame and a P frame following the I frame; each video frame in each IP frame group has the same picture content as a video frame with the same DTS in the first video stream; and the number of P frames included in each IP frame group is the same as the number of reference frames of the P frames in the first video stream;
[0069] Step S304: Send the first video stream and the second video stream to the server.
[0070] In this embodiment, a first video stream and a second video stream can be generated. The first video stream includes multiple consecutive GOPs, and the second video stream includes multiple intra-frame coded image frames and predictively coded image frame IP frame groups spaced at intervals of a specific number of video frames. Each GOP in the first video stream corresponds to one or more IP frame groups in the second video stream. Each IP frame group includes an intra-frame coded image frame (I frame) and a predictively coded image frame (P frame) following the I frame. Each video frame in each IP frame group has the same image content as a video frame with the same DTS in the first video stream, and the number of P frames included in each IP frame group is the same as the number of reference frames for the P frames in the first video stream. Because the IP frame groups in the second video stream are spaced at intervals of a specific number of video frames, the second video stream is not a complete video stream. Therefore, during video distribution, encoding and sending the second video stream can reduce encoding resource consumption, reduce the bit rate of the sent video, and reduce the delay in pushing the live image to the terminal device, thereby ensuring the real-time performance of the live broadcast.
[0071] In step S302, a first terminal device, such as a terminal device on the host's side, records the host's live broadcast process via a camera to obtain a video, and generates a first video stream and a second video stream based on the recorded video. The first video stream is a video stream encoded according to a codec standard, including but not limited to any coding standard such as H.264 and H.265. The GOP size of the first video stream can be set as required. Since the first video stream has multiple consecutive GOPs and the second video stream does not have consecutive GOPs, only the first video stream can be played normally.
[0072] The second video stream includes multiple IP frame groups spaced at intervals of a specific number of video frames. Each IP frame group includes one I-frame and one or more P-frames following the I-frame. Because the IP frame groups are non-contiguous, the second video stream composed of the IP frame groups is not a continuous stream. Therefore, if the second video stream exists independently of the first video stream, the second video stream cannot be played normally.
[0073] In one embodiment, the specific number has multiple values. The value of the specific number is different between different IP frame groups in the second video stream. Multiple specific numbers are set to achieve different intervals of the second video stream. For example, for the first IP frame group and the second IP frame group of the second video stream, the two IP frame groups are set at an interval of 5 video frames. For the second IP frame group and the third IP frame group of the second video stream, the two IP frame groups are set at an interval of 10 video frames. The specific value of the specific number can be flexibly configured according to needs.
[0074] In the first video stream and the second video stream, each GOP of the first video stream corresponds to one or more IP frame groups of the second video stream. Each video frame in each IP frame group has the same screen content as the video frame with the same DTS in the first video stream. Furthermore, the number of P frames included in each IP frame group is the same as the number of reference frames of the P frames in the first video stream.
[0075] In one embodiment, when the first video stream does not include B frames, the I frames in the IP frame group have the same picture content as the P frames with the same DTS in the first video stream, the P frames in the IP frame group have the same picture content as the P frames with the same DTS in the first video stream, and all video frames in the IP frame group are set continuously.
[0076] Figure 4a is a schematic diagram of a first video stream and a second video stream provided by an embodiment of the present disclosure. Since video frames are typically decoded in their decoding order, Figure 4a illustrates the first and second video streams using DTS order as an example. In Figure 4a, n is a positive integer greater than or equal to 0, and the video frames are arranged in DTS order. The first video stream includes multiple consecutive GOPs of relatively long length, each GOP including one I-frame and multiple P-frames. The second video stream includes multiple IP frame groups spaced at intervals of a specific number of video frames in DTS order. An IP frame group may include one I-frame and one or more P-frames. The number of P-frames is determined by the number of reference frames for the P-frames in the first video stream. For example, if the number of reference frames required for encoding and decoding the P-frames in the first video stream is one, then an IP frame group may include one I-frame and one P-frame. Alternatively, if the number of reference frames required for encoding and decoding the P-frames in the first video stream is two, then an IP frame group may include one I-frame and two P-frames.
[0077] As shown in Figure 4a, when the first video stream does not include B frames, the video frames in the IP frame group have the same DTS as the corresponding video frames in the first video stream, and the decoded picture content is also the same. Specifically, the I frames in the IP frame group have the same picture content as the P frames with the same DTS in the first video stream, and the P frames in the IP frame group have the same picture content as the P frames with the same DTS in the first video stream. All video frames in the IP frame group are arranged continuously. As can be seen from Figure 4a, the second video stream is not a complete video stream, but only multiple video clips. Therefore, the second video stream cannot be played normally, the video bit rate of the second video stream is relatively low, and the efficiency of encoding the second video stream is relatively high.
[0078] As can be seen, in this embodiment, IP frame groups can be encoded even when the first video stream does not include B frames. Furthermore, as can be seen from the structure of the IP frame group, the method for encoding the IP frame group is simple and the encoding efficiency is high. Setting the I-frames in the IP frame group to have the same screen content as the P-frames with the same DTS in the first video stream, and setting the P-frames in the IP frame group to have the same screen content as the P-frames with the same DTS in the first video stream, ensures that the video frames in the first and second video streams can be correctly decoded and played when they are sent to the terminal devices of live viewers.
[0079] It is worth mentioning that the second video stream has the same reference frame configuration for encoding and decoding as the first video stream. The reference frame configuration includes at least the number of reference frames for the P frames. For example, if the P frames in the first video stream are encoded and decoded with reference to the previous frame, then the P frames in the second video stream are also encoded and decoded with reference to the previous frame. Figure 4a illustrates the second and first video streams with a reference frame number of 1. The difference between the second video stream and the first video stream lies only in the reference relationship during encoding. That is, the video frames in the second video stream are encoded only with reference to the video frames in the second video stream, while the video frames in the first video stream are encoded only with reference to the video frames in the first video stream. During decoding, the video frames in the second video stream are decoded with reference to the video frames in the second video stream. There are cases where the video frames in the first video stream are decoded with reference to the video frames in the second video stream, which will be explained later.
[0080] As shown in FIG4 a , the interval between adjacent IP frame groups is less than the length of the GOP of the first video stream. The interval between adjacent IP frame groups can be represented by the interval of I frames between the two IP frame groups. In FIG4 a , the interval between adjacent IP frame groups is equal to the interval of I frames between the two IP frame groups, which is equal to 5 frames.
[0081] In one embodiment, considering that each GOP of the first video stream corresponds to one or more IP frame groups of the second video stream, it is also possible to set, for each GOP of the first video stream, the interval between the first I-frame in the GOP and the I-frame in the first IP frame group corresponding to the GOP, to be equal to the interval between adjacent IP frame groups, and the interval between the I-frame in the last IP frame group corresponding to the GOP and the last P-frame in the GOP to be equal to the interval between adjacent IP frame groups minus 1. As shown in FIG4a , the interval between the first frame (I-frame) of a GOP in the first video stream and the I-frame in the first IP frame group corresponding to the GOP is equal to the interval between adjacent IP frame groups, which is equal to 5 frames, and the interval between the I-frame in the last IP frame group corresponding to the GOP and the last P-frame in the GOP is 4 (not shown). Setting the IP frame groups in the above manner is equivalent to combining the I-frames in the IP frame groups with the I-frames in the first video stream to form a new, shorter GOP.
[0082] In one embodiment, real-time requirements for live broadcast are obtained. For example, the real-time requirements include a maximum allowable delay time for live broadcast images, such as a delay of 2 seconds. Generally, if one second contains 25 frames of images, it can be obtained that 2 seconds contain 50 frames of images, that is, the maximum allowable delay of live broadcast images is 50 frames. Based on this, the above real-time requirements can be met by setting the interval frame number between adjacent IP frame groups in the second video stream to be less than or equal to 50 frames, that is, the interval frame number between two I frames between adjacent IP frame groups is less than or equal to 50 frames.
[0083] In one embodiment, when the first video stream includes B frames, the I frames in the IP frame group have the same picture content as the P frames with the same DTS in the first video stream, the P frames in the IP frame group have the same picture content as the P frames with the same DTS in the first video stream, and the interval between two adjacent video frames in the IP frame group is N frames, where N is an integer greater than 0, and N is the number of B frames between two adjacent P frames in the first video stream.
[0084] Figure 4b is a schematic diagram of a first video stream and a second video stream provided by another embodiment of the present disclosure. In Figure 4b, n is a positive integer greater than or equal to 0, and the video frames are arranged in DTS order. In Figure 4b, when the first video stream includes B frames, the video frames with the same DTS in the first and second video streams have the same image content and image quality, such as resolution. Specifically, the I frames in the IP frame group have the same image content as the P frames with the same DTS in the first video stream, and the P frames in the IP frame group have the same image content as the P frames with the same DTS in the first video stream. Assuming that the IP frame group and the P frames in the first video stream are encoded and decoded with reference to the previous frame, each IP frame group consists of an I frame separated by a subsequent P frame. Comparing Figures 4a and 4b, it can be seen that the interval between two adjacent video frames in the IP frame group is N frames, where N is an integer greater than 0 and is the number of B frames between two adjacent P frames in the first video stream. In Figure 4b, the number of B frames is 1, so the interval between two adjacent video frames in the IP frame group is 1 frame. That is, the B frame does not need to be additionally encoded in the second video stream. The P frame in the IP frame group is coded and decoded with reference to the I frame in the IP frame group.
[0085] It can be seen that in this embodiment, when the first video stream includes B frames, the IP frame group can be encoded, and it can be seen from the structure of the IP frame group that the B frames do not need to be encoded, thereby saving the code stream of the IP frame group. In addition, the method for encoding the IP frame group is simple and the encoding efficiency is high. The I frames in the IP frame group are set to have the same picture content as the P frames with the same DTS in the first video stream, and the P frames in the IP frame group are set to have the same picture content as the P frames with the same DTS in the first video stream. This ensures that when the video frames in the first video stream and the second video stream are sent to the terminal devices of the live broadcast audience, the video frames can be correctly decoded and played.
[0086] As previously described, the number of P frames included in each IP frame group is the same as the number of reference frames for the P frames in the first video stream. Based on this, in one embodiment, when the number of reference frames for the P frames in the first video stream is 1, the IP frame group includes 1 I frame and 1 P frame. Therefore, when the number of reference frames for the P frames in the first video stream is 1, setting the IP frame group to include 1 I frame and 1 P frame can reduce the difficulty of generating the IP frame group and improve the efficiency of generating the IP frame group.
[0087] In one embodiment, when the number of reference frames for P frames in the first video stream is greater than one, the IP frame group includes one I frame and multiple P frames, and the number of P frames included in the IP frame group is the same as the number of reference frames for the P frames in the first video stream. Therefore, when the number of reference frames for P frames in the first video stream is greater than one, setting the number of P frames included in the IP frame group to be the same as the number of reference frames for the P frames in the first video stream enables the first and second video streams to be correctly decoded and played by the terminal device.
[0088] In Figures 4a and 4b above, the first and second video streams are illustrated using the example of the number of reference frames for P frames in the first video stream being 1. Figure 4c is a schematic diagram of the first and second video streams provided by another embodiment of the present disclosure. In Figure 4c, the first and second video streams are illustrated using the example of the first video stream not containing B frames and the number of reference frames for P frames in the first video stream being 2. It can be seen that in the second video stream, each IP frame group contains two P frames. Figure 4d is a schematic diagram of the first and second video streams provided by another embodiment of the present disclosure. In Figure 4d, the first and second video streams are illustrated using the example of the first video stream containing B frames and the number of reference frames for P frames in the first video stream being 2. It can be seen that in the second video stream, each IP frame group contains two P frames, and the interval between two adjacent video frames in the IP frame group is 1 frame. The number of B frames between two adjacent P frames in the first video stream is also 1 frame.
[0089] In practical applications, the following two methods can be used to encode the first video stream and the second video stream:
[0090] (1) The host encodes the collected video images to obtain a first video stream and a second video stream respectively, and uploads the first video stream and the second video stream to the server, which is the technical solution in Figure 3 above;
[0091] (2) The host encodes the collected video images to obtain a group of video streams with a smaller GOP, and uploads the group of video streams with a smaller GOP to the server. The server encodes the video streams to obtain a first video stream and a second video stream with a larger GOP. The GOP length of the first video stream is greater than the preset length.
[0092] In the above-mentioned various methods, the second video stream can be obtained by encoding the video screen on the host side, or by encoding the first video stream. For example, the first video stream is obtained by encoding the video screen on the host side, and the first video stream is directly encoded to obtain the second video stream, or the first video stream and the second video stream are respectively obtained by encoding the video screen on the host side.
[0093] In one embodiment, after generating the first video stream and the second video stream, the server may store the first video stream and the second video stream in any of the following ways:
[0094] Method 1: The first video stream and the second video stream are stored as two independent video streams respectively. The two video streams are aligned through the DTS of the video frames. The video frames with the same DTS have the same picture content and picture quality. The picture quality includes parameters such as resolution.
[0095] Method 2: Encode the video frames of the second video stream into the first video stream. For example, the H.264 / H.265 coding standard provides an SEI (Supplemental Enhancement Information) module, which can carry developer-defined information and can be encoded together with the video information. In this embodiment, the video frames in the second video stream are encoded in the SEI field, eliminating the need to store the second video stream separately. Since the live broadcast screen is subsequently pushed to the live broadcast audience, only one IP frame group needs to be pushed when the live broadcast audience enters the live broadcast room, and the push of the first video stream is subsequently switched. Therefore, under this storage method, only the information of one IP frame group needs to be read from the SEI module and sent to the live broadcast audience. The SEI information carried by the first video stream does not need to be sent to the live broadcast audience.
[0096] In the disclosed embodiment, after the first and second video streams are generated through the above process, step S304 is further performed to send the first and second video streams to the server, so that the server can transmit the live video to the live viewers based on the first and second video streams. The process of the server transmitting the live video to the live viewers based on the first and second video streams will be described later.
[0097] The above describes the process of the first terminal device generating and sending the first video stream and the second video stream. Based on the same technical concept, an embodiment of the present disclosure also provides a video processing method, which is applied to a server and executed by the server.
[0098] FIG5 is a flow chart of a video processing method provided by another embodiment of the present disclosure. As shown in FIG5 , the method is executed by a server and includes:
[0099] Step S502: Based on the video captured by the first terminal device, a first video stream and a second video stream are generated, wherein the first video stream includes multiple continuous GOPs, and the second video stream includes multiple IP frame groups spaced at intervals of a specific number of video frames; each GOP of the first video stream corresponds to one or more IP frame groups of the second video stream; each IP frame group includes an I frame and a P frame following the I frame; each video frame in each IP frame group has the same picture content as the video frame with the same DTS in the first video stream; the number of P frames included in each IP frame group is the same as the number of reference frames of the P frames in the first video stream.
[0100] In this embodiment, since the IP frame groups in the second video stream are spaced at intervals of a specific number of video frames, the second video stream is not a complete video stream. Therefore, when distributing the video, by encoding and sending the second video stream, the encoding resource consumption can be reduced, the sent video bit rate can be reduced, and the delay in pushing the live broadcast image to the terminal device can be reduced, thereby ensuring the real-time nature of the live broadcast.
[0101] According to the previous introduction to the first video stream and the second video stream, in one embodiment, when the first video stream does not include B frames, the I frames in the IP frame group have the same picture content as the P frames with the same DTS in the first video stream, the P frames in the IP frame group have the same picture content as the P frames with the same DTS in the first video stream, and all video frames in the IP frame group are set continuously.
[0102] In one embodiment, when the first video stream includes B frames, the I frame in the IP frame group has the same picture content as the P frame with the same DTS in the first video stream, the P frame in the IP frame group has the same picture content as the P frame with the same DTS in the first video stream, and the interval between two adjacent video frames in the IP frame group is M frames; M is the number of B frames between two adjacent P frames in the first video stream, and M is an integer greater than 0.
[0103] In one embodiment, when the number of reference frames of the P frame in the first video stream is 1, the IP frame group includes 1 I frame and 1 P frame.
[0104] In one embodiment, when the number of reference frames of the P frame in the first video stream is greater than 1, the IP frame group includes 1 I frame and multiple P frames, and the number of P frames included in the IP frame group is the same as the number of reference frames of the P frame in the first video stream.
[0105] For the specific structures of the first video stream and the second video stream, please refer to the previous description and will not be repeated here.
[0106] FIG6 is a flow chart of a video processing method provided by another embodiment of the present disclosure. As shown in FIG6 , based on FIG5 , the flow further includes:
[0107] Step S504: receiving a live data push request from the second terminal device;
[0108] Step S506: In response to the live data push request, a target IP frame group and a target playback stream are sent to the second terminal device. The target IP frame group is an IP frame group in which the DTS in the second video stream is located before the receiving time of the live data push request and the time interval between the DTS and the receiving time is within a preset time length. The target playback stream is a video frame in which the DTS in the first video stream is located after the last P frame in the target IP frame group and starts from the P frame.
[0109] In step S504, the server receives a live data push request from a second terminal device. The second terminal device may be a terminal device of a live broadcast viewer. Specifically, when a live broadcast viewer enters the live broadcast room, the second terminal device generates a live data push request based on the live broadcast viewer's entry into the live broadcast room and sends the live data push request to the server, thereby receiving the live data push request sent by the second terminal device.
[0110] In step S506, the server responds to the live data push request and, based on the time the live data push request was received, determines a target IP frame group in the second video stream and pushes the target IP frame group to the second terminal device. Furthermore, the server determines a target playback stream in the first video stream and pushes the target playback stream to the second terminal device. Since the time the server receives the live data push request, the time the second terminal device sends the live data push request, and the time the live viewer enters the live broadcast room are substantially the same, in other embodiments, the server may also determine a target IP frame group in the second video stream and push the target IP frame group to the terminal device based on the time the second terminal device sends the live data push request or the time the live viewer enters the live broadcast room, and further, determine a target playback stream in the first video stream and push the target playback stream to the second terminal device.
[0111] The target IP frame group is an IP frame group in which the DTS of the second video stream is located before the time at which the live data push request is received, and the time interval between the DTS and the receiving time is within a preset time length. Specifically, the target IP frame group is an IP frame group containing I frames in which the DTS of the second video stream is located before the time at which the live data push request is received, and the time interval between the DTS and the receiving time is within a preset time length. Since the DTS of the target IP frame group is located before the time at which the live data push request is received, and the time interval between the DTS and the receiving time is within a preset time length, for example, within 1 millisecond, the time delay of the live broadcast image can be ensured to be small, thereby meeting the live viewing needs of live viewers.
[0112] In one embodiment, the target IP frame group is an IP frame group in the second video stream whose DTS is before the time when the live data push request is received and whose DTS is closest to the time when the live data push request is received. Specifically, the target IP frame group is an IP frame group in which an I frame in the second video stream whose DTS is before the time when the live data push request is received and whose DTS is closest to the time when the live data push request is received.
[0113] The IP frame group containing the I frame whose DTS is before the above-mentioned receiving time and closest to the receiving time can be determined in the second video stream, and the determined IP frame group is used as the target IP frame group. Since the target IP frame group is the IP frame group whose DTS is before the receiving time of the live data push request and closest to the receiving time in the second video stream, the time delay of the live picture can be minimized to the greatest extent, thereby improving the live viewing experience of the live audience. Since the video frames in the IP frame group have the same picture content as the corresponding video frames in the first video stream, the effect of pushing the live picture is also achieved by pushing the target IP frame group.
[0114] In addition to pushing the target IP frame group, the target playback stream is also pushed. The target playback stream is the video frame in the first video stream whose DTS is located after the last P frame in the target IP frame group and starts from the P frame. In one embodiment, the order in which the video frames are pushed to the terminal device is the same as the DTS order of the video frames. Therefore, the first video stream and the second video stream are arranged in the order of DTS. In this arrangement, the first P frame whose DTS is located after the target IP frame group is determined in the first video stream, and the video stream starting from and including the first P frame is pushed to the second terminal device as the target playback stream. It can be seen that through this embodiment, after the target IP frame group is pushed, it is possible to switch to the first video stream to continue pushing the live picture to the second terminal device, thereby taking advantage of the larger GOP of the first video stream, reducing the video bit rate, and improving the video transmission efficiency.
[0115] In one embodiment, the above method further includes:
[0116] Determine whether the I frame whose DTS is before the receiving time and closest to the receiving time is in the first video stream or in the second video stream;
[0117] If the most recent I frame is located in the second video stream, the target IP frame group and the target playback stream are sent to the second terminal device in response to the live data push request.
[0118] Specifically, after determining the reception time of the live data push request, it is determined whether the I-frame with the DTS preceding the reception time and closest to the reception time is located in the first video stream or in the second video stream. For example, if the reception time is 10:00:00, and the DTS of the first video stream has an I-frame preceding the reception time, specifically 10:00:03, and the DTS of the second video stream has an I-frame preceding the reception time, specifically 10:00:01, then it can be determined that the I-frame with the DTS preceding the reception time and closest to the reception time is located in the second video stream.
[0119] Of course, on the contrary, if the receiving time is 10:00:00, the DTS with an I frame in the first video stream is located before the receiving time, specifically 10:00:01, and the DTS with an I frame in the second video stream is located before the receiving time, specifically 10:00:03, then it can be determined that the I frame with DTS located before the receiving time and closest to the receiving time is located in the first video stream.
[0120] It is understandable that when pushing live images to the second terminal device, it is necessary to start pushing from the I frame to ensure that the live images are correctly decoded and played. Therefore, after determining that the I frame with a DTS that is before the reception time and closest to the reception time is located in the second video stream, the step of sending the target IP frame group and the target playback stream to the second terminal device in response to the live data push request is executed. Of course, if it is determined that the I frame with a DTS that is before the reception time and closest to the reception time is located in the first video stream, the first video stream can be pushed directly to the second terminal device starting from this closest I frame in the first video stream.
[0121] The following describes in detail the process of determining the target IP frame group and the target playback stream, pushing the target IP frame group and the target playback stream to the second terminal device through specific drawings.
[0122] FIG7a is a schematic diagram of pushing a first video stream and a second video stream according to an embodiment of the present disclosure. In FIG7a , n is a positive integer greater than or equal to 0, and each video frame is arranged in the order of the DTS. As shown in FIG7a , when the first video stream does not include B frames and the number of reference frames of the P frame is 1, for example, when the time when the server receives the live data push request from the second terminal device corresponds to the DTS of the n+13 video frame, for example, the DTS of the n+13 video frame is 0 minutes and 15 seconds, and the time when the request is received is 0 minutes and 16 seconds, that is, the n+13 video frame is the video frame whose DTS is before the time of receiving the request and is closest to the time of receiving the request. Through the above process, the I frame whose DTS is before the time of receiving the request and is closest to the time of receiving the request is found in the second video stream, thereby determining that the IP frame group containing the closest I frame is the target IP frame group, and the target IP frame group includes the IP frame group consisting of the n+10 I frame and the n+11 P frame. Thus, the IP frame group consisting of the n+10 I frame and the n+11 P frame is pushed to the second terminal device in sequence, and after the IP frame group is pushed, the live picture is continued to be pushed to the second terminal device starting from the n+12 P frame in the first video stream. It can be seen that through this embodiment, there is only a delay of 3 frames, namely the three frames n+10, n+11, and n+12. The n+13 frame is the first frame that the live audience should theoretically see after entering the live broadcast room, thereby ensuring the real-time nature of the live broadcast.
[0123] FIG7 b is a schematic diagram of pushing a first video stream and a second video stream according to another embodiment of the present disclosure. In FIG7 b , n is a positive integer greater than or equal to 0, and each video frame is arranged in the order of the DTS. As shown in FIG7 b , when the first video stream includes B frames, the number of reference frames of the P frames is 1, and the number of B frames is 1, for example, when the server receives the live data push request from the second terminal device at a time corresponding to the DTS of the n+16 video frame, for example, the DTS of the n+16 video frame is 0 minutes and 15 seconds, and the time of receiving the request is 0 minutes and 16 seconds, that is, the n+16 video frame is the video frame whose DTS is before the time of receiving the request and is closest to the time of receiving the request. Through the above process, the I frame whose DTS is before the time of receiving the request and is closest to the time of receiving the request is found in the second video stream, thereby determining that the IP frame group containing the closest I frame is the target IP frame group, and the target IP frame group includes the IP frame group consisting of the n+11 I frame and the n+13 P frame. Thus, the IP frame group consisting of the n+11 I frame and the n+13 P frame is pushed to the second terminal device in sequence, and after the IP frame group is pushed, since the n+14 frame in the first video stream is a B frame, the live broadcast picture is continued to be pushed to the second terminal device starting from the n+15 P frame in the first video stream. It can be seen that through this embodiment, when there is a B frame in the first video stream, the delay of the live broadcast picture can also be reduced to ensure the real-time nature of the live broadcast.
[0124] FIG7c is a schematic diagram of pushing a first video stream and a second video stream according to another embodiment of the present disclosure. In FIG7c, n is a positive integer greater than or equal to 0, and each video frame is arranged in the order of the DTS. As shown in FIG7c, when the first video stream does not include B frames, the number of reference frames of the P frame is 2. For example, when the time when the server receives the live data push request from the second terminal device corresponds to the DTS of the n+13 video frame, for example, the DTS of the n+13 video frame is 0 minutes and 15 seconds, and the time when the request is received is 0 minutes and 16 seconds, that is, the n+13 video frame is the video frame whose DTS is before the time of receiving the request and is closest to the time of receiving the request. Through the above process, the I frame whose DTS is before the time of receiving the request and is closest to the time of receiving the request is found in the second video stream, and the IP frame group containing the closest I frame is determined to be the target IP frame group. The target IP frame group includes the IP frame group consisting of the n+10 I frame, the n+11 P frame, and the n+12 P frame. Thus, the IP frame group consisting of the n+10 I frame, the n+11 P frame and the n+12 P frame is pushed to the second terminal device in sequence, and after the IP frame group is pushed, the live broadcast picture continues to be pushed to the second terminal device starting from the n+13 P frame in the first video stream. It can be seen that through this embodiment, there is only a delay of 3 frames, namely the three frames n+10, n+11, and n+12. The n+13 frame is the first frame that the live audience should theoretically see after entering the live broadcast room, thereby ensuring the real-time nature of the live broadcast.
[0125] FIG7 d is a schematic diagram of pushing a first video stream and a second video stream according to another embodiment of the present disclosure. In FIG7 d , n is a positive integer greater than or equal to 0, and each video frame is arranged in the order of the DTS. As shown in FIG7 d , when the first video stream includes B frames, the number of reference frames of the P frame is 2, and the number of B frames is 1, for example, when the server receives the live data push request from the second terminal device at a time corresponding to the DTS of the n+16 video frame, for example, the DTS of the n+16 video frame is 0 minutes and 15 seconds, and the time of receiving the request is 0 minutes and 16 seconds, that is, the n+16 video frame is the video frame whose DTS is before the time of receiving the request and is closest to the time of receiving the request. Through the above process, the I frame whose DTS is before the time of receiving the request and is closest to the time of receiving the request is found in the second video stream, thereby determining that the IP frame group containing the closest I frame is the target IP frame group. The target IP frame group includes the IP frame group consisting of the n+11 I frame, the n+13 P frame, and the n+15 P frame. Thus, the IP frame group consisting of the n+11 I frame, the n+13 P frame and the n+15 P frame is pushed to the second terminal device in sequence, and after the IP frame group is pushed, since the n+16 frame in the first video stream is a B frame, the live broadcast picture is continued to be pushed to the second terminal device starting from the n+17 P frame in the first video stream. It can be seen that through this embodiment, when there are B frames in the first video stream, the delay of the live broadcast picture can also be reduced to ensure the real-time nature of the live broadcast.
[0126] After receiving the video frame pushed by the server, the second terminal device needs to decode and play it. Based on this, Figure 8 is a flow chart of a video processing method provided by another embodiment of the present disclosure. This method is applied to the above-mentioned second terminal device, that is, the terminal device of the live broadcast audience, and is executed by the second terminal device. As shown in Figure 8, the method includes:
[0127] Step S802: Send a live data push request to the server;
[0128] Step S804: Receive a target IP frame group and a target playback stream from the server. The target IP frame group is an IP frame group in the second video stream whose DTS is before the reception time of the live data push request and whose time interval between the DTS and the reception time is within a preset time length. The target playback stream is a video frame in the first video stream whose DTS is after the last P frame in the target IP frame group and starts with the P frame.
[0129] Step S806: Decode the target IP frame group and the target playback stream.
[0130] In this embodiment, when distributing the video, by sending the target IP frame group and the target playback stream, the encoding resource consumption can be reduced, the sent video bit rate can be reduced, and the delay in pushing the live image to the terminal device can be reduced, thereby ensuring the real-time nature of the live broadcast.
[0131] In step S802, the second terminal device sends a live broadcast data push request to the server. Specifically, after the second terminal device determines that the user has entered the live broadcast room, it determines the user as a live broadcast viewer, generates a live broadcast data push request, and sends it to the server.
[0132] In the above step S804, the second terminal device receives the target IP frame group and the target playback stream from the server. The contents of the target IP frame group and the target playback stream can be referred to the above introduction and will not be repeated here.
[0133] In the above step S806, the second terminal device decodes the target IP frame group and the target playback stream.
[0134] In one embodiment, decoding the target IP frame group and the target playback stream includes:
[0135] When the target playback stream does not include B frames, the number of reference frames in the target playback stream is 1, and the first video frame in the target playback stream is not an I frame, the first P frame in the target playback stream is decoded with reference to the P frames in the target IP frame group, and the P frames in the target IP frame group are decoded with reference to the I frames in the target IP frame group.
[0136] Specifically, when the target playback stream does not include B frames and the number of reference frames in the target playback stream is 1, the IP frame group includes one I frame and one P frame. The first video frame in the target playback stream is not an I frame but a P frame. In this case, because the video frames with the same DTS in the first and second video streams have the same picture content, the first P frame in the target playback stream is decoded with reference to the P frame in the target IP frame group. Furthermore, based on the reference relationship, the P frame in the target IP frame group is decoded with reference to the I frame in the target IP frame group.
[0137] As shown in Figure 7a, when the target playback stream does not include B frames and the number of reference frames in the target playback stream is 1, it can be seen that the IP frame group includes 1 I frame and 1 P frame, and the first video frame in the target playback stream is not an I frame. In this case, the n+12 P frame is used as the first P frame in the target playback stream, and is decoded with reference to the P frame in the target IP frame group, that is, the n+11 P frame. The P frame in the target IP frame group, that is, the n+11 P frame, is decoded with reference to the I frame in the target IP frame group, that is, the n+10 I frame. When the second terminal device plays the live broadcast image, it first plays the n+10 I frame and the n+11 P frame in the target IP frame group, and then starts playing the target playback stream from the n+12 P frame in the target playback stream.
[0138] It can be seen that through this embodiment, when the target playback stream does not include B frames, the number of reference frames in the target playback stream is 1, and the first video frame in the target playback stream is not an I frame, normal decoding and playback of the target playback stream and the target IP frame group can be achieved, thereby ensuring the live viewing experience of the live audience.
[0139] In one embodiment, decoding the target IP frame group and the target playback stream includes:
[0140] When the target playback stream does not include B frames, and the number of reference frames in the target playback stream is T, and the first video frame in the target playback stream is not an I frame, the first P frame in the target playback stream is decoded with reference to the first P frame to the Tth P frame in the target IP frame group, the Sth P frame in the target IP frame group is decoded with reference to the first S-1 P frames in the target IP frame group and the I frame in the target IP frame group, and the first P frame in the target IP frame group is decoded with reference to the I frame in the target IP frame group; wherein T is an integer greater than 1, and S is an integer greater than or equal to 2 and less than or equal to T.
[0141] Specifically, when the target playback stream does not include B frames and the number of reference frames in the target playback stream is T, the IP frame group includes one I frame and T P frames. T is an integer greater than 1. The first video frame in the target playback stream is not an I frame but a P frame. In this case, since the video frames with the same DTS in the first video stream and the second video stream have the same picture content, the first P frame in the target playback stream is decoded with reference to the first P frame to the Tth P frame in the target IP frame group when it is necessary to decode with reference to T P frames, and, based on the reference relationship, the Sth P frame in the target IP frame group is decoded with reference to the first S-1 P frames in the target IP frame group and the I frame in the target IP frame group, and the first P frame in the target IP frame group is decoded with reference to the I frame in the target IP frame group, where S is an integer greater than or equal to 2 and less than or equal to T. For example, when T is equal to 3, the first P frame in the target playback stream is decoded with reference to the first P frame to the third P frame in the target IP frame group, the third P frame in the target IP frame group is decoded with reference to the first two P frames in the target IP frame group and the I frame in the target IP frame group, the second P frame in the target IP frame group is decoded with reference to the first P frame in the target IP frame group and the I frame in the target IP frame group, and the first P frame in the target IP frame group is decoded with reference to the I frame in the target IP frame group.
[0142] As shown in Figure 7c, using T equal to 2 as an example, when the target playback stream does not include B frames and the number of reference frames in the target playback stream is 2, it can be seen that the IP frame group includes 1 I frame and two P frames, and the first video frame in the target playback stream is not an I frame. In this case, the P frame n+13 is used as the first P frame in the target playback stream and is decoded with reference to the two P frames in the target IP frame group, namely the P frame n+11 and the P frame n+12. The P frame n+12 in the target IP frame group is decoded with reference to the P frame n+11 and the I frame n+10 in the target IP frame group. The P frame n+11 in the target IP frame group is decoded with reference to the I frame in the target IP frame group, namely the I frame n+10. When the second terminal device plays the live video, it first plays the I frame n+10, the P frame n+11, and the P frame n+12 in the target IP frame group, and then starts playing the target playback stream from the P frame n+13 in the target playback stream.
[0143] It can be seen that through this embodiment, when the target playback stream does not include B frames, the number of reference frames in the target playback stream is T, and the first video frame in the target playback stream is not an I frame, the target playback stream and the target IP frame group can be decoded and played normally, thereby ensuring the live viewing experience of the live audience.
[0144] In one embodiment, decoding the target IP frame group and the target playback stream includes:
[0145] When the target playback stream includes B frames, and the number of reference frames in the target playback stream is 1, and the first video frame in the target playback stream is not an I frame, the first P frame in the target playback stream is decoded with reference to the P frame in the target IP frame group, and the P frame in the target IP frame group is decoded with reference to the I frame in the target IP frame group.
[0146] Specifically, when the target playback stream includes B frames and the number of reference frames in the target playback stream is 1, the IP frame group includes one I frame and one P frame, and the number of video frames between the I frame and the P frame is equal to the number of B frames. The first video frame in the target playback stream is not an I frame but a P frame. In this case, since the video frames with the same DTS in the first and second video streams have the same picture content, the first P frame in the target playback stream is decoded with reference to the P frame in the target IP frame group, and based on the reference relationship, the P frame in the target IP frame group is decoded with reference to the I frame in the target IP frame group.
[0147] As shown in Figure 7b, when the target playback stream includes B frames and the number of reference frames in the target playback stream is 1, taking the number of B frames as an example, it can be seen that the IP frame group includes 1 I frame and 1 P frame, with one video frame between the I frame and the P frame, and the first video frame in the target playback stream is a P frame rather than an I frame. In this case, the P frame n+15 is used as the first P frame in the target playback stream and is decoded with reference to the P frame in the target IP frame group, that is, the P frame n+13. The P frame in the target IP frame group, that is, the P frame n+13, is decoded with reference to the I frame in the target IP frame group, that is, the I frame n+11.
[0148] However, since the video frames pushed to the terminal device lack B frames n+12 and n+14, and considering that the original correct playback order of the first video stream is n+12-n+11-n+14-n+13-n+16-n+15, this embodiment directly starts playing from the P frame n+13 in the IP frame group, that is, plays the last P frame in the target IP frame group, and then plays the B frame n+16 and the P frame n+15 in the first video stream. For the n+16 B frame in the first video stream, it should refer to the n+13 P frame and the n+15 P frame in the first video stream. Since the picture content of the n+13 P frame is the same in the first video stream and the second video stream, the n+16 B frame in the first video stream can be decoded with reference to the n+13 P frame in the second video stream and the n+15 P frame in the first video stream. That is, when the B frame in the target playback stream needs to be decoded with reference to the P frame in the first video stream, if the referenced P frame in the first video stream is not sent to the second terminal device, the second terminal device uses the P frame with the same DTS in the target IP frame group to decode the B frame in the target playback stream.
[0149] It can be seen that through this embodiment, when the target playback stream includes B frames, the number of reference frames in the target playback stream is 1, and the first video frame in the target playback stream is not an I frame, each video frame in the target playback stream and the target IP frame group can be decoded and the live picture can be played, thereby ensuring the live viewing experience of the live audience.
[0150] In one embodiment, decoding the target IP frame group and the target playback stream includes:
[0151] When the target playback stream includes B frames, and the number of reference frames in the target playback stream is X, and the first video frame in the target playback stream is not an I frame, the first P frame in the target playback stream is decoded with reference to the first P frame to the Xth P frame in the target IP frame group, the Yth P frame in the target IP frame group is decoded with reference to the first Y-1 P frames in the target IP frame group and the I frame in the target IP frame group, and the first P frame in the target IP frame group is decoded with reference to the I frame in the target IP frame group; wherein X is an integer greater than 1, and Y is an integer greater than or equal to 2 and less than or equal to X.
[0152] Specifically, when the target playback stream includes B frames and the number of reference frames in the target playback stream is X, the IP frame group includes one I frame and X P frames, and the number of interval frames between two adjacent video frames in the IP frame group is equal to the number of B frames, where X is an integer greater than 1. The first video frame in the target playback stream is not an I frame but a P frame. In this case, since the video frames with the same DTS in the first and second video streams have the same picture content, the first P frame in the target playback stream is decoded with reference to the first P frame to the Xth P frame in the target IP frame group when decoding with reference to X P frames is required. Moreover, based on the reference relationship, the Yth P frame in the target IP frame group is decoded with reference to the first Y-1 P frames in the target IP frame group and the I frame in the target IP frame group, and the first P frame in the target IP frame group is decoded with reference to the I frame in the target IP frame group, where Y is an integer greater than or equal to 2 and less than or equal to X. For example, when X is equal to 3, the first P frame in the target playback stream is decoded with reference to the first to third P frames in the target IP frame group, the third P frame in the target IP frame group is decoded with reference to the first two P frames in the target IP frame group and the I frame in the target IP frame group, the second P frame in the target IP frame group is decoded with reference to the first P frame in the target IP frame group and the I frame in the target IP frame group, and the first P frame in the target IP frame group is decoded with reference to the I frame in the target IP frame group.
[0153] As shown in Figure 7d, X is equal to 2 as an example. When the target playback stream includes B frames and the number of reference frames in the target playback stream is 2, taking the number of B frames as 1 as an example, it can be seen that the IP frame group includes 1 I frame and two P frames, and there is a video frame between the I frame and the P frame, and the first video frame in the target playback stream is not an I frame. In this case, the n+17 P frame is used as the first P frame in the target playback stream, and is decoded with reference to the two P frames in the target IP frame group, namely the n+13 P frame and the n+15 P frame. The n+15 P frame in the target IP frame group is decoded with reference to the n+13 P frame and the n+11 I frame in the target IP frame group. The n+13 P frame in the target IP frame group is decoded with reference to the I frame in the target IP frame group, namely the n+11 I frame.
[0154] However, since the video frames pushed to the terminal device lack B frames n+12, n+14 and n+16, and considering that the original correct playback order of the first video stream is n+12-n+11-n+14-n+13-n+16-n+15-n+18-n+17, this embodiment directly starts playing from the P frame n+15 in the IP frame group, that is, plays the last P frame in the target IP frame group, and then plays the B frame n+18 and the P frame n+17 in the first video stream.
[0155] For the B frame n+18 in the first video stream, it should refer to the P frame n+15 and the P frame n+17 in the first video stream. Since the picture content of the P frame n+15 is the same in the first video stream and the second video stream, the B frame n+18 in the first video stream can be decoded with reference to the P frame n+15 in the second video stream and the P frame n+17 in the first video stream. That is, when the B frame in the target playback stream needs to be decoded with reference to the P frame in the first video stream, if the referenced P frame in the first video stream is not sent to the second terminal device, the second terminal device uses the P frame with the same DTS in the target IP frame group to decode the B frame in the target playback stream.
[0156] It can be seen that through this embodiment, when the target playback stream includes B frames, the number of reference frames in the target playback stream is X, and the first video frame in the target playback stream is not an I frame, the video frames in the target playback stream and the target IP frame group can be decoded and the live screen can be played, thereby ensuring the live viewing experience of the live audience.
[0157] In summary, the various embodiments of the present disclosure provide a video processing method that, when encoding a first video stream, encodes a second video stream consisting solely of IP frame groups. At the initial stage of a live broadcast, when viewers enter the live broadcast room, the live broadcast delay caused by directly sending the first video stream is reduced by sending the IP frame groups. Furthermore, when sending the IP frame groups, the amount of additional data that needs to be sent is not determined by the GOP in the first video stream, but rather by the position at which the IP frame groups are inserted in the first video stream. Therefore, by increasing the GOP of the first video stream, the video bit rate can be reduced, thereby lowering the transmission cost of the live broadcast and the network bandwidth requirement.
[0158] Furthermore, since the second video stream is incomplete, its data size is minimal and does not consume excessive encoding resources. By switching from the IP frame group to the first video stream, the viewing experience of live viewers during the initial broadcast phase is not degraded, thus improving the live broadcast experience. In experimental tests, the GOP length of the first video stream can be 10 seconds, reducing the video bitrate by approximately 10%. The bitrate of the second video stream is only one-tenth of the bitrate of the first video stream.
[0159] The present disclosure further provides a video processing device. FIG9 is a schematic structural diagram of a video processing device provided by an embodiment of the present disclosure. As shown in FIG9 , the video processing device includes:
[0160] A first generating unit 901 is configured to generate, by the first terminal device, a first video stream and a second video stream based on a video captured by a first terminal device, wherein the first video stream includes a plurality of consecutive GOPs, and the second video stream includes a plurality of intra-frame coded image frames and predictive coded image frame IP frame groups arranged at intervals of a specific number of video frames; each GOP of the first video stream corresponds to one or more IP frame groups of the second video stream; each IP frame group includes an intra-frame coded image frame I frame and a predictive coded image frame P frame following the I frame; each video frame in each IP frame group has the same picture content as a video frame having the same decoding timestamp DTS in the first video stream; and the number of P frames included in each IP frame group is the same as the number of reference frames of the P frames in the first video stream;
[0161] The first sending unit 902 is configured to send the first video stream and the second video stream to a server.
[0162] Optionally, when the first video stream does not include bidirectionally predictive coded image frames B frames, the I frames in the IP frame group have the same picture content as the P frames with the same decoding timestamp DTS in the first video stream, the P frames in the IP frame group have the same picture content as the P frames with the same decoding timestamp DTS in the first video stream, and all video frames in the IP frame group are set continuously.
[0163] Optionally, when the first video stream includes bidirectionally predictive coded image frames B frames, the I frames in the IP frame group have the same picture content as the P frames with the same decoding timestamp DTS in the first video stream, the P frames in the IP frame group have the same picture content as the P frames with the same decoding timestamp DTS in the first video stream, and the interval between two adjacent video frames in the IP frame group is N frames; N is the number of B frames between two adjacent P frames in the first video stream.
[0164] Optionally, when the number of reference frames of the P frame in the first video stream is 1, the IP frame group includes 1 I frame and 1 P frame.
[0165] Optionally, when the number of reference frames of the P frame in the first video stream is greater than 1, the IP frame group includes 1 I frame and multiple P frames, and the number of P frames included in the IP frame group is the same as the number of reference frames of the P frame in the first video stream.
[0166] The video processing device in the embodiment of the present disclosure can implement the various processes of the above-mentioned video processing method embodiment applied to the first terminal device and achieve the same effects and functions, which will not be repeated here.
[0167] The present disclosure further provides a video processing device. FIG10 is a schematic structural diagram of a video processing device provided by another embodiment of the present disclosure. As shown in FIG10 , the video processing device includes:
[0168] The second generation unit 1001 is used to generate a first video stream and a second video stream based on the video captured by the first terminal device, wherein the first video stream includes multiple continuous GOPs, and the second video stream includes multiple intra-frame coded image frames and predictive coded image frame IP frame groups arranged at intervals of a specific number of video frames; each GOP of the first video stream corresponds to one or more IP frame groups of the second video stream; each IP frame group includes an intra-frame coded image frame I frame and a predictive coded image frame P frame after the I frame; each video frame in each IP frame group has the same picture content as the video frame with the same decoding timestamp DTS in the first video stream; the number of P frames included in each IP frame group is the same as the number of reference frames of the P frames in the first video stream.
[0169] Optionally, the device further comprises:
[0170] A request receiving unit, configured to receive a live broadcast data push request from a second terminal device;
[0171] A video sending unit is used to send a target IP frame group and a target playback stream to the second terminal device in response to a live data push request, wherein the target IP frame group is an IP frame group in the second video stream whose DTS is located before the reception time of the live data push request and the time interval between the DTS and the reception time is within a preset time length; and the target playback stream is a video frame in the first video stream whose DTS is located after the last P frame in the target IP frame group and starts from the P frame.
[0172] Optionally, the target IP frame group is an IP frame group in the second video stream whose DTS is before the receiving time and whose DTS is closest to the receiving time.
[0173] Optionally, when the first video stream does not include bidirectionally predictive coded image frames B frames, the I frames in the IP frame group have the same picture content as the P frames with the same decoding timestamp DTS in the first video stream, the P frames in the IP frame group have the same picture content as the P frames with the same decoding timestamp DTS in the first video stream, and all video frames in the IP frame group are set continuously.
[0174] Optionally, when the first video stream includes bidirectionally predictive coded image frames B frames, the I frames in the IP frame group have the same picture content as the P frames with the same decoding timestamp DTS in the first video stream, the P frames in the IP frame group have the same picture content as the P frames with the same decoding timestamp DTS in the first video stream, and the interval between two adjacent video frames in the IP frame group is M frames; M is the number of B frames between two adjacent P frames in the first video stream.
[0175] Optionally, when the number of reference frames of the P frame in the first video stream is 1, the IP frame group includes 1 I frame and 1 P frame.
[0176] Optionally, when the number of reference frames of the P frame in the first video stream is greater than 1, the IP frame group includes 1 I frame and multiple P frames, and the number of P frames included in the IP frame group is the same as the number of reference frames of the P frame in the first video stream.
[0177] The video processing device in the embodiment of the present disclosure can implement each process of the above-mentioned video processing method embodiment applied to the server and achieve the same effects and functions, which will not be repeated here.
[0178] The present disclosure further provides a video processing device. FIG11 is a structural diagram of a video processing device provided by another embodiment of the present disclosure. As shown in FIG11 , the video processing device includes:
[0179] The request sending unit 1101 is used to send a live data push request to the server;
[0180] The data receiving unit 1102 is configured to receive a target IP frame group and a target playback stream from the server, wherein the target IP frame group is an IP frame group in the second video stream whose decoding timestamp DTS is before the reception time of the live data push request and the time interval between the DTS and the reception time is within a preset time length; and the target playback stream is a video frame in the first video stream whose DTS is after the last predictively coded image frame P frame in the target IP frame group and starts with the P frame;
[0181] The data decoding unit 1103 is configured to decode the target IP frame group and the target playback stream.
[0182] Optionally, decoding the target IP frame group and the target playback stream includes:
[0183] When the target playback stream does not include a bidirectionally predictive coded image frame B frame, the number of reference frames in the target playback stream is 1, and the first video frame in the target playback stream is not an I frame, the first P frame in the target playback stream is decoded with reference to the P frame in the target IP frame group, and the P frame in the target IP frame group is decoded with reference to the I frame in the target IP frame group.
[0184] Optionally, decoding the target IP frame group and the target playback stream includes:
[0185] When the target playback stream does not include a bidirectionally predictive coded image frame B frame, and the number of reference frames in the target playback stream is T, and the first video frame in the target playback stream is not an I frame, the first P frame in the target playback stream is decoded with reference to the first P frame to the Tth P frame in the target IP frame group, the Sth P frame in the target IP frame group is decoded with reference to the first S-1 P frames in the target IP frame group and the I frame in the target IP frame group, and the first P frame in the target IP frame group is decoded with reference to the I frame in the target IP frame group; wherein, T is an integer greater than 1, and S is an integer greater than or equal to 2 and less than or equal to T.
[0186] Optionally, decoding the target IP frame group and the target playback stream includes:
[0187] When the target playback stream includes bidirectionally predictive coded image frames B frames, the number of reference frames in the target playback stream is 1, and the first video frame in the target playback stream is not an I frame, the first P frame in the target playback stream is decoded with reference to the P frames in the target IP frame group, and the P frames in the target IP frame group are decoded with reference to the I frames in the target IP frame group.
[0188] Optionally, decoding the target IP frame group and the target playback stream includes:
[0189] When the target playback stream includes bidirectionally predictive coded image frames B frames, the number of reference frames in the target playback stream is X, and the first video frame in the target playback stream is not an I frame, the first P frame in the target playback stream is decoded with reference to the first P frame to the Xth P frame in the target IP frame group, the Yth P frame in the target IP frame group is decoded with reference to the first Y-1 P frames in the target IP frame group and the I frame in the target IP frame group, and the first P frame in the target IP frame group is decoded with reference to the I frame in the target IP frame group; wherein X is an integer greater than 1, and Y is an integer greater than or equal to 2 and less than or equal to X.
[0190] The video processing device in the embodiment of the present disclosure can implement the various processes of the above-mentioned video processing method embodiment applied to the second terminal device and achieve the same effects and functions, which will not be repeated here.
[0191] An embodiment of the present disclosure also provides an electronic device. FIG12 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. As shown in FIG12 , the electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors 1201 and memory 1202. One or more applications or data may be stored in the memory 1202. Among them, the memory 1202 may be a temporary storage or a persistent storage. The application stored in the memory 1202 may include one or more modules (not shown in the figure), each module may include a series of computer-executable instructions in the electronic device. Furthermore, the processor 1201 may be configured to communicate with the memory 1202 and execute a series of computer-executable instructions in the memory 1202 on the electronic device. The electronic device may also include one or more power supplies 1203, one or more wired or wireless network interfaces 1204, one or more input or output interfaces 1205, one or more keyboards 1206, etc.
[0192] In a specific embodiment, the electronic device includes a processor; and a memory configured to store computer-executable instructions, wherein when the computer-executable instructions are executed, the processor implements the following process:
[0193] Based on a video captured by a first terminal device, a first video stream and a second video stream are generated by the first terminal device, wherein the first video stream includes a plurality of consecutive GOPs, and the second video stream includes a plurality of intra-frame coded image frames and predictive coded image frame IP frame groups arranged at intervals of a specific number of video frames; each GOP of the first video stream corresponds to one or more IP frame groups of the second video stream; each IP frame group includes an intra-frame coded image frame I frame and a predictive coded image frame P frame following the I frame; each video frame in each IP frame group has the same picture content as a video frame having the same decoding timestamp DTS in the first video stream; and the number of P frames included in each IP frame group is the same as the number of reference frames of the P frames in the first video stream;
[0194] The first video stream and the second video stream are sent to a server.
[0195] The electronic device in the embodiment of the present disclosure can implement the various processes of the above-mentioned video processing method embodiment applied to the first terminal device and achieve the same effects and functions, which will not be repeated here.
[0196] In another specific embodiment, an electronic device includes a processor; and a memory configured to store computer-executable instructions, wherein when the computer-executable instructions are executed, the processor implements the following process:
[0197] Based on the video captured by the first terminal device, a first video stream and a second video stream are generated, wherein the first video stream includes multiple continuous GOPs, and the second video stream includes multiple intra-frame coded image frames and predictive coded image frame IP frame groups arranged at intervals of a specific number of video frames; each GOP of the first video stream corresponds to one or more IP frame groups of the second video stream; each IP frame group includes an intra-frame coded image frame I frame and a predictive coded image frame P frame after the I frame; each video frame in each IP frame group has the same picture content as the video frame with the same decoding timestamp DTS in the first video stream; the number of P frames included in each IP frame group is the same as the number of reference frames of the P frames in the first video stream.
[0198] The electronic device in the embodiment of the present disclosure can implement each process of the above-mentioned video processing method embodiment applied to the server and achieve the same effects and functions, which will not be repeated here.
[0199] In another specific embodiment, an electronic device includes a processor; and a memory configured to store computer-executable instructions, wherein when the computer-executable instructions are executed, the processor implements the following process:
[0200] Send live data push request to the server;
[0201] receiving a target IP frame group and a target playback stream from the server, wherein the target IP frame group is an IP frame group in the second video stream whose decoding timestamp DTS is before the reception time of the live data push request and the time interval between the DTS and the reception time is within a preset time length; and the target playback stream is a video frame in the first video stream whose DTS is after the last predictively coded image frame P frame in the target IP frame group and starts from the P frame;
[0202] The target IP frame group and the target playback stream are decoded.
[0203] The electronic device in the embodiment of the present disclosure can implement the various processes of the above-mentioned video processing method embodiment applied to the second terminal device and achieve the same effects and functions, which will not be repeated here.
[0204] Another embodiment of the present disclosure further provides a computer-readable storage medium for storing computer-executable instructions. When the computer-executable instructions are executed by a processor, the following process is implemented:
[0205] Based on a video captured by a first terminal device, a first video stream and a second video stream are generated by the first terminal device, wherein the first video stream includes a plurality of consecutive GOPs, and the second video stream includes a plurality of intra-frame coded image frames and predictive coded image frame IP frame groups arranged at intervals of a specific number of video frames; each GOP of the first video stream corresponds to one or more IP frame groups of the second video stream; each IP frame group includes an intra-frame coded image frame I frame and a predictive coded image frame P frame following the I frame; each video frame in each IP frame group has the same picture content as a video frame having the same decoding timestamp DTS in the first video stream; and the number of P frames included in each IP frame group is the same as the number of reference frames of the P frames in the first video stream;
[0206] The first video stream and the second video stream are sent to a server.
[0207] The storage medium in the embodiment of the present disclosure can implement the various processes of the above-mentioned video processing method embodiment applied to the first terminal device and achieve the same effects and functions, which will not be repeated here.
[0208] Another embodiment of the present disclosure further provides a computer-readable storage medium for storing computer-executable instructions. When the computer-executable instructions are executed by a processor, the following process is implemented:
[0209] Based on the video captured by the first terminal device, a first video stream and a second video stream are generated, wherein the first video stream includes multiple continuous GOPs, and the second video stream includes multiple intra-frame coded image frames and predictive coded image frame IP frame groups arranged at intervals of a specific number of video frames; each GOP of the first video stream corresponds to one or more IP frame groups of the second video stream; each IP frame group includes an intra-frame coded image frame I frame and a predictive coded image frame P frame after the I frame; each video frame in each IP frame group has the same picture content as the video frame with the same decoding timestamp DTS in the first video stream; the number of P frames included in each IP frame group is the same as the number of reference frames of the P frames in the first video stream.
[0210] The storage medium in the embodiment of the present disclosure can implement each process of the above-mentioned video processing method embodiment applied to the server and achieve the same effects and functions, which will not be repeated here.
[0211] Another embodiment of the present disclosure further provides a computer-readable storage medium for storing computer-executable instructions. When the computer-executable instructions are executed by a processor, the following process is implemented:
[0212] Send live data push request to the server;
[0213] receiving a target IP frame group and a target playback stream from the server, wherein the target IP frame group is an IP frame group in the second video stream whose decoding timestamp DTS is before the reception time of the live data push request and the time interval between the DTS and the reception time is within a preset time length; and the target playback stream is a video frame in the first video stream whose DTS is after the last predictively coded image frame P frame in the target IP frame group and starts from the P frame;
[0214] The target IP frame group and the target playback stream are decoded.
[0215] The storage medium in the embodiment of the present disclosure can implement each process of the above-mentioned video processing method embodiment applied to the second terminal device and achieve the same effects and functions, which will not be repeated here.
[0216] In various embodiments of the present disclosure, the computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0217] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures such as diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.
[0218] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.
[0219] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0220] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing the embodiments of the present disclosure, the functions of each unit can be implemented in the same or multiple software and / or hardware.
[0221] Those skilled in the art will appreciate that one or more embodiments of the present disclosure may be provided as a method, system, or computer program product. Therefore, one or more embodiments of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, one or more embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0222] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0223] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0224] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0225] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.
[0226] One or more embodiments of the present disclosure may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. One or more embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.
[0227] The various embodiments of this disclosure are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so their description is relatively simple. For relevant portions, refer to the description of the method embodiments.
[0228] The foregoing is merely an embodiment of the present disclosure and is not intended to limit the present disclosure. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure are intended to be included within the scope of the claims of the present disclosure.
Claims
1. A video processing method, wherein: include: Based on the video captured by the first terminal device, the first terminal device generates a first video stream and a second video stream, wherein the first video stream includes a plurality of continuous GOPs, and the second video stream includes a plurality of intra-frame coded image frames and predictive coded image frame IP frame groups arranged at intervals of a specific number of video frames; each GOP of the first video stream corresponds to one or more IP frame groups of the second video stream; each IP frame group includes an intra-frame coded image frame I frame and a predictive coded image frame P frame after the I frame; each video frame in each IP frame group has the same picture content as a video frame having the same decoding timestamp DTS in the first video stream; the number of P frames included in each IP frame group is the same as the number of reference frames of the P frames in the first video stream; The first video stream and the second video stream are sent to a server.
2. The method according to claim 1, wherein: When the first video stream does not include a bidirectional predictive coded image frame B frame, the I frame in the IP frame group has the same picture content as the P frame with the same decoding timestamp DTS in the first video stream, the P frame in the IP frame group has the same picture content as the P frame with the same decoding timestamp DTS in the first video stream, and all video frames in the IP frame group are set continuously.
3. The method according to claim 1, wherein: When the first video stream includes bidirectional predictive coded image frames B frames, the I frames in the IP frame group have the same picture content as the P frames with the same decoding timestamp DTS in the first video stream, the P frames in the IP frame group have the same picture content as the P frames with the same decoding timestamp DTS in the first video stream, and the interval between two adjacent video frames in the IP frame group is N frames; N is the number of B frames between two adjacent P frames in the first video stream.
4. The method according to any one of claims 1 to 3, wherein: When the number of reference frames of the P frame in the first video stream is 1, the IP frame group includes 1 I frame and 1 P frame.
5. The method according to any one of claims 1 to 3, wherein: When the number of reference frames of the P frames in the first video stream is greater than 1, the IP frame group includes 1 I frame and multiple P frames, and the number of P frames included in the IP frame group is the same as the number of reference frames of the P frames in the first video stream.
6. A video processing method, wherein: include: Based on the video captured by the first terminal device, a first video stream and a second video stream are generated, wherein the first video stream includes multiple continuous GOPs, and the second video stream includes multiple intra-frame coded image frames and predictive coded image frame IP frame groups arranged at intervals of a specific number of video frames; each GOP of the first video stream corresponds to one or more IP frame groups of the second video stream; each IP frame group includes an intra-frame coded image frame I frame and a predictive coded image frame P frame after the I frame; each video frame in each IP frame group has the same picture content as the video frame with the same decoding timestamp DTS in the first video stream; the number of P frames included in each IP frame group is the same as the number of reference frames of the P frames in the first video stream.
7. The method according to claim 6, wherein: The method further comprises: Receiving a live broadcast data push request from a second terminal device; In response to a live data push request, a target IP frame group and a target playback stream are sent to the second terminal device, wherein the target IP frame group is an IP frame group in which the DTS in the second video stream is located before the receiving time of the live data push request and the time interval between the DTS and the receiving time is within a preset time length, and the target playback stream is a video frame in which the DTS in the first video stream is located after the last P frame in the target IP frame group and starts from the P frame.
8. The method according to claim 7, wherein: The target IP frame group is an IP frame group in which the DTS in the second video stream is located before the receiving time and the DTS is closest to the receiving time.
9. The method according to claim 6, wherein: When the first video stream does not include a bidirectional predictive coded image frame B frame, the I frame in the IP frame group has the same picture content as the P frame with the same decoding timestamp DTS in the first video stream, the P frame in the IP frame group has the same picture content as the P frame with the same decoding timestamp DTS in the first video stream, and all video frames in the IP frame group are set continuously.
10. The method according to claim 6, wherein: When the first video stream includes bidirectional predictive coded image frames B frames, the I frames in the IP frame group have the same picture content as the P frames with the same decoding timestamp DTS in the first video stream, the P frames in the IP frame group have the same picture content as the P frames with the same decoding timestamp DTS in the first video stream, and the interval between two adjacent video frames in the IP frame group is M video frames; M is the number of B frames between two adjacent P frames in the first video stream.
11. The method according to any one of claims 6 to 10, wherein: When the number of reference frames of the P frame in the first video stream is 1, the IP frame group includes 1 I frame and 1 P frame.
12. The method according to any one of claims 6 to 10, wherein: When the number of reference frames of the P frames in the first video stream is greater than 1, the IP frame group includes 1 I frame and multiple P frames, and the number of P frames included in the IP frame group is the same as the number of reference frames of the P frames in the first video stream.
13. A video processing method, wherein: include: Send a live data push request to the server; Receive a target IP frame group and a target playback stream from the server, wherein the target IP frame group is an IP frame group in the second video stream whose decoding timestamp DTS is located before the receiving time of the live data push request and the time interval between DTS and the receiving time is within a preset time length, and the target playback stream is a video frame in the first video stream whose DTS is located after the last predictive coded image frame P frame in the target IP frame group and starts from the P frame; The target IP frame group and the target playback stream are decoded.
14. The method according to claim 13, wherein: The decoding of the target IP frame group and the target playback stream includes: When the target playback stream does not include a bidirectionally predictive coded image frame B frame, and the number of reference frames in the target playback stream is 1, and the first video frame in the target playback stream is not an I frame, the first P frame in the target playback stream is decoded with reference to the P frame in the target IP frame group, and the P frame in the target IP frame group is decoded with reference to the I frame in the target IP frame group.
15. The method according to claim 13, wherein: The decoding of the target IP frame group and the target playback stream includes: When the target playback stream does not include a bidirectional predictive coded image frame B frame, and the number of reference frames in the target playback stream is T, and the first video frame in the target playback stream is not an I frame, the first P frame in the target playback stream is decoded with reference to the first P frame to the Tth P frame in the target IP frame group, the Sth P frame in the target IP frame group is decoded with reference to the first S-1 P frames in the target IP frame group and the I frame in the target IP frame group, and the first P frame in the target IP frame group is decoded with reference to the I frame in the target IP frame group; wherein T is an integer greater than 1, and S is an integer greater than or equal to 2 and less than or equal to T.
16. The method according to claim 13, wherein: The decoding of the target IP frame group and the target playback stream includes: When the target playback stream includes bidirectionally predictive coded image frames B frames, the number of reference frames in the target playback stream is 1, and the first video frame in the target playback stream is not an I frame, the first P frame in the target playback stream is decoded with reference to the P frames in the target IP frame group, and the P frames in the target IP frame group are decoded with reference to the I frames in the target IP frame group.
17. The method according to claim 13, wherein: The decoding of the target IP frame group and the target playback stream includes: When the target playback stream includes bidirectional predictive coded image frames B frames, and the number of reference frames in the target playback stream is X, and the first video frame in the target playback stream is not an I frame, the first P frame in the target playback stream is decoded with reference to the first P frame to the Xth P frame in the target IP frame group, the Yth P frame in the target IP frame group is decoded with reference to the first Y-1 P frames in the target IP frame group and the I frame in the target IP frame group, and the first P frame in the target IP frame group is decoded with reference to the I frame in the target IP frame group; wherein X is an integer greater than 1, and Y is an integer greater than or equal to 2 and less than or equal to X.
18. A video processing device, wherein: include: The first generating unit is configured to generate a first video stream and a second video stream by the first terminal device based on a video captured by the first terminal device, wherein the first video stream includes a plurality of continuous GOPs. The second video stream includes a plurality of intra-frame coded image frames and predictive coded image frame IP frame groups arranged at intervals of a specific number of video frames; each GOP of the first video stream corresponds to one or more IP frame groups of the second video stream; each IP frame group includes an intra-frame coded image frame I frame and a predictive coded image frame P frame after the I frame; each video frame in each IP frame group has the same picture content as a video frame having the same decoding timestamp DTS in the first video stream; the number of P frames included in each IP frame group is the same as the number of reference frames of the P frames in the first video stream; The first sending unit is used to send the first video stream and the second video stream to a server.
19. A video processing device, wherein: include: The second generating unit is used to generate a first video stream and a second video stream based on the video captured by the first terminal device, wherein the first video stream includes a plurality of continuous GOPs, and the second video stream includes a plurality of intra-frame coded image frames and predictive coded image frame IP frame groups arranged at intervals of a specific number of video frames; each GOP of the first video stream corresponds to one or more IP frame groups of the second video stream; each IP frame group includes an intra-frame coded image frame I frame and a predictive coded image frame P frame after the I frame; each video frame in each IP frame group has the same picture content as the video frame with the same decoding timestamp DTS in the first video stream; the number of P frames included in each IP frame group is the same as the number of reference frames of the P frames in the first video stream.
20. A video processing device, wherein: include: A request sending unit, used to send a live data push request to the server; a data receiving unit, configured to receive a target IP frame group and a target playback stream from the server, wherein the target IP frame group is an IP frame group in the second video stream whose decoding timestamp DTS is located before the reception time of the live data push request and whose time interval between DTS and the reception time is within a preset time length, and the target playback stream is a video frame in the first video stream whose DTS is located after the last predictive coded image frame P frame in the target IP frame group and starts from the P frame; A data decoding unit is used to decode the target IP frame group and the target playback stream.
21. An electronic device, wherein: include: processor; as well as, A memory configured to store computer executable instructions, which, when executed, cause the processor to implement the steps of the method according to any one of claims 1 to 5.
22. An electronic device, wherein: include: processor; as well as, A memory configured to store computer executable instructions, which, when executed, cause the processor to implement the steps of the method according to any one of claims 6 to 12.
23. An electronic device, wherein: include: processor; as well as, A memory configured to store computer executable instructions, which, when executed, cause the processor to implement the steps of the method according to any one of claims 13 to 17.
24. A computer-readable storage medium, wherein: The computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions, when executed by a processor, implement the steps of the method described in any one of claims 1 to 5.
25. A computer-readable storage medium, wherein: The computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions, when executed by a processor, implement the steps of the method described in any one of claims 6 to 12.
26. A computer-readable storage medium, wherein: The computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the method described in any one of claims 13 to 17 are implemented.
Citation Information
Patent Citations
Video live-broadcasting method and video live-broadcasting server
CN107690073A
Video processing method and device, electronic equipment and storage medium
CN116614649A
Video encoding method and apparatus
EP3890329A1
Video distribution device, video transmission device, video distribution system, video distribution method, and video distribution program
JP2021141549A
