Video processing method and related product

Abstract

The invention provides a video processing method and related products, the method is executed by a server, and the method comprises the following steps: receiving a first request, the first request being used for requesting to regenerate a key frame; determining a first target video frame according to the first request, wherein the first target video frame is a first video frame ranked at a first position and determined from a to-be-forwarded first encoding code stream after the server receives the first request; generating a key frame according to the first target video frame; generating a connection forward prediction frame according to the key frame and a second video frame adjacent to the first video frame; generating fusion compensation information according to the connection forward prediction frame and a third video frame adjacent to the second video frame; generating a synchronous packet code stream according to the key frame, the connection forward prediction frame and the fusion compensation information; and at least directionally sending the synchronous packet code stream to a receiving end corresponding to the first request, so that the receiving end synchronously decodes the first coded code stream again. According to the embodiment of the invention, the increase of the overall code stream data volume of the server can be avoided.

Description

Technical Field

[0001] The present invention belongs to the technical field of video processing, and in particular relates to a video processing method and related products. Background Art

[0002] RTC (Real-Time Communication) is a real-time communication technology that allows people to communicate instantly in different geographical locations. In RTC communication systems, video encoding is a key link because it directly affects the quality of the video and the load of the network.

[0003] In order to reduce traffic, the RTC communication system can use infinite group of pictures (Infinite GOP) coding technology to encode video. Traditionally, the length of GOP is fixed, but in some advanced coding technologies, in order to improve coding efficiency and video quality, the length of GOP can become infinite, so it is called Infinite GOP. Infinite GOP can be achieved by setting the "--keyint infinite" parameter.

[0004] For example, this encoding technology is used in many video conferences, live broadcast applications, or distance learning. It can minimize network bandwidth consumption while ensuring the continuity and clarity of the video. However, since there are very few key frames (i.e., I frames) in InfiniteGOP, when an error occurs in one frame during transmission, it may affect multiple subsequent frames and cannot be repaired until the next key frame appears. This may lead to a decrease in video quality and playback interruption, and may also cause problems such as asynchrony between audio and video.

[0005] In order to solve the above problems, a new video processing method is urgently needed. Summary of the invention

[0006] In order to overcome the above-mentioned defects in the prior art, the present invention provides a video processing method, which is executed by a server, and the method includes: receiving a first request, the first request is used to request regeneration of a key frame; determining a first target video frame according to the first request, the first target video frame being a first video frame determined by the server to be arranged in the first position from a first coded code stream to be forwarded after receiving the first request; generating a key frame according to the first target video frame; generating a connected forward prediction frame according to the key frame and a second video frame adjacent to the first video frame; generating fusion compensation information according to the connected forward prediction frame and a third video frame adjacent to the second video frame; generating a synchronization data packet according to the key frame, the connected forward prediction frame and the fusion compensation information; and at least sending the synchronization data packet directionally to a receiving end corresponding to the first request, so that the receiving end can decode the first coded code stream synchronously again.

[0007] The present invention also provides a video processing method, which is executed by a receiving end, and the method includes: sending a first request, the first request is used to request the regeneration of a key frame; receiving a synchronization packet stream and a first encoding stream forwarded by a server, the synchronization packet stream is generated based on the key frame, a connected forward prediction frame and fusion compensation information, and is obtained through encoding processing, the key frame is generated based on a first target video frame, the first target video frame is a first video frame ranked in the first position determined by the server from the first encoding stream to be forwarded after receiving the first request, the connected forward prediction frame is generated based on the key frame and a second video frame adjacent to the first video frame, and the fusion compensation information is generated based on the connected forward prediction frame and a third video frame adjacent to the second video frame; decoding is performed based on the synchronization packet stream and the first encoding stream to obtain an original video frame matching the playback format of the receiving end.

[0008] The present invention also provides a video processing device, which is configured at a server end, and includes: a request receiving unit, which is used to receive a first request, and the first request is used to request the regeneration of a key frame; a target frame determining unit, which is used to determine a first target video frame according to the first request, and the first target video frame is a first video frame that is determined to be arranged in the first position from a first coded code stream to be forwarded by the server end after receiving the first request; a key frame generating unit, which is used to generate a key frame according to the first target video frame; a prediction frame generating unit, which is used to generate a connected forward prediction frame according to the key frame and a second video frame that is adjacent to the first video frame; a fusion information generating unit, which is used to generate fusion compensation information according to the connected forward prediction frame and a third video frame that is adjacent to the second video frame; a synchronization packet code stream generating unit, which is used to generate a synchronization packet code stream according to the key frame, the connected forward prediction frame and the fusion compensation information; and a code stream forwarding unit, which is used to at least send the synchronization packet code stream in a directionally manner to a receiving end corresponding to the first request, so that the receiving end can decode the first coded code stream synchronously again.

[0009] The present invention also provides a video processing device, which is configured with a client, and includes: a request sending unit, which is used to send a first request to a server, and the first request is used to request the regeneration of a key frame; a code stream receiving unit, which is used to receive a synchronization packet code stream and a first encoded code stream forwarded by the server, the synchronization packet code stream is generated according to the key frame, the connected forward prediction frame and the fusion compensation information, and is obtained after encoding processing, the key frame is generated according to the first target video frame, the first target video frame is the first video frame determined by the server to be arranged in the first position from the first encoded code stream to be forwarded after receiving the first request, the connected forward prediction frame is generated according to the key frame and the second video frame adjacent to the first video frame, and the fusion compensation information is generated according to the connected forward prediction frame and the third video frame adjacent to the second video frame; a code stream decoding unit, which is used to decode according to the synchronization packet code stream and the first encoded code stream to obtain the original video frame matching the playback format of the receiving end.

[0010] The present invention also provides a computer-readable storage medium on which a computer program is stored. The program is executed by a processor to implement the video processing method described above.

[0011] The present invention further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the video processing method described above when executing the computer program.

[0012] The video processing method and related products provided by the present invention are executed by a server, the method first receives a first request, the first request is used to request the regeneration of a key frame; then determines a first target video frame according to the first request, the first target video frame is the first video frame that the server determines to be arranged in the first position from the first coded code stream to be forwarded after receiving the first request; generates a key frame according to the first target video frame; generates a connected forward prediction frame according to the key frame and the second video frame adjacent to the first video frame; generates fusion compensation information according to the connected forward prediction frame and the third video frame adjacent to the second video frame; generates a synchronization data packet according to the key frame, the connected forward prediction frame and the fusion compensation information; at least sends the synchronization data packet directionally to the receiving end corresponding to the first request, so that the receiving end decodes the first coded code stream synchronously again. The embodiment provided by the present invention avoids sending unnecessary IDR frames to all receiving ends by generating a synchronization data packet, thereby reducing the overall code stream data volume, alleviating the pressure on network transmission, and further improving the quality of the decoded image of the receiving end by fusion compensation information. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0014] Figure 1 A schematic diagram showing an application scenario of the video processing method provided by an embodiment of the present invention is shown;

[0015] Figure 2 A schematic flow chart of a video processing method 200 provided in an embodiment of the present invention is shown;

[0016] Figure 3 A schematic diagram of a flow chart of a video processing method 300 provided in one embodiment of the present invention is shown;

[0017] Figure 4 A schematic flow chart of a video processing method 400 provided in one embodiment of the present invention is shown;

[0018] Figure 5 A schematic diagram showing a first encoding code stream and a synchronization packet code stream provided by an embodiment of the present invention is shown;

[0019] Figure 6 A schematic diagram showing a framework of a video processing method provided by an embodiment of the present invention;

[0020] Figure 7 A schematic diagram showing the structure of a video processing device 700 provided in one embodiment of the present invention is shown;

[0021] Figure 8 A schematic diagram of the structure of a video processing device 800 provided in one embodiment of the present invention is shown;

[0022] Fig. 9 A schematic diagram of the structure of a video processing system 900 provided in one embodiment of the present invention is shown;

[0023] Fig.10 A schematic diagram of the structure of a computer device 1000 provided in an embodiment of the present invention is shown.

[0024] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0025] In order to better understand and explain the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings. The present invention is not limited to these specific embodiments. On the contrary, modifications or equivalent replacements made to the present invention should all be included in the scope of the claims of the present invention.

[0026] It should be noted that many specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present invention can also be implemented without these specific details. In the multiple specific embodiments given below, the principles, structures and components well known in the art are not described in detail in order to highlight the main purpose of the present invention.

[0027] like Figure 1 As shown in the figure, the RTC service platform (Real-Time Communication Service Platform) is a platform that provides real-time audio and video communication services. It uses advanced real-time audio and video processing and transmission technology, combined with powerful cloud computing capabilities, to provide users with stable, high-quality real-time audio and video services. Through the RTC service platform, users can quickly build multi-platform real-time audio and video applications and realize cross-platform real-time audio and video communication.

[0028] The RTC service platform usually includes three parts: the push end 102, the server end 101 and the pull end 103. The server end 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server end 101 can also provide signaling services for media negotiation and channel establishment between the communicating parties.

[0029] The streaming end 102 is used to push media streams such as audio, video or data to the server 101 in real time. The streaming end 102 can be a web application, a mobile application or a dedicated hardware device.

[0030] The stream pulling terminals 103, 104, 105, 106 may also be referred to as receiving clients or receiving terminals, and are used to obtain media streams from the server 101 and play or process them locally. The stream pulling terminal 103 may be a web application, a mobile application, or a dedicated hardware device. The stream pulling terminals 103, 104, 105, 106 may be installed on electronic devices, and the electronic devices may include but are not limited to mobile devices such as smart phones and tablet computers, and computer devices such as desktop computers. Among them, the stream pulling terminal 105 is the receiving terminal that discovers packet loss, and the stream pulling terminal 106 is the receiving terminal that newly joins the call.

[0031] The push end 102 and the server end 101 can be connected via a network, and the server end 101 and the pull end 103 can be connected via a network. The network is used to support the push end 102 or the pull end 103 to connect to the server end 101 under any network conditions. It may include but is not limited to access agents, intelligent gateways and other devices. Optionally, the above-mentioned network may include a wireless network or a wired network, and standard communication technology and / or protocols are used through the wireless network or the wired network. The network is usually the Internet, and it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network.

[0032] During the whole process, the streaming end 102 and the streaming end 103 can interact through a browser or an application, while the server 101 is responsible for processing and forwarding the media stream, thereby achieving high-quality real-time communication.

[0033] In order to ensure a smooth playback experience in live broadcast or real-time video communication, Infinite GOP can be used to avoid buffering and delays caused by frequent insertion of I frames. In this mode, the encoder will try not to insert key frames unless it automatically detects that a key frame is required for scene switching. By setting fewer key frames, the compression rate can be maximized. At the same time, by reducing the number of key frames, the Infinite GOP mode also helps improve video quality and encoding efficiency.

[0034] However, in live broadcast or real-time video communication, users may encounter video freezes caused by unstable network. Due to network instability, the receiving end finds continuous frame loss, and the receiving end sends a request to the sending end to resynchronize the video. In the related art, after the sending end receives the picture loss indication request (full name Picture Loss Indicator, English abbreviation PLI) or full internal frame request (full name Full Instra Request, English abbreviation FIR request) information sent by the receiving end, in the main code stream, a P frame is replaced with an IDR (full name Instantaneous Decoder Refresh Frame, English abbreviation IDR) frame, and the subsequent P frame depends on this IDR frame, and then the main code stream with the temporarily inserted IDR frame and the subsequent P frame is sent to all receiving ends. After this operation, the receiving end with the problem can resynchronize and use the same video code stream. The problem caused by this solution is that for a receiving end with a problem, all receiving ends need to receive an unnecessary IDR frame, which increases the overall code stream data volume, puts pressure on network transmission, and may cause potential network transmission problems such as congestion and delay.

[0035] Therefore, the present invention hopes to propose a video processing method to solve the above problems and improve the user experience.

[0036] Please refer to Figure 2 , Figure 2 FIG. 2 is a flow chart of a video processing method 200 provided in an embodiment of the present invention. The method can be implemented by a video processing device, and the video processing device is configured on the server side. Figure 2 As shown, the method includes:

[0037] Step S201: receiving a first request, where the first request is used to request regeneration of a key frame.

[0038] In the above steps, the first request is used to request the server to regenerate a key frame. The key frame is an instant decoding refresh frame, i.e., an IDR frame. An IDR frame is a special I frame that not only has the characteristics of an I frame, but is also used to instruct the encoder and decoder to clear the decoded image cache. This means that all frames after the IDR frame cannot refer to the frames before the IDR frame, making the previous and next video sequences completely independent.

[0039] The first request may be a complete intra frame request or a picture loss indication request.

[0040] The full intra-frame request is a request mechanism used in real-time video transmission, which mainly requests the sender to regenerate and send a complete key frame (I frame) so that the receiver can correctly decode the video stream. This mechanism is particularly important in video conferencing and video streaming transmission, especially when the network environment is unstable or new participants join the meeting. In practical applications, such as video conferencing or live broadcast scenarios, new participants may only receive P frames or B frames when they join, and cannot be correctly decoded due to the lack of necessary I frames as reference. In this case, the new participant will send an FIR request to ask other participants to resend a data packet containing an I frame to ensure that they can decode and watch the video smoothly.

[0041] Picture Loss Indication Request is a mechanism used in real-time video transmission to indicate the frame loss detected by the receiver and request the sender to take corrective measures. This mechanism is very important in video conferencing and online streaming, especially in unstable network environments or packet loss. When this happens, the receiver sends a PLI message to the sender, and the sender can respond by retransmitting the lost packets or generating a new key frame (I frame).

[0042] In practical applications, such as video conferencing or live broadcasting, when the network environment is unstable and causes sudden packet loss, the receiver will send a PLI request to inform the sender which frames are affected. After receiving the PLI message, the sender can choose to retransmit the lost data packets or generate a new key frame to send to the receiver to ensure that the video can be decoded and played smoothly.

[0043] The first request may be received by the streaming end from the streaming end and then forwarded to the server end by the streaming end; the first request may also be received by the server end directly from the streaming end.

[0044] If the receiving end with the problem sends a first request directly to the sending end, the sending end can use its own encoder to generate a first video frame, then decode and reconstruct the first video frame, and then use the decoding and reconstruction result to generate an IDR frame, and then the sending end sends the IDR frame to the server.

[0045] If the receiving end with the problem directly sends the first request to the server end, since the server end (ie, the media forwarding server) is closer to the receiving end, the server end will respond to the first request more quickly after receiving the first request.

[0046] Step S202: determining a first target video frame according to the first request, where the first target video frame is a first video frame ranked at a first position determined by the server from the first encoded bitstream to be forwarded after receiving the first request.

[0047] In the above steps, the server receives the real-time original code stream from the streaming end. Before transcoding or encoding, the server usually needs to decode the real-time original code stream. For example, FFmpeg or other similar video processing libraries are used to parse and obtain the original video format. The decoded result is then transcoded or encoded to adapt the transcoding or encoding processing result to a specific playback device or network condition.

[0048] For example, re-encode to generate a new bitstream that meets the target format requirements. Generate a new bitstream using H.264, H.265 (HEVC), AVC or other coding standards. In order to distinguish, the new bitstream obtained by re-encoding the real-time original bitstream received from the push end by the server is called the main bitstream or the first encoded bitstream.

[0049] In some embodiments, the server forwards the first encoded bitstream to the corresponding stream-pulling end. When the stream-pulling end detects packet loss, it sends a first request to the stream-pushing end or the server. The server receives and parses the first request, and then determines the first video frame in the first encoded bitstream that is not forwarded to the stream-pulling end after the packet loss is detected according to the first request. Figure 6 As shown, the server end forwards the first coded stream {P1, P2, ..., P8} to the stream pulling end. When the stream pulling end finds that one frame or multiple consecutive frames are lost or there is a decoding error in the received first coded stream, assuming that the stream pulling end directly sends a PLI request to the server end as the first request, the server end responds to the first request and determines the video frame that is adjacent to the position of the lost consecutive frames in the first coded stream that is not forwarded to the stream pulling end as the first video frame. For example, if P2 and P3 frames are lost continuously, the P4 frame is determined to be the first video frame. That is, the first video frame ranked in the first position is determined from the first coded stream to be forwarded. Assuming that the first coded stream to be forwarded is {P4, P5, ..., P8}, the first video frame ranked in the first position is the P4 frame.

[0050] The first video frame can also be the most recently encoded video frame that is waiting to be transmitted. For example, the server forwards the first encoded code stream {P1, P2, ..., P8} to the stream pulling end. When the stream pulling end finds a frame loss, it immediately sends a first request to the server. The first request carries the information and timestamp of the lost video frame. When the server receives the first request, it determines the time of receiving the first request, and determines the most recently encoded video frame after that time as the first video frame. Assuming that the most recently encoded video frame is P6, P6 is determined to be the first video frame.

[0051] Using the latest encoded video frame as the first video frame is conducive to the receiving end to better solve the problem of video playback lag at the receiving end. After using the latest encoded video frame as the first video frame, the reconstructed frame of the latest encoded video frame can be cached so that subsequent decoding results all depend on the reconstructed frame of the latest encoded video frame.

[0052] When the newly encoded video frame is reconstructed, the previously cached reconstructed frame can be discarded.

[0053] Step S203: generating a key frame according to the first target video frame.

[0054] In the above steps, the server generates a key frame according to the first target video frame, that is, generates an IDR frame. Generating a key frame according to the first target video frame may include reconstructing the first target video frame to obtain first video frame data; and re-encoding the first video frame data in an internal encoding manner to obtain a key frame.

[0055] Optionally, after reconstructing the first target video frame to obtain the first video frame data, the encoder may be notified to generate an IDR frame by setting the frame type to an IDR frame. For example, this may be achieved by checking the properties of the encoder or directly setting the frame type. Then, a specific function such as x264_reference_reset is used to clear the reference frame queue of the decoder to ensure that the decoder starts decoding again from the IDR frame.

[0056] In some embodiments, the first request carries at least a user identifier and frame loss information. The user identifier may be a user name or any other identification information that can uniquely identify a user, such as a hardware identifier.

[0057] In the embodiment of the present application, an IDR frame is generated according to the first video frame, which can effectively refresh the decoder state and provide a reliable random access point, thereby improving the efficiency and quality of video transmission and playback.

[0058] like Figure 6 As shown, the code stream corresponding to the P4 frame is decoded in the first coded code stream to obtain the P4 frame, and then the P4 frame is input into the first reconstruction unit to obtain the first video frame data. Then the first video frame data is input into the IDR encoding unit, and the IDR encoding unit is used to re-encode the first video frame data in an internal encoding manner to generate an IDR frame. The key frame generator may include at least the first reconstruction unit and the IDR encoding unit.

[0059] The encoding process of IDR frames involves steps such as initialization, frame type selection, macroblock segmentation, prediction mode selection, transformation and quantization, entropy coding, generation of IDR frames, and output. The IDR coding unit is used to initialize the settings including parameters such as resolution, bit rate, frame rate, etc. at the beginning of the video coding sequence. The IDR coding unit is used to divide the video frame into macroblocks, which are the basic units of coding. Each macroblock can also be further divided into smaller blocks, such as 4x4 or 8x8 blocks in the H.264 coding standard. For IDR frames, all macroblocks use intra-frame prediction mode, that is, they do not rely on information from other frames. The IDR coding unit is used to determine the best intra-frame prediction mode to minimize the prediction error and transform the prediction residual, usually using discrete cosine transform (DCT) or its variants, such as integer transform. The transformed coefficients are quantized to reduce the amount of data. The quantized coefficients are entropy encoded to further compress the data. The IDR encoding unit is also used to encode auxiliary information such as intra-frame prediction mode and motion vector, and pack the encoded data into an IDR frame. The IDR frame may contain intra-frame encoded image data and related header information and auxiliary information.

[0060] Step S204: Generate a connected forward prediction frame according to the key frame and a second video frame that is immediately adjacent to the first video frame.

[0061] In the above steps, the connected forward prediction frame is a video frame obtained by re-encoding the second video frame in the first encoding code stream. The connected forward prediction frame is used to connect the IDR frame and the third video frame in the first encoding code stream. When forward encoding is performed with reference to the reconstructed data of the IDR frame, the encoding parameters used for forward encoding can inherit the encoding parameters of the forward encoding of the second video frame. On this basis, the encoding parameters are further adjusted so that the reconstructed data of the connected forward video frame is as close as possible to the reconstructed data of the second video frame.

[0062] The second video frame that is immediately adjacent to the first video frame refers to a video frame that is adjacent to the first video frame in the first coded bitstream. Figure 5 As shown, in the main code stream, the P5 frame adjacent to the P4 frame is the second video frame.

[0063] In some embodiments, a connected forward prediction frame is generated based on a key frame and a second video frame adjacent to a first video frame, including: reconstructing the second key frame to obtain second video frame data; performing inter-frame prediction coding with reference to the reconstructed data of the key frame to obtain a forward prediction frame to be adjusted; reconstructing the forward prediction frame to be adjusted to obtain third video frame data; determining a first residual value between the second video frame data and the third video frame data; and adjusting the coding parameters for inter-frame prediction coding based on the first residual value, so that when the residual value is less than a threshold value, the forward prediction frame to be adjusted is output as a connected forward prediction frame.

[0064] By adjusting the quantization parameter of the encoding, the residual value between the second video frame data and the third video frame data can be made close to a certain threshold, so that the second video frame data and the third video frame data are approximately equal.

[0065] In video coding, inter-frame prediction coding is an efficient method to remove temporal redundancy. The pixel values ​​of the encoded adjacent image (reference frame) are used to predict the pixel values ​​of the current image, and the predicted residual is encoded. For example, each pixel block in the second video frame data searches for the best matching block in the reconstructed data of the key frame. Within a given search range, a specific search algorithm (such as global search, three-step search method, diamond search method, etc.) can be used to find the reference block that is most similar to the current block. Then, matching criteria such as SAD (sum of absolute difference), SATD (sum of absolute difference after Hadamard transform) or MSE (mean square error) are used to determine the best matching block to minimize the prediction residual. Finally, the prediction accuracy can be improved by using sub-pixel precision motion estimation. For example, H.264 supports motion estimation with 1 / 4 pixel precision, while H.265 supports higher sub-pixel precision, such as 1 / 8 pixel interpolation.

[0066] After the motion vector is obtained according to motion estimation, the prediction block is obtained from the reconstructed data of the key frame and the prediction value of the current block is constructed. Then, the prediction block is subtracted from the current block to generate a residual signal for further encoding. The motion vector of the current block can be predicted using the motion vector of the neighboring block in the same image. Or other similar methods can be used to predict and encode the motion vector.

[0067] like Figure 6 As shown, the first coded code stream and the code stream corresponding to the P5 frame are decoded to obtain the P5 frame, the P5 frame is input into the fourth reconstruction unit to obtain the third video frame data, and then the third video frame data is input into the coding parameter adjustment unit for adjustment; the IDR frame is input into the second reconstruction unit to obtain the IDR reconstruction data, the IDR reconstruction data is input into the LP coding unit to obtain the coding data, the coding data is input into the third reconstruction unit to obtain the fourth video frame data, and then the fourth video frame data is input into the coding parameter adjustment unit, and the coding parameters of the LP coding unit are adjusted according to the residual between the third video frame data and the fourth video frame data, and finally a linked forward prediction frame, i.e., the LP frame (full name Link Predictive-Frame) is output, so that the residual between the LP frame and the P5 frame reconstruction data is minimized.

[0068] Step S205 , generating fusion compensation information according to the connected forward prediction frame and a third video frame that is adjacent to the second video frame.

[0069] In the above steps, the fusion compensation information refers to information used to correct the decoding distortion result after switching to the first coded bitstream. The fusion compensation information is generated by the difference between the YUV data reconstructed according to the third video frame in the first coded bitstream and the YUV data reconstructed according to the connected forward prediction frame when encoding the third video frame. After decoding the third video frame in the first coded bitstream at the decoding end, the fusion compensation information can be used for compensation to obtain a better quality image result.

[0070] Step S206, generating a synchronous packet stream according to the key frame, the connected forward prediction frame and the fusion compensation information.

[0071] In the above steps, entropy coding is performed on the transformed and quantized data, such as using Huffman coding as entropy coding.

[0072] The server generates a synchronization data packet based on the IDR frame, the connected forward prediction frame and the fusion compensation information. Figure 5 As shown, the synchronous packet code stream is obtained by encoding the IDR frame, the connected forward prediction frame (LP frame) and the fusion compensation information respectively. In some embodiments, the fusion compensation information is stored in the side information (English full name Side Information, English abbreviation SI).

[0073] The coded code stream corresponding to the P6 frame to the P8 frame is the coded code stream corresponding to the P6 frame to the P8 frame in the first coded code stream to be forwarded. The server generates a synchronous packet code stream and only sends the synchronous packet code stream to the receiving end with the problem in a directional manner, thereby effectively avoiding an increase in the number of overall code streams on the server and reducing the pressure on network transmission.

[0074] Step S207: at least send the synchronous packet code stream directionally to the receiving end corresponding to the first request, so that the receiving end can decode the first encoded code stream synchronously again.

[0075] In the above steps, when the server receives the first request, it records the user identifier of the receiving end that sends the first request; after generating the synchronous packet code stream, it forwards the synchronous packet code stream directionally to the receiving end corresponding to the user identifier according to the user identifier, and forwards the first encoded code stream to all receiving ends.

[0076] The embodiment provided by the present application, by sending the synchronous packet code stream and the first encoding code stream to the receiving end where the problem occurs, the receiving end where the problem occurs can synchronously decode the first encoding code stream, thereby saving the time of the receiving end where the problem occurs waiting for the fault to be resolved, reducing the overall code stream data volume of the server, and reducing the pressure on network transmission.

[0077] A network connection is established between the server and the receiver with the problem or the newly established receiver. TCP is usually used for video transmission that requires high reliability, such as live broadcast or video conferencing. Before sending the encoded bitstream, the encoded bitstream will be encapsulated, which includes adding necessary header information, timestamps, sequence numbers, etc. to ensure that the client can correctly parse and play the received video data. The encapsulation process may also involve slicing the bitstream to adapt to the maximum transmission unit of the network and optimize transmission efficiency. Once the encapsulation is completed, the encapsulation result is forwarded to the client. It is usually completed using the network protocol stack, and address addressing and routing selection are performed through the lower layer protocol (such as IP protocol). During the forwarding process, the server may need to process feedback information from the network, such as congestion control and retransmission requests, to ensure the stability of data transmission.

[0078] After the data is received at the receiving end, it is first decapsulated to restore the original coded stream. Then, the decoder at the receiving end will decode the received coded stream and finally present the video content.

[0079] In the embodiment provided by the present invention, when a new receiving end joins the RTC service platform or a certain receiving end has a packet loss problem, the server generates a synchronous packet code stream after receiving the first request, and only sends the synchronous packet code stream to the receiving end in a directed manner, which can effectively avoid an increase in the overall code stream data volume of the server and reduce the pressure on network transmission.

[0080] Combine the following Figure 3 , further describing the video processing method provided by the embodiment of the present application. Please refer to Figure 3 , Figure 3 FIG. 3 is a flow chart of a video processing method 300 provided by an embodiment of the present invention. The method can be implemented by a video processing device, which is configured on the server side. Figure 3 As shown, the method includes:

[0081] Step S301: receiving a first request sent by a client, where the first request is used to request regeneration of a key frame.

[0082] Step S302: determining a first target video frame according to the first request, where the first target video frame is a first video frame ranked at a first position determined by the server from the first encoded bitstream to be forwarded after receiving the first request.

[0083] Step S303: Generate a key frame according to the first target video frame.

[0084] Step S304, performing inter-frame prediction coding using the mode data of the second standard video frame to obtain a coding result.

[0085] Step S305, reconstructing the encoding result to obtain fourth video frame data.

[0086] Step S306, performing inter-frame prediction coding on the key frame to obtain a forward prediction frame to be adjusted; reconstructing the forward prediction frame to be adjusted to obtain third video frame data.

[0087] Step S307: determining a second residual value between the third video frame data and the fourth video frame data.

[0088] Step S308, adjusting the coding parameters for inter-frame prediction coding according to the second residual value, so that when the residual value is less than the threshold, the forward prediction frame to be adjusted is output as the connected forward prediction frame.

[0089] The mode data of a video frame refers to the data information used to describe the internal structure and properties of a video frame during the video encoding process. The mode data of a video frame includes, but is not limited to, partition, motion and other mode data. For example, partition mode data refers to the way in which a video frame is divided into different regions (or blocks) during video encoding. The purpose of partitioning is to process the video content more accurately so that the most appropriate encoding method can be applied to different regions. Motion mode data is mainly used to describe the motion trajectory of an object in a video frame.

[0090] Through accurate partitioning and motion estimation, as well as appropriate coding mode selection, the compression rate of video data can be greatly improved, while ensuring the smoothness and quality of video playback. In practical applications, the characteristics and advantages of various mode data should be comprehensively considered, and appropriate coding strategies and technologies should be selected in a targeted manner to achieve the best coding effect.

[0091] The following takes the H.264 coding standard as an example to illustrate the process of using the mode data of the second video frame to perform inter-frame prediction coding to obtain the coding result. First, select one or more encoded frames as reference frames for the current frame. For example, select the IDR frame as the reference frame and the second video frame as the current frame. For each macroblock in the current frame, a motion vector is calculated by comparing its difference with the macroblock at the corresponding position in the reference frame. The motion vector is used to describe the position offset of the macroblock in the current frame relative to the macroblock at the corresponding position in the IDR frame. According to the motion vector, the corresponding pixel value is obtained from the IDR frame and filled into the corresponding position of the current frame to obtain the predicted frame. The predicted frame is subtracted from the current frame to obtain the residual frame. The residual frame contains the difference information between the current frame and the predicted frame. Finally, the residual frame is subjected to discrete cosine transform and quantization operations to obtain the coding result.

[0092] Step S309: determine to connect the forward prediction frame as a reference frame, reconstruct the third video frame adjacent to the second video frame, and obtain fifth video frame data.

[0093] Step S310: determine the second video frame as a reference frame, reconstruct a third video frame adjacent to the second video frame, and obtain sixth video frame data.

[0094] Step S311, obtaining fusion compensation information according to a third residual value between the fifth video frame data and the sixth video frame data.

[0095] In the above steps, the fusion compensation information is used to perform compensation when the client decodes the second encoded code stream.

[0096] In the above steps, the second video frame is a video frame in the first coded bitstream, and the connected forward prediction frame is the reconstructed decoded data of the reference IDR frame, and the second video frame is reconstructed and decoded to obtain the video frame. The decoded IDR frame is used as a reference frame to predict, compensate and reconstruct the second video frame in order to utilize the temporal redundant information in the video frame sequence, improve the compression efficiency and maintain the video quality.

[0097] After determining the reference frame, the residual between the current frame and the predicted frame is calculated using the predicted frame obtained from the reference frame. The residual reflects the error after prediction.

[0098] In order to ensure that subsequent frames can be correctly decoded, the third video frame is reconstructed with reference to the connected forward prediction frame and the second video frame, respectively, to obtain the fifth video frame data and the sixth video frame data corresponding to the third video frame. Then, according to the third residual value between the fifth video frame data and the sixth video frame data, the fusion compensation information is obtained, and the fusion compensation information is combined with the motion vector and mode decision information of the third video frame through entropy coding to form the coded bit stream of the P6 frame.

[0099] In the embodiment of the present application, new frames are reconstructed by referring to decoded video frames, which effectively reduces the amount of data required for encoding while ensuring the continuity and smoothness of video playback.

[0100] Step S312, generating a synchronous packet stream according to the key frame, the connected forward prediction frame and the fusion compensation information.

[0101] Step S313: at least send the synchronous packet code stream directionally to the receiving end corresponding to the first request, so that the receiving end can decode the first encoded code stream synchronously again.

[0102] In the embodiment provided by the present invention, after the server receives the first request, a new IDR frame is generated, and then a connected forward prediction frame is generated using the mode data of the second video frame and the reconstructed data of the IDR frame, which can effectively improve the encoding speed; furthermore, by generating fusion compensation information, it helps to improve the quality of decoding the first encoded code stream at the receiving end with problems, thereby ensuring the smoothness of the video playback at the receiving end with problems.

[0103] The following describes the video processing method provided by this application from the perspective of the receiving end. Figure 4 , Figure 4 FIG. 4 is a flow chart of a video processing method 400 provided by an embodiment of the present invention. The method can be implemented by a video processing device, which is configured at the receiving end. Figure 4 As shown, the method includes:

[0104] Step S401: a first request is sent to a server, where the first request is used to request regeneration of a key frame.

[0105] Step S402: receiving the synchronous packet stream and the first encoding stream forwarded by the server.

[0106] In the above steps, the synchronization packet code stream is generated based on the key frame, the connected forward prediction frame and the fusion compensation information, and is obtained through encoding processing. The key frame is generated based on the first target video frame. The first target video frame is the first video frame ranked in the first position determined by the server from the first encoded code stream to be forwarded after receiving the first request. The connected forward prediction frame is generated based on the key frame and the second video frame adjacent to the first video frame. The fusion compensation information is generated based on the connected forward prediction frame and the third video frame adjacent to the second video frame.

[0107] Step S403: Decode according to the synchronous packet code stream and the first encoded code stream to obtain an original video frame matching the playback format of the receiving end.

[0108] In the above steps, the decoder at the receiving end is used to parse the code stream corresponding to the IDR frame. For example, metadata such as frame header information, sequence parameter set, and image parameter set are extracted. Since the IDR frame is a completely independent frame, it does not rely on any previously received frames, so there is no need to refer to other frames for prediction. Then, the decoder at the receiving end is also used to perform entropy decoding operations to convert variable-length coded codewords into quantization coefficients. The quantization coefficients obtained after entropy decoding need to be reconstructed through inverse transformation (such as inverse discrete cosine transform) and inverse quantization process to reconstruct pixel values. Inverse transformation is to convert frequency domain data back to spatial domain data, while inverse quantization is to restore the approximation of the original pixel value based on the quantization parameter.

[0109] To improve image quality, the decoder at the receiving end is also used to perform loop filtering on the decoded image. This includes deblocking filtering and other image enhancement techniques to reduce distortion and noise introduced during the encoding process. Finally, the decoder at the client end is used to output the decoded image data to the display device, completing the decoding process.

[0110] After decoding the IDR frame, the decoder at the receiving end continues to decode the received code stream corresponding to the connected forward prediction frame. The decoder at the receiving end parses the motion vector and mode decision information, and then uses this information to reconstruct the connected forward prediction frame. The client decoder is then used to perform inverse quantization and inverse DCT operations on the residual data to restore the residual frame. Finally, the decoder at the receiving end adds the residual frame back to the predicted frame to obtain a reconstructed frame corresponding to the connected forward prediction frame.

[0111] In some embodiments, the fusion compensation information is obtained based on a third residual value between fifth video frame data and sixth video frame data, the fifth video frame data is obtained by reconstructing a third video frame adjacent to the second video frame with reference to a connected forward prediction frame, and the sixth video frame data is obtained by reconstructing a third video frame adjacent to the second video frame with reference to the second video frame, and decoding is performed according to the synchronization packet stream and the first encoding stream, including: decoding the synchronization packet stream to obtain a key frame, a connected forward prediction frame and fusion compensation information; decoding the first encoding stream with reference to the connected forward prediction frame to obtain a reconstructed frame corresponding to the third video frame adjacent to the second video frame; and compensating the reconstructed frame using the fusion compensation information so that the compensated result is similar to the reconstructed frame of the third video frame.

[0112] In the embodiment provided by the present invention, when the receiving end finds packet loss or newly joins the RTC service platform, it actively sends a first request, expecting the service end to respond quickly, thereby reducing the waiting time of the receiving end with problems or new receiving ends. After the receiving end receives the synchronous packet stream, the synchronous packet stream can be used to connect the first encoding stream, thereby reducing the time required for the receiving end with problems or new receiving ends to wait for decoding synchronization, greatly improving the user experience.

[0113] Please refer to Figure 7 , Figure 7 The schematic diagram of the structure of a video processing device 700 provided in an embodiment of the present invention is shown. The device is configured at the server end, and the device includes:

[0114] The request receiving unit 701 is used to receive a first request sent by a client, where the first request is used to request to regenerate a key frame;

[0115] The target frame determination unit 702 is used to determine a first target video frame according to the first request, where the first target video frame is a first video frame that is determined by the server to be at a first position from the first coded bitstream to be forwarded after receiving the first request;

[0116] A key frame generating unit 703, configured to generate a key frame according to the first target video frame;

[0117] A prediction frame generating unit 704, configured to generate a connected forward prediction frame according to the key frame and a second video frame that is immediately adjacent to the first video frame;

[0118] A fusion information generating unit 705, configured to generate fusion compensation information according to the connected forward prediction frame and a third video frame that is adjacent to the second video frame;

[0119] A synchronization packet code stream generating unit 706, configured to generate a synchronization packet code stream according to key frames, connected forward prediction frames and fusion compensation information;

[0120] The code stream forwarding unit 707 is used to at least send the synchronization packet code stream in a directionally manner to the receiving end corresponding to the first request, so that the receiving end can synchronously decode the first encoded code stream again.

[0121] Optionally, the key frame generating unit 703 is further configured to reconstruct the first target video frame to obtain first video frame data; and re-encode the first video frame data in an internal encoding manner to obtain a key frame.

[0122] Optionally, the prediction frame generation unit 704 is also used to reconstruct the second standard video frame to obtain second video frame data; perform inter-frame prediction coding with reference to the reconstructed data of the key frame to obtain a forward prediction frame to be adjusted; reconstruct the forward prediction frame to be adjusted to obtain third video frame data; determine a first residual value between the second video frame data and the third video frame data; and adjust the coding parameters used for inter-frame prediction coding according to the first residual value, so that when the first residual value is less than a threshold value, the forward prediction frame to be adjusted is output as a connected forward prediction frame.

[0123] Optionally, the prediction frame generation unit 704 is also used to perform inter-frame prediction coding using the pattern data of the second video frame to obtain a coding result; reconstruct the coding result to obtain fourth video frame data; perform inter-frame prediction coding with reference to the reconstructed data of the key frame to obtain a forward prediction frame to be adjusted; reconstruct the forward prediction frame to be adjusted to obtain third video frame data; determine a second residual value between the third video frame data and the fourth video frame data; and adjust the coding parameters used for inter-frame prediction coding according to the second residual value, so that when the second residual value is less than a threshold value, the forward prediction frame to be adjusted is output as a connected forward prediction frame.

[0124] Optionally, the fusion information generating unit 705 also includes: a fifth reconstruction subunit, used to determine the connected forward prediction frame as a reference frame, reconstruct the third video frame adjacent to the second video frame, and obtain fifth video frame data; a sixth reconstruction subunit, used to determine the second video frame as a reference frame, reconstruct the third video frame adjacent to the second video frame, and obtain sixth video frame data; and a fusion compensation subunit, used to obtain fusion compensation information according to a third residual value between the fifth video frame data and the sixth video frame data.

[0125] In the embodiment provided by the present application, the video processing device is configured on the server side. After receiving the first request, the server first generates a new IDR frame, and then generates a connected forward prediction frame using the mode data of the second video frame and the reconstruction data of the IDR frame. It can effectively improve the encoding speed, thereby saving the waiting time of the receiving end where the problem occurs.

[0126] Furthermore, by generating the fusion compensation information, it is helpful to improve the quality of decoding the first encoded code stream at the receiving end with the problem, thereby ensuring the smoothness of the video playback at the receiving end with the problem.

[0127] Please refer to Figure 8 , Figure 8 The structure diagram of a video processing device 800 provided by an embodiment of the present invention is shown. The device is configured at a receiving end, and the device includes:

[0128] A request sending unit 801 is used to send a first request, where the first request is used to request to regenerate a key frame;

[0129] The bitstream receiving unit 802 is used to receive the synchronization packet bitstream and the first coded bitstream forwarded by the server, the synchronization packet bitstream is generated according to the key frame, the connected forward prediction frame and the fusion compensation information, and is obtained through encoding processing, the key frame is generated according to the first target video frame, the first target video frame is the first video frame that is determined to be arranged in the first position from the first coded bitstream to be forwarded by the server after receiving the first request, the connected forward prediction frame is generated according to the key frame and the second video frame that is adjacent to the first video frame, and the fusion compensation information is generated according to the connected forward prediction frame and the third video frame that is adjacent to the second video frame;

[0130] The code stream decoding unit 803 is used to decode the synchronous packet code stream and the first encoded code stream to obtain the original video frame matching the playback format of the receiving end.

[0131] Optionally, the fusion compensation information is obtained based on the third residual value between the fifth video frame data and the sixth video frame data, the fifth video frame data is obtained by reconstructing the third video frame adjacent to the second video frame with reference to the connected forward prediction frame, and the sixth video frame data is obtained by reconstructing the third video frame adjacent to the second video frame with reference to the second video frame. The bitstream decoding unit 803 is also used to parse the synchronization packet bitstream to obtain key frames, connected forward prediction frames and fusion compensation information; decode the first encoded bitstream with reference to the connected forward prediction frame to obtain a reconstructed frame corresponding to the third video frame adjacent to the second video frame; and compensate the reconstructed frame using the fusion compensation information so that the compensated result is similar to the reconstructed frame of the third video frame.

[0132] In the embodiment provided by the present invention, the video processing device is configured on the receiving end side. When the receiving end finds packet loss or newly joins the RTC service platform, it actively sends a first request, expecting the service end to respond quickly, which can reduce the waiting time of the receiving end with problems or new receiving ends. After receiving the synchronous packet stream, the receiving end uses the synchronous packet stream to connect and decode the first encoding stream, which reduces the time required for the receiving end with problems or new receiving ends to wait for decoding synchronization, greatly improving the user experience.

[0133] Please refer to Fig. 9 , Fig. 9 FIG. 9 is a schematic diagram showing the structure of a video processing system 900 provided by an embodiment of the present invention. The system includes a server 901 and at least one receiving end 902. The server includes Figure 7 The video processing device shown in FIG. 1 includes a receiving end 902 including: Figure 8 The video processing device shown.

[0134] Please refer to Fig.10 , Fig.10 FIG. 1 shows a schematic diagram of the structure of a computer device 1000 provided in an embodiment of the present invention. The computer device 1000 may be used to forward a coded stream to a client. Fig.10 As shown, the computer device 1000 includes at least a memory 1001 and a processor 1002. For example, the computer device may also include Fig.10 The embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned Figure 2-Figure 4 Describe the method.

[0135] In particular, according to the embodiment provided by the present invention, the above reference flow chart Figure 2-Figure 4 The described process can be implemented as a computer software program. For example, the embodiments provided by the present invention include a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program contains program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), the above-mentioned functions defined in the system of the present invention are executed.

[0136] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0137] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the methods and computer program products described in various embodiments provided by the present invention. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the aforementioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0138] The units or modules involved in the embodiments described in the present invention may be implemented by software or hardware. The units or modules described may also be arranged in a processor. For example, they may be described as: a processor includes a request receiving unit, a target frame determination unit, a key frame generation unit, a prediction frame generation unit, a fusion information generation unit, and a code stream forwarding unit. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves. For example, the request receiving unit may also be described as a "unit for receiving a first request."

[0139] As another aspect, an embodiment of the present invention further provides a computer-readable storage medium, which may be included in the computer device described in the above embodiment; or may exist independently and not be assembled into the computer device. The above computer-readable storage medium stores one or more programs, and when the above programs are used by one or more processors to execute the video processing method described in the present invention.

[0140] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) to form a technical solution.

Claims

1. A video processing method, characterized in that: The method is executed by the server, and includes: receiving a first request, wherein the first request is used to request regeneration of a key frame; Determine a first target video frame according to the first request, where the first target video frame is a first video frame that is determined by the server to be at a first position from the first encoded bitstream to be forwarded after receiving the first request; generating a key frame according to the first target video frame; generating a connected forward prediction frame according to the key frame and a second video frame that is immediately adjacent to the first video frame; generating fusion compensation information according to the connected forward prediction frame and a third video frame that is immediately adjacent to the second video frame; Generate a synchronous packet stream according to the key frame, the connected forward prediction frame and the fusion compensation information; At least the synchronous packet code stream is directionally sent to a receiving end corresponding to the first request, so that the receiving end can synchronously decode the first encoded code stream again.

2. The method according to claim 1, characterized in that: Generating a key frame according to the first target video frame includes: Reconstructing the first target video frame to obtain first video frame data; The first video frame data is re-encoded in an internal encoding manner to obtain the key frame.

3. The method according to claim 1, characterized in that Generating a connected forward prediction frame according to the key frame and a second video frame that is immediately adjacent to the first video frame, comprising: Reconstructing the second video frame to obtain second video frame data; Performing inter-frame prediction coding with reference to the reconstructed data of the key frame to obtain a forward prediction frame to be adjusted; Reconstructing the forward prediction frame to be adjusted to obtain third video frame data; Determine a first residual value between the second video frame data and the third video frame data; The coding parameters used for inter-frame prediction coding are adjusted according to the first residual value, so that when the first residual value is less than a threshold, the forward prediction frame to be adjusted is output as the connected forward prediction frame.

4. The method according to claim 1, characterized in that: Generating a connected forward prediction frame according to the key frame and a second video frame that is immediately adjacent to the first video frame, comprising: Performing inter-frame prediction coding using the mode data of the second video frame to obtain a coding result; Reconstructing the encoding result to obtain fourth video frame data; Performing inter-frame prediction coding with reference to the reconstructed data of the key frame to obtain a forward prediction frame to be adjusted; Reconstructing the forward prediction frame to be adjusted to obtain third video frame data; Determine a second residual value between the third video frame data and the fourth video frame data; The coding parameters used for inter-frame prediction coding are adjusted according to the second residual value, so that when the second residual value is less than a threshold, the forward prediction frame to be adjusted is output as the connected forward prediction frame.

5. The method according to claim 1, characterized in that The method further includes: When receiving the first request, recording a user identifier corresponding to the receiving end that sends the first request; The step of at least directionally sending the synchronous packet code stream to a receiving end corresponding to the first request comprises: Directively forwarding the synchronous packet code stream to a receiving end corresponding to the user identifier; And continue to forward the first encoded code stream to all receiving ends that establish communication connections with the server.

6. The method according to any one of claims 1 to 5, generating fusion compensation information according to the connected forward prediction frame and a third video frame immediately adjacent to the second video frame, comprising: Determine the connected forward prediction frame as a reference frame, reconstruct a third video frame that is adjacent to the second video frame, and obtain fifth video frame data; Determine the second video frame as a reference frame, reconstruct a third video frame that is adjacent to the second video frame, and obtain sixth video frame data; Fusion compensation information is obtained according to a third residual value between the fifth video frame data and the sixth video frame data.

7. A video processing method, characterized in that: The method is executed by a receiving end, and the method includes: Sending a first request, wherein the first request is used to request regeneration of a key frame; Receive a synchronization packet stream and a first coded stream forwarded by a server, wherein the synchronization packet stream is generated according to the key frame, the connected forward prediction frame and the fusion compensation information, and is obtained through encoding processing, the key frame is generated according to the first target video frame, the first target video frame is a first video frame that is determined by the server to be arranged in the first position from the first coded stream to be forwarded after receiving the first request, the connected forward prediction frame is generated according to the key frame and a second video frame that is adjacent to the first video frame, and the fusion compensation information is generated according to the connected forward prediction frame and a third video frame that is adjacent to the second video frame; Decoding is performed according to the synchronous packet code stream and the first encoded code stream to obtain original video frames that match the playback format of the receiving end.

8. The method according to claim 7, characterized in that The fusion compensation information is obtained according to a third residual value between the fifth video data and the sixth video data, the fifth video data is obtained by reconstructing a third video frame adjacent to the second video frame with reference to the connected forward prediction frame, the sixth video data is obtained by reconstructing a third video frame adjacent to the connected forward prediction frame with reference to the second video frame, and the decoding according to the synchronous packet code stream and the first encoded code stream includes: Decoding the synchronous packet code stream to obtain the key frame, the connected forward prediction frame and the fusion compensation information; Referring to the connected forward prediction frame, decoding the first coded bitstream to obtain a reconstructed frame corresponding to a third video frame immediately adjacent to the second video frame; The reconstructed frame is compensated using the fusion compensation information so that a compensated result is similar to the reconstructed frame of the third video frame.

9. A video processing device, characterized in that: The device is configured on the server side, and includes: A request receiving unit, configured to receive a first request, wherein the first request is used to request regeneration of a key frame; a target frame determining unit, configured to determine a first target video frame according to the first request, wherein the first target video frame is a first video frame that is determined by the server to be at a first position from the first coded bitstream to be forwarded after receiving the first request; A key frame generating unit, configured to generate a key frame according to the first target video frame; A prediction frame generating unit, configured to generate a connected forward prediction frame according to the key frame and a second video frame that is immediately adjacent to the first video frame; A fusion information generating unit, configured to generate fusion compensation information according to the connected forward prediction frame and a third video frame that is immediately adjacent to the second video frame; A synchronous packet code stream generating unit, configured to generate a synchronous packet code stream according to the key frame, the connected forward prediction frame and the fusion compensation information; The code stream forwarding unit is used to at least send the synchronous packet code stream in a directionally manner to a receiving end corresponding to the first request, so that the receiving end can synchronously decode the first encoded code stream again.

10. A video processing device, characterized in that: The device is configured at a receiving end, and includes: A request sending unit, configured to send a first request to a server, wherein the first request is used to request regeneration of a key frame; a code stream receiving unit, configured to receive a synchronization packet code stream and a first coded code stream forwarded by a server, wherein the synchronization packet code stream is generated according to the key frame, the connected forward prediction frame and the fusion compensation information, and is obtained through encoding processing, wherein the key frame is generated according to the first target video frame, and the first target video frame is a first video frame that is determined by the server to be arranged in the first position from the first coded code stream to be forwarded after receiving the first request, wherein the connected forward prediction frame is generated according to the key frame and a second video frame that is adjacent to the first video frame, and wherein the fusion compensation information is generated according to the connected forward prediction frame and a third video frame that is adjacent to the second video frame; The code stream decoding unit is used to decode the synchronization packet code stream and the first encoded code stream to obtain an original video frame that matches the playback format of the receiving end.

11. A video processing system, characterized in that: The system comprises a server and at least one receiving end, wherein the server comprises the video processing device according to claim 9, and the client comprises the video processing device according to claim 10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the video processing method according to any one of claims 1 to 8.

13. A computer device comprising a memory and a processor, wherein: The memory stores a computer program, and the processor implements the video processing method according to any one of claims 1 to 8 when executing the computer program.

Images