Video processing method and device

Abstract

The invention provides a video processing method and device. The method comprises the steps of decoding a first code stream to obtain a plurality of video frames included in a first video and coding information of the plurality of video frames, determining at least one first target video frame included in the plurality of video frames according to the coding information of the plurality of video frames, and sending a second code stream, the at least one first target video frame is a video frame needing to be discarded in the plurality of video frames, the second code stream carries a plurality of second target video frames, and the plurality of second target video frames are other video frames except the at least one first target video frame in the plurality of video frames. According to the method, whether each video frame is suitable for being reserved or discarded as a to-be-interpolated frame is analyzed, which video frames are suitable for being discarded can be more accurately decided, and a receiving end can effectively utilize the reserved video frames due to the fact that a transmitting end discards the video frames which are easy to interpolate; and the corresponding to-be-interpolated frame can be easily generated, so that the video frame interpolating quality can be ensured.

Description

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on July 4, 2023, with application number 202310813148.6 and application name “A Video Transcoding Method and Device”, the entire contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to a video processing method and device. Background Art

[0003] Video coding is a video compression technology that removes redundant information by performing operations such as block division, prediction, transformation, quantization, inverse quantization, inverse transformation, and entropy coding on video frames, and then encodes the video frames without redundant information into a bitstream for transmission, thereby greatly reducing the amount of data transmitted. However, although the compression effect of video coding is constantly being optimized with the continuous updating of video coding standards, it also brings about a higher encoding and decoding time complexity, which makes high-performance encoders unavailable for commercial use. Therefore, the compression effect of video is still very limited, especially in poor network environments. Due to limited transmission bandwidth, it is difficult for users to have a clear and smooth video viewing experience.

[0004] In response to the above problems, one solution is to perform time-domain downsampling on the video sequence first, so that the amount of data is reduced before the video enters the encoder, thereby greatly reducing the bitstream information after encoding. Although this solution can further save bandwidth and improve the user's video viewing experience, the video fluency is reduced due to the reduction in the number of video sequence frames decoded by the client. In order to solve the problem of the reduction in the number of video sequence frames decoded by the client, one solution is to use interpolation technology to generate some video frames to fill the original video sequence. Although this solution can ensure the smoothness of the video, due to the violent movement of some objects in the video sequence and the disappearance of some objects due to occlusion, the motion information such as optical flow estimated by the interpolation technology is not accurate enough, resulting in artifacts and other distortions in the generated video frames. Summary of the invention

[0005] The present application provides a video processing method and device for improving the quality of video frame insertion.

[0006] In a first aspect, the present application provides a video processing method, which can be performed by a video processing device (such as a first video processing device) or a component (such as a chip, chip system or circuit, etc.) that can support the video processing device to implement the functions required by the method. Exemplarily, the first video processing device can be a server side (such as a cloud server or a data center) or a component (such as a plug-in, component, chip or circuit, etc.) that can support the server side to implement the functions required by the method. For example, the server side can be a cloud server or a data center or a server or other forms of computing devices. Optionally, taking the first video processing device executing the video processing method as an example, the method can include the following steps:

[0007] The first video processing device may first decode the first code stream to obtain multiple video frames included in the first video and encoding information of the multiple video frames, wherein the encoding information may include one or more of block division information, motion vector information, prediction mode and block residual information of the multiple video frames; then, the first video processing device may determine at least one first target video frame included in the multiple video frames according to the encoding information of the multiple video frames, wherein the at least one first target video frame is a video frame that needs to be discarded among the multiple video frames; then, the first video processing device may send a second code stream, wherein the second code stream carries multiple second target video frames, and the multiple second target video frames are other video frames among the multiple video frames except for the at least one first target video frame.

[0008] In this method, the first video processing device analyzes the attributes of each video frame (i.e., whether each video frame is suitable for being retained or suitable for being discarded as a frame to be inserted (or can be understood as a frame extraction)) according to the encoding information of each video frame in multiple video frames, and can more accurately decide which video frames are suitable for being discarded according to the attributes of multiple video frames. Afterwards, the first video processing device can encode the retained video frames into a code stream for transmission. Then, after the second video processing device (such as a client) obtains the code stream, it can decode the code stream to obtain the retained video frames. Because the first video processing device discards the video frames that are easier to interpolate (i.e., the discarded video frames are more suitable as the frames to be inserted), the second video processing device can generate the corresponding frames to be inserted more easily based on the retained video frames, so that the quality of the video interpolation can be ensured, and it helps to alleviate the phenomenon that some video frames with more intense motion are discarded, and the quality of the generated frames to be inserted is poor due to inaccurate estimation of the motion vector information of these video frames with more intense motion. In addition, since the first video processing device performs more accurate time domain downsampling operations on multiple video frames, it can discard some video frames to the maximum extent, and the data transmitted by the first video processing device is encoded, the amount of data of the code stream received by the second video processing device can be further reduced, thereby further reducing the transmission bandwidth, which helps to alleviate the phenomenon of poor video viewing experience for users in a weak network environment.

[0009] In a possible design, a first video processing device determines at least one first target video frame included in multiple video frames based on encoding information of multiple video frames, including: the first video processing device can first determine a texture complexity index value of the first video frame based on the encoding information of the first video frame, wherein the first video frame is any one of the multiple video frames; thereafter, when the texture complexity index value of the first video frame meets a texture complexity threshold, it can be determined that the first video frame is a first target video frame.

[0010] In the above design, by using the texture complexity index value of each video frame, it is possible to more accurately determine whether the video frame is the first target video frame (i.e., a video frame that is more suitable as a frame to be interpolated (or can be understood as a video frame that is easier to interpolate)), so that the second video processing device can more easily generate the frame to be interpolated, effectively reducing the interpolation pressure of the second video processing device, and ensuring that the quality of the frame to be interpolated (or can be called an interpolated frame image) is relatively high.

[0011] In a possible design, each of a plurality of video frames includes a plurality of coding blocks, the coding information of each video frame includes prediction modes of the plurality of coding blocks of the video frame, the texture complexity index value includes a first ratio and / or a second ratio, and the texture complexity threshold includes a first threshold and / or a second threshold; when the texture complexity index value of the first video frame satisfies the texture complexity threshold, the first video processing device determines that the first video frame is a first target video frame, including: when the first ratio corresponding to the first video frame is greater than the first threshold and / or the second ratio corresponding to the first video frame is less than the second threshold, the first video processing device can determine that the first video frame is a first target video frame, wherein the first ratio is used to indicate the proportion of the number of pixels in the area where the coding blocks whose prediction mode is an inter-frame prediction mode are located in the first video frame, and the second ratio is used to indicate the proportion of the number of pixels in the area where the coding blocks whose prediction mode is an intra-frame prediction mode are located in the first video frame.

[0012] In the above design, the first threshold and / or the second threshold can be used to relatively accurately determine whether each video frame is the first target video frame, so that it can be more accurately determined which video frames are suitable for being discarded as the first target video frames.

[0013] In a possible design, the method further includes: when the first ratio is less than or equal to a first threshold, or the second ratio is greater than or equal to a second threshold, the first video processing device may determine that the first video frame is a second target video frame.

[0014] In the above design, the first threshold and / or the second threshold can be used to relatively accurately determine whether each video frame is the second target video frame, so that it can be more accurately determined which video frames are suitable to be retained as the second target video frames.

[0015] In one possible design, the method also includes: the first video processing device sends a third code stream, wherein the third code stream carries attribute information of at least one first target video frame, and the attribute information is used to indicate that a to-be-inserted frame corresponding to the first target video frame is generated in multiple second target video frames according to the attribute information.

[0016] In the above design, the first video processing device can encode the attribute information of at least one first target video frame into a third code stream for transmission, so that the second video processing device that obtains the third code stream can effectively and comprehensively utilize the attribute information of the discarded video frames and the retained video frames to generate corresponding frames to be inserted, so that the quality of video insertion can be further guaranteed, thereby solving the problem of many artifacts in the generated video due to inaccurate estimation of motion vector information of the frames to be inserted in the existing solution, and can further effectively alleviate the phenomenon that after some video frames with more intense motion are discarded, the quality of the generated frames to be inserted is poor due to inaccurate estimation of motion vector information of these video frames with more intense motion.

[0017] In one possible design, the method also includes: when the prediction mode of the first coding block corresponding to the first target video frame is the inter-frame prediction mode, the first video processing device can scale the motion vector information of the first coding block to obtain the scaled motion vector information of the first coding block; or when the prediction mode of the first coding block is the intra-frame prediction mode, if there is a second coding block whose prediction mode is the inter-frame prediction mode on the left, above or above the left of the first coding block, the first video processing device can scale the motion vector information of the second coding block to obtain the motion vector information of the first coding block; if there is no second coding block whose prediction mode is the inter-frame prediction mode in the first coding block, the first video processing device can use (0,0) as the motion vector information of the first coding block; thereafter, the first video processing device can determine the attribute information of the first target video frame according to the motion vector information of multiple coding blocks included in the first target video frame and the position information of the first target video frame in the first video.

[0018] In the above design, for each first target video frame, the current motion vector information of each coding block included in the video frame can be calculated relatively accurately based on the prediction mode of each coding block included in the first target video frame, so that it is convenient to subsequently use the attribute information of at least one discarded first target video frame and the retained multiple second target video frames to accurately generate corresponding frames to be inserted, which helps to ensure the quality of video insertion and thus ensure the picture quality and smoothness of the video.

[0019] In a possible design, the method also includes: when the first video processing device decodes video frames of a first number threshold, it can generate a to-be-inserted frame corresponding to the first target video frame based on motion vector information of a first target video frame included in the video frames of the first number threshold and two second target video frames located before and after the first target video frame; thereafter, when the first video processing device determines that the quality index value of the to-be-inserted frame is less than or equal to the quality threshold, it reduces the second threshold; or when it is determined that the number of first target video frames included in the video frames of the first number threshold is less than or equal to the second number threshold, it reduces the first threshold and / or increases the second threshold.

[0020] In the above design, because a fixed first threshold and / or second threshold does not have good robustness, when analyzing which video frames are suitable for discarding and which video frames are suitable for retaining, the analysis strategy is dynamically adjusted according to the actual situation of the video sequence encountered, so that the video processing solution can have a better processing effect when facing most scenarios.

[0021] In a possible design, the method further includes: the first video processing device can determine a peak signal-to-noise ratio corresponding to the frame to be inserted based on the first target video frame and the frame to be inserted, wherein the peak signal-to-noise ratio can be used as a quality indicator value of the frame to be inserted.

[0022] In the above design, by using the peak signal-to-noise ratio, the quality of the frame to be inserted can be more accurately represented.

[0023] In a second aspect, the present application provides a video processing method, which can be performed by a video processing device (such as a second video processing device) or a component (such as a chip, a chip system or a circuit, etc.) that can support the video processing device to implement the functions required by the method. Exemplarily, the second video processing device can be a client or a component (such as a plug-in, a component, a chip or a circuit, etc.) that can support the client to implement the functions required by the method. For example, the terminal device installed with the client can be a computing device. When the second video processing device is a client, the client can be used to process a code stream from a server, and the server is used to process a first code stream. Among them, the first code stream carries multiple video frames included in the first video and encoding information of multiple video frames. Among them, at least one first target video frame among the multiple video frames is a video frame that needs to be discarded, and multiple second target video frames among the multiple video frames are other video frames among the multiple video frames except at least one first target video frame. The encoding information may include one or more of block division information, motion vector information, prediction mode and block residual information of multiple video frames. Optionally, taking the second video processing device performing the video processing method as an example, the method may include the following steps:

[0024] The second video processing device may receive a second code stream from the server, wherein the second code stream carries a plurality of second target video frames. Then, the second video processing device may generate a second video according to the plurality of second target video frames.

[0025] For the technical effects that can be achieved in the second aspect, please refer to the technical effects that can be achieved in the first aspect mentioned above, and no further details will be given here.

[0026] In one possible design, the second video processing device generates a second video based on multiple second target video frames, including: the second video processing device receives a third code stream from a server, wherein the third code stream carries attribute information of at least one first target video frame, and the attribute information is used to indicate that a to-be-inserted frame corresponding to the first target video frame is generated in the multiple second target video frames according to the attribute information; thereafter, the second video processing device can generate the second video based on the attribute information of the multiple second target video frames and at least one first target video frame.

[0027] In a possible design, the attribute information of each first target video frame in at least one first target video frame includes the position information of the first target video frame in the first video and the motion vector information of multiple coding blocks included in the first target video frame; the second video processing device generates a second video according to the attribute information of the multiple second target video frames and the at least one first target video frame, including: the second video processing device can generate a to-be-inserted frame corresponding to each first target video frame according to the motion vector information of the multiple coding blocks corresponding to each first target video frame and the two second target video frames located before and after each first target video frame, wherein the position information of the to-be-inserted frame corresponding to each first target video frame is the same as the position information of the first target video frame in the first video; then, the second video processing device can insert the to-be-inserted frame corresponding to each first target video frame into the multiple second target video frames according to the position information of each first target video frame in the first video to obtain the second video.

[0028] In a third aspect, the present application provides a possible computing device, which has the function of implementing the behavior of the first video processing device in the method example of the first aspect above. The beneficial effects can be found in the description of the first aspect and will not be repeated here. The computing device includes a communication interface, a memory, and a processor. Among them, the communication interface is used to communicate with other devices, the memory is used to store computer programs or instructions, and the processor is coupled to the memory and the communication interface. When the processor executes the computer program or instruction, the computing device executes the method in any possible design of the first aspect above.

[0029] Optionally, the computing device may also have the function of implementing the behavior of the second video processing device in the method example of the second aspect above. The beneficial effects can be found in the description of the second aspect and will not be repeated here. The computing device includes a communication interface, a memory, and a processor. The communication interface is used to communicate with other devices, the memory is used to store computer programs or instructions, and the processor is coupled to the memory and the communication interface. When the processor executes the computer program or instruction, the computing device executes the method in any possible design of the second aspect above.

[0030] In a fourth aspect, the present application provides a possible computing device cluster, which has the function of implementing the behavior of the video processing device in the method example of the first aspect above. The beneficial effects can be found in the description of the first aspect, which will not be repeated here. The computing device cluster includes at least one computing device, any computing device includes a communication interface, a processor and a memory, and the processor in any computing device is configured to support the video processing device to perform part or all of the functions in the method in any possible design of the first aspect above. The memory is coupled to the processor, which stores the computer program instructions and data necessary for the first video processing device.

[0031] Optionally, the computing device cluster may also have the function of implementing the behavior of the second video processing device in the method example of the second aspect above. The beneficial effects can be found in the description of the second aspect and will not be repeated here. The computing device cluster includes at least one computing device, any computing device includes a communication interface, a processor and a memory, and the processor in any computing device is configured to support the second video processing device to perform part or all of the functions of the method in any possible design of the second aspect above. The memory is coupled to the processor, which stores the computer program instructions and data necessary for the second video processing device.

[0032] In a fifth aspect, the present application provides a computer-readable storage medium, which stores computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes a method in any possible design of the first aspect or a method in any possible design of the second aspect.

[0033] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device, enables the computing device to execute a method in any possible design of the first aspect or a method in any possible design of the second aspect.

[0034] In the seventh aspect, the present application also provides a computer chip, which is coupled to a memory, and is used to read a computer program stored in the memory to execute any possible method in the design of the first aspect or any possible method in the design of the second aspect.

[0035] In an eighth aspect, the present application further provides a chip system, the chip system comprising a processor for supporting a computer device to implement the method in any possible design of the first aspect or the method in any possible design of the second aspect. In one possible design, the chip system further comprises a memory for storing the necessary programs and data of the computer device. The chip system may be composed of a computer chip, or may include a computer chip and other discrete devices.

[0036] Based on the implementations provided in the above aspects, the present application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of a possible application scenario provided by an embodiment of the present application is exemplified;

[0038] Figure 2 A schematic diagram exemplarily illustrates a flow chart of a video processing method provided in an embodiment of the present application;

[0039] Figure 3 A schematic diagram of calculating current motion vector information of each coding block included in a first target video frame provided by an embodiment of the present application is exemplarily shown;

[0040] Figure 4a A schematic diagram showing a possible structure of a video processing device provided by an embodiment of the present application is exemplified;

[0041] Figure 4b A schematic diagram showing the structure of another possible video processing device provided by an embodiment of the present application is exemplified;

[0042] Figure 5a A schematic diagram showing a possible structure of a computing device provided by an embodiment of the present application is exemplified;

[0043] Figure 5b A schematic diagram showing the structure of another possible computing device provided by an embodiment of the present application is exemplified;

[0044] Figure 6a A schematic diagram showing a possible computing device cluster structure provided by an embodiment of the present application is exemplified;

[0045] Figure 6b A schematic diagram showing the structure of another possible computing device cluster provided by an embodiment of the present application is exemplified;

[0046] Figure 7a A schematic diagram showing a possible structure of a computing device cluster provided by an embodiment of the present application is exemplified;

[0047] Figure 7b A schematic diagram of the structure of another possible computing device cluster provided in an embodiment of the present application is exemplified. DETAILED DESCRIPTION

[0048] Before introducing the technical solution provided by the present application, some of the terms involved in the present application are first explained to facilitate understanding by those skilled in the art.

[0049] (1) Interpolation: Generate new frames between video frames in the original video sequence.

[0050] (2) Skip mode: refers to a prediction mode in the prediction stage when encoding a video frame, which is a type of inter-frame prediction mode. In Skip mode, the coding block obtains predicted pixel values ​​through motion estimation and directly uses the predicted pixel values ​​as reconstructed pixel values ​​without transmitting residual values. It is generally used when the predicted pixel values ​​are relatively accurate.

[0051] (3) Intra mode: refers to a prediction mode in the prediction stage when encoding a video frame, which belongs to the intra prediction mode. In the intra mode, the coding block uses the surrounding reference pixels in the current frame to obtain the predicted pixel value without referring to the coding blocks of other frames.

[0052] (4) Video transcoding: refers to converting a compressed and encoded video stream into another video stream to adapt to different network bandwidths, different terminal processing capabilities, and different user needs.

[0053] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0054] The following is an introduction to possible application scenarios to which the video processing method provided by the present application is applicable. It should be noted that these introductions are intended to facilitate understanding by those skilled in the art and do not limit the scope of protection claimed by the present application.

[0055] Figure 1 The following is a schematic diagram showing a possible application scenario to which the present application embodiment is applicable. Figure 1 As shown, the application scenario may include a client 100 and a server 200. Optionally, the terminal device where the client 100 is located and the server 200 may be connected to each other through one or more networks. The network may be a wired network or a wireless network, for example, the wireless network may be a wireless fidelity (WIREless-Fidelity, WIFI) network, or a mobile cellular network, or other forms of networks, which are not limited in the embodiments of the present application.

[0056] Among them, the client 100 can be installed on a terminal device to provide video services (such as live broadcast service, video recording service, live broadcast viewing service or video viewing service, etc.) to users. Optionally, in a video transcoding scenario based on end-cloud collaboration, after collecting the user's relevant video, the client 100 encodes the user's relevant video into a corresponding code stream and transmits it to the server 200, so that the server 200 can transcode the user's relevant video. Exemplarily, after acquiring the user's relevant video, the server 200 can perform a time domain downsampling operation on the user's relevant video, that is, the server discards some video frames included in the relevant video, such as the server can discard the first target video frame included in the relevant video (or a video frame that can be understood as a discarded frame). The first target video frame may refer to a video frame that is more suitable as a frame to be interpolated (or can be called a video frame that is relatively easy to interpolate).

[0057] Optionally, the terminal device may also be referred to as a terminal, user equipment (UE), access terminal equipment, vehicle-mounted terminal, industrial control terminal, UE unit, UE station, mobile station, mobile station (MS), mobile terminal (MT), remote station, remote terminal equipment, mobile equipment, UE terminal equipment, terminal equipment, wireless communication equipment, UE agent or UE device, etc. In the embodiment of the present application, the terminal device may be fixed or mobile, and the implementation of the present application does not limit this.

[0058] Exemplarily, the terminal device may be a mobile phone, a tablet computer, a subscriber unit, a cellular phone, a smart phone, a wireless data card, a personal digital assistant (PDA) computer, a wireless modem, a handheld device (handset), a laptop computer, a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a terminal device capable of sidelink communication (such as a vehicle-mounted terminal device or a handheld terminal for V2X communication, etc.), a wireless terminal in self-driving, a wireless terminal in remote medical, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, a wearable terminal device, a vehicle, a drone, a helicopter, an airplane, a factory machine / equipment, a machine type communication (Machine Type Communication, MTC) terminals, ships or robots, etc. The embodiments of the present application do not limit the specific technology and specific device form adopted by the terminal device.

[0059] Exemplarily, the client 100 may include a capture module (or video capture module), an encoding module (or video encoding module), a decoding module (or video decoding module) and a video interpolation module. It should be noted that: Figure 1 The connection relationship between the various functional modules in the client 100 is only an example and does not constitute a limitation of the present application. The functions of the various functional modules included in the client 100 are described below.

[0060] The acquisition module is used to provide video acquisition services for users. For example, the acquisition module is used to collect videos generated by users' live broadcasts or videos recorded by users, and send the relevant videos of users to the encoding module.

[0061] The encoding module is used to encode the video collected by the acquisition module to obtain the encoded multiple video frames (or pictures or images) corresponding to the video and the encoding information corresponding to the multiple video frames. Afterwards, the encoding module can generate a corresponding code stream according to the encoded multiple video frames and the encoding information corresponding to the multiple video frames, and can send the code stream to the server 200. Optionally, the code stream carries the encoded multiple video frames and the encoding information corresponding to the multiple video frames. For example, taking the acquisition module to collect video A, video A includes 5 video frames (such as video frame A1, video frame A2, video frame A3, video frame A4 and video frame A5) as an example, the encoding module can encode the 5 video frames included in video A respectively, and obtain the encoded 5 video frames and the encoding information corresponding to the 5 video frames. Afterwards, the encoding module can generate a corresponding code stream according to the encoded 5 video frames and the encoding information corresponding to the 5 video frames, and send the code stream to the server 200. Optionally, the code stream carries the encoded 5 video frames and the encoding information corresponding to the 5 video frames. Exemplarily, the encoding information corresponding to any video frame may include at least one of the following: block division information, motion vector (or motion vector) information, prediction mode, position information of reference video frame or block residual information, etc.

[0062] The decoding module is used to decode the bitstream (such as a standard bitstream or a private bitstream) from the server end 200. For example, the decoding module decodes the standard bitstream from the server end 200 to obtain a plurality of video frames belonging to normal encoding frames, and the decoding module decodes the private bitstream from the server end 200 to obtain attribute information of at least one video frame belonging to a discarded frame.

[0063] The video frame insertion module is used to generate at least one frame to be inserted according to the attribute information of each video frame belonging to the discarded frame decoded by the decoding module and the video frames belonging to the normal coded frames near the playback position of each video frame belonging to the discarded frame, and generate a complete video according to the at least one frame to be inserted and multiple video frames belonging to the normal coded frames.

[0064] The server end 200 may refer to a device (such as a cloud server or cloud computing device) that provides services for users to perform corresponding operations on the client end 100. Exemplarily, the server end 200 may be a cloud server (or may be referred to as a cloud, server end or cloud computing device) for providing cloud services, cloud computing, cloud storage, cloud communications, network services, security services, and cloud computing services such as big data, or may also be a common data center or server or other forms of computing devices.

[0065] Exemplarily, the server side 200 may include a decoding module, a video characteristic analysis module, a time domain downsampling module, an interpolation frame quality feedback module and an encoding module. It should be noted that: Figure 1 The connection relationship between the various functional modules in the server side 200 is only an example and does not constitute a limitation of the present application. The functions of the various functional modules included in the server side 200 are described below.

[0066] The decoding module is used to decode the code stream from the client 100 to obtain multiple video frames and encoding information corresponding to the multiple video frames.

[0067] The video characteristic analysis module is used to analyze the encoding information corresponding to the multiple video frames to obtain the video frame analysis results. Optionally, the video frame analysis results include the attributes of each video frame in the multiple video frames (i.e., whether the video frame belongs to a discarded frame or a normal coded frame). Exemplarily, the video frame analysis results may include but are not limited to which video frames belong to discarded frames (or which video frames are marked as discarded frames or which video frames have discarded attributes) or which video frames belong to normal coded frames (or which video frames are marked as normal coded frames or which video frames have reserved attributes, i.e., video frames that need to be retained for subsequent encoding), etc. For example, continuing to take the above-mentioned video A as an example, the video characteristic analysis module obtains the video frame analysis results corresponding to video A after analyzing and counting the encoding information corresponding to the 5 video frames included in video A. Exemplarily, the video frame analysis results corresponding to video A include that video frames A3 and A5 belong to discarded frames and video frames A1, A2, and A4 belong to normal coded frames.

[0068] Optionally, the video characteristic analysis module is also used to determine, for each video frame that is a discarded frame, motion vector information (or may be referred to as motion vector information) of multiple coding blocks corresponding to the video according to a prediction mode of multiple coding blocks corresponding to the video frame.

[0069] The time domain downsampling module is used to discard the video frames that are discarded frames and retain the video frames that are normal coded frames according to the video frame analysis results of the video characteristic analysis module, so as to ensure the quality of video interpolation. For example, continuing to take the above video A as an example, after obtaining the attributes of the 5 video frames included in video A, the time domain downsampling module can discard video frames A3 and video frames A5 according to the attributes of the 5 video frames, and retain video frames A1, video frames A2 and video frames A4.

[0070] The interpolation quality feedback module is used to provide feedback to the video characteristic analysis module, thereby changing the influence strategy of the video characteristic analysis module on other modules. Because a fixed video processing scheme may not have good robustness, during the execution of the video processing scheme, by dynamically adjusting the scheme strategy (such as the frame dropping strategy) according to the actual situation of the video sequence encountered, the video processing scheme can have a better processing effect when facing most scenes. Optionally, when determining that the decoding module decodes each video frame that meets the first number threshold (such as 50 or an integer multiple of 50 or other numbers, etc.), the interpolation quality feedback module can generate a to-be-interpolated frame corresponding to the video frame belonging to the discarded frame according to the motion vector information of a video frame belonging to the discarded frame included in the video frame of the first number threshold and two video frames belonging to the normal coding frames located before and after the video frame belonging to the discarded frame. Afterwards, the interpolation quality feedback module can determine the peak signal-to-noise ratio (PSNR) corresponding to the to-be-interpolated frame based on the to-be-interpolated frame and the video frame belonging to the discarded frame.

[0071] In one example, when the peak signal-to-noise ratio is greater than or equal to a set threshold, the interpolation quality feedback module reduces the second threshold (such as Th_intra) and sends the reduced second threshold to the video characteristic analysis module. The video characteristic analysis module needs to use the first threshold and the second threshold when analyzing the encoding information corresponding to multiple video frames. In another example, when the peak signal-to-noise ratio is greater than or equal to the set threshold, the interpolation quality feedback module generates indication information and sends the indication information to the video characteristic analysis module. The indication information is used to indicate that the second threshold is reduced. In another example, when the peak signal-to-noise ratio is less than the set threshold and the number of video frames belonging to discarded frames included in the video frames of the first number threshold is less than the second number threshold, the interpolation quality feedback module increases the second threshold (such as Th_intra) and reduces the first threshold (such as Th_skip), and sends the reduced first threshold and the increased second threshold to the video characteristic analysis module. The second number threshold is less than the first number threshold. In another example, when the peak signal-to-noise ratio is less than a set threshold and the number of video frames that are discarded frames included in the video frames of the first number threshold is less than the second number threshold, the interpolation frame quality feedback module generates indication information and sends the indication information to the video characteristic analysis module. The indication information is used to indicate that the first threshold is reduced and the second threshold is increased.

[0072] The encoding module is used to perform normal encoding operations on multiple video frames belonging to normal encoding frames to obtain a standard bitstream, and transmit the standard bitstream to the server end 200. Exemplarily, the standard bitstream can carry multiple encoded video frames belonging to normal encoding frames. It can be understood that after the time domain downsampling operation, the video (or it can be called a video sequence including multiple video frames) becomes a new video sequence with fewer video frames. After the new video sequence is subjected to standard encoding, it forms a standard bitstream and is transmitted to the server end 200. In addition, the encoding module is also used to perform encoding operations on the attribute information of at least one video frame belonging to a discarded frame to obtain a private bitstream, and transmit the private bitstream to the server end 200.

[0073] Optionally, the video processing method provided in the embodiment of the present application can be applied to live transcoding scenarios based on end-cloud collaboration, and can also be applied to other transcoding scenarios (such as video recording transcoding scenarios, video on demand transcoding scenarios, or video broadcast transcoding scenarios, etc.).

[0074] It should be noted that Figure 1 Only one possible application scenario is provided schematically, and the schematic application scenario is intended to more clearly illustrate the technical solution of the embodiment of the present application, and does not limit the application scenario of the video processing method provided by the present application. Figure 1 The form and quantity of each module in the application scenario shown are only for example and do not constitute a limitation of the present application. Figure 1 The names of the modules in the application scenarios shown are only examples, and the names of the modules in the specific implementation may also be other names, which are not specifically limited in this application. In addition, it is known to those skilled in the art that with the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0075] The following is based on Figure 1 The application scenario shown is used to introduce the specific implementation of the video processing method in the embodiment of the present application in detail.

[0076] Figure 2 A schematic diagram showing a flow chart of a video processing method provided by an embodiment of the present application is exemplified. The method can be but is not limited to being applicable to Figure 1The application scenario illustrated. The method flow may be implemented by data interaction between multiple video processing devices (such as a first video processing device and a second video processing device). Optionally, the first video processing device may be a server (such as a cloud server or a data center) or a component (such as a plug-in, component, chip or circuit, etc.) that can support the server to implement the functions required by the method, and the second video processing device may be a client or a component (such as a plug-in, component, chip or circuit, etc.) that can support the client to implement the functions required by the method. Exemplarily, the first video processing device may be Figure 1 For example, the server end 200 may be a computing device with a video processing function or a computing device cluster with a video processing function, and the second video processing device may be Figure 1 The client 100 shown in the figure has a video processing function. For example, the terminal device installed with the client 100 can be a computing device. For example, taking the server 200 as a computing device with a video processing function, the first video processing device can be the computing device with the video processing function, or it can also be an independent unit embedded in the computing device for realizing the video processing function, or it can also be a functional component (such as a chip) encapsulated in the computing device for realizing the video processing function. In order to facilitate the introduction of the technical solution provided in the embodiment of the present application, the following takes the first video processing device as the server and the second video processing device as the client as an example to introduce the process of realizing the video processing method through data interaction between the first video processing device and the second video processing device. Figure 2 As shown, the method includes:

[0077] Step 201: The server decodes the first bit stream to obtain a plurality of video frames and encoding information of the plurality of video frames included in the first video.

[0078] Optionally, after acquiring the first bitstream, the server may decode the first bitstream to obtain multiple video frames and encoding information of the multiple video frames included in the first video. Exemplarily, the first bitstream may be sent by a user through a client, or may be sent by a user through a device.

[0079] Exemplarily, the first code stream may carry multiple video frames included in the first video and the encoding information of the multiple video frames. Optionally, the first code stream may also carry the position information of the multiple video frames included in the first video. For example, the position information of any video frame may include the playback order (or may be called the playback sequence number) of the video frame in the first video, the playback time or the playback index, and other information that can characterize the playback position. Exemplarily, the encoding information of any video frame may include the prediction mode (such as the inter-frame prediction mode or the intra-frame prediction mode) of the multiple coding blocks corresponding to the video frame, the block division information, the block residual information, the positions of the multiple coding blocks in the video frame, etc. Optionally, the encoding information of one or several coding blocks among the multiple coding blocks corresponding to the video frame may also include the motion vector information of the one or several coding blocks and the position information of the reference video frame of the one or several coding blocks. Among them, the reference video frame of a certain coding block may refer to the video frame required to refer to determine the motion vector information of the coding block. In other words, there are one or more coding blocks in the reference video frame that are similar to the pixel values ​​of the coding block. In one example, when the prediction mode of a coding block is an inter-frame prediction mode, the coding information of the coding block may include the motion vector information of the coding block and the position information of the reference video frame of the coding block. In another example, when the prediction mode of a coding block is an intra-frame prediction mode, the coding information of the coding block does not include the motion vector information of the coding block and the position information of the reference video frame of the coding block.

[0080] In the case where users have higher and higher requirements for video quality, in order to meet users' increasingly high pursuit of high-fidelity, more realistic video experience, and to solve the problem that low frame rate will cause obvious freezes in some intense sports scenes, affecting people's visual experience, the embodiment of the present application is achieved by using the video processing technology on the server side. In addition, by using the video processing technology on the server side, not only can the smoothness of the video picture be improved, thereby improving the user's audio and video experience, but also the cost of introducing professional equipment and manual processing can be reduced.

[0081] For example, in a live transcoding scenario, the first bitstream comes from a client (such as a short video client). To ensure that the live video can be viewed clearly and smoothly by users in a weak network environment or when the network signal is stuck, the client can encode the collected live video to form a first bitstream. The first bitstream carries multiple video frames and encoding information of multiple video frames included in the live video. Afterwards, the client can transmit the first bitstream to the server so that the server can process the first bitstream accordingly.

[0082] Optionally, when encoding the collected live video, the client may divide each video frame included in the live video into multiple blocks. Afterwards, the client may encode the multiple blocks corresponding to the video frame to obtain multiple coding blocks corresponding to the video frame and coding information of the multiple coding blocks. In this way, the first bitstream actually carries multiple coding blocks corresponding to each of the multiple video frames and coding information of the multiple coding blocks corresponding to each of the multiple video frames.

[0083] For example, the live video collected by the client is video 1, and video 1 includes 4 video frames (such as video frame 1, video frame 2, video frame 3 and video frame 4). Assume that the client divides video frame 1 into 2 blocks (such as block 11 and block 12), divides video frame 2 into 3 blocks (such as block 21, block 22 and block 23), divides video frame 3 into 3 blocks (such as block 31, block 32 and block 33), and divides video frame 4 into 2 blocks (such as block 41 and block 42). Afterwards, for video frame 1, the client can encode blocks 11 and 12 corresponding to video frame 1 respectively, and obtain encoding block 11, encoding block 12, encoding information of encoding block 11, and encoding information of encoding block 12. For video frame 2, the client may encode blocks 21, 22, and 23 corresponding to video frame 2, respectively, to obtain encoded blocks 21, encoded blocks 22, encoded blocks 23, encoding information of encoded blocks 21, encoding information of encoded blocks 22, and encoding information of encoded blocks 23. For video frame 3, the client may encode blocks 31, 32, and 33 corresponding to video frame 3, respectively, to obtain encoded blocks 31, 32, 33, encoding information of encoded blocks 31, encoding information of encoded blocks 32, and encoding information of encoded blocks 33. For video frame 4, the client may encode blocks 41 and 42 corresponding to video frame 4, respectively, to obtain encoded blocks 41, 42, encoding information of encoded blocks 41, and encoding information of encoded blocks 42. Then, the client may send the first code stream to the server. Optionally, the first code stream carries coding block 11, coding block 12, coding information of coding block 11, and coding information of coding block 12 corresponding to video frame 1, coding block 21, coding block 22, coding block 23, coding information of coding block 21, coding information of coding block 22, and coding information of coding block 23 corresponding to video frame 2, coding block 31, coding block 32, coding block 33, coding information of coding block 31, coding information of coding block 32, and coding information of coding block 33 corresponding to video frame 3, and coding block 41, coding block 42, coding information of coding block 41, and coding information of coding block 42 corresponding to video frame 4.

[0084] Step 202: The server determines at least one first target video frame included in the multiple video frames according to the encoding information of the multiple video frames.

[0085] After receiving the first code stream, the server decodes the first code stream to obtain multiple coding blocks corresponding to each video frame in the multiple video frames and coding information of the multiple coding blocks corresponding to each video frame. Optionally, when the first code stream also carries the position information of each video frame in the multiple video frames, the server decodes the first code stream to obtain the position information of each video frame in the first video. Afterwards, the server can determine the attributes of the first video frame for any video frame in the multiple video frames (such as the first video frame) according to the coding information of the multiple coding blocks corresponding to the first video frame, that is, whether the first video frame is a first target video frame or a second target video frame (or it can also be understood as whether the first video frame has a discarding feature (that is, it is more suitable to be discarded as a frame to be inserted) or has a retaining feature). Among them, the first target video frame can refer to a video frame that needs to be discarded; the second target video frame (or it can be understood as a video frame belonging to a normal coding frame) can refer to a video frame that needs to be retained.

[0086] Optionally, the server side may determine the texture complexity index value of the first video frame based on the encoding information of the first video frame. The first video frame is any one of the multiple video frames. Afterwards, the server side may determine whether the first video frame is the first target video frame by judging whether the texture complexity index value of the first video frame meets the texture complexity threshold. When the texture complexity index value of the first video frame meets the texture complexity threshold, the server side may determine that the first video frame is the first target video frame. Exemplarily, the texture complexity index may include but is not limited to block partition depth, block residual, absolute value of the horizontal and vertical coordinates of the motion vector of the entire coding block, ratio of discrete cosine transform (Discrete cosine transform) 2, first ratio or second ratio, etc.

[0087] For example, a video frame with a small block division depth is more suitable as the first target video frame; a video frame with a smaller sum of the absolute values ​​of the horizontal and vertical coordinates of the motion vector of the entire block is more suitable as the first target video frame; a video frame with a smaller block residual is more suitable as the first target video frame. In addition, there are also transformation methods including coding blocks that can be used to determine which video frames are more suitable as the first target video frame. For example, blocks with a higher proportion of DCT2 are theoretically more suitable for interpolation. It should be noted that the above-mentioned multiple texture complexity indicators can be combined with each other to make decisions, or these texture complexity indicators can be processed in some special ways, and then some indicators can be used to measure whether they are more suitable for interpolation. In summary, this information can theoretically reflect that the image texture information is relatively simple, and simple texture information means that it is easier to interpolate. By analogy, there is an image with only one circle and a landscape painting. The image of the circle with simple texture is definitely easier to interpolate.

[0088] Based on the above content, the following takes the texture complexity threshold including the first threshold and / or the second threshold as an example to introduce the implementation process of the server side determining the attribute of the first video frame through the following possible implementation methods.

[0089] Implementation method 1: When the first ratio corresponding to the first video frame is greater than the first threshold (such as Th_skip) and / or the second ratio corresponding to the first video frame is less than the second threshold (such as Th_intra), the server can determine that the first video frame is the first target video frame.

[0090] Among them, the first ratio can be used to indicate the ratio of the number of pixels in the area where the coding block in the first video frame is located whose prediction mode is the inter-frame prediction mode (such as the Skip mode), and the second ratio can be used to indicate the ratio of the number of pixels in the area where the coding block in the first video frame is located whose prediction mode is the intra-frame prediction mode (such as the Intra mode).

[0091] The following is an introduction to the implementation process of the server side calculating the first ratio and the second ratio corresponding to the first video frame: the server side can first determine the first number of multiple pixels included in the area where the coding block whose prediction mode is the inter-frame prediction mode in the first video frame is located, and can determine the second number of multiple pixels included in the area where the coding block whose prediction mode is the intra-frame prediction mode in the first video frame is located. Afterwards, the server side can use the ratio of the first number to the total number of multiple pixels included in the first video frame as the first ratio, and can use the ratio of the second number to the total number of multiple pixels included in the first video frame as the second ratio.

[0092] Implementation method two: When the first ratio corresponding to the first video frame is less than or equal to the first threshold (such as Th_skip), or the second ratio corresponding to the first video frame is greater than or equal to the second threshold (such as Th_intra), or the first ratio corresponding to the first video frame is greater than the first threshold and the second ratio corresponding to the first video frame is greater than or equal to the second threshold, the server can determine that the first video frame is the second target video frame.

[0093] Exemplarily, based on the above-mentioned implementation method 1 and implementation method 2, the implementation process of determining the attributes of each video frame included in the video 1 on the server side is introduced below by taking the above-mentioned video 1 as an example.

[0094] For video frame 1: Take the prediction mode of coding block 11 corresponding to the above video frame 1 as Skip mode, and the prediction mode of coding block 12 as Intra mode as an example. For coding block 11, the server side can count the number of pixels a1 included in the area corresponding to coding block 11 in video frame 1. For coding block 12, the server side can count the number of pixels a2 included in the area corresponding to coding block 12 in video frame 1. Afterwards, the server side can calculate the first ratio corresponding to video frame 1 as a1 / (a1+a2), and can calculate the second ratio corresponding to video frame 1 as a2 / (a1+a2).

[0095] In one example, when the first ratio a1 / (a1+a2) corresponding to video frame 1 is greater than Th_skip and / or the second ratio a2 / (a1+a2) corresponding to video frame 1 is less than Th_intra, the server side may determine that video frame 1 is the first target video frame. In another example, when the first ratio a1 / (a1+a2) corresponding to video frame 1 is less than or equal to Th_skip, or the second ratio a2 / (a1+a2) corresponding to video frame 1 is greater than or equal to Th_intra, or the first ratio a1 / (a1+a2) corresponding to video frame 1 is greater than Th_skip and the second ratio a2 / (a1+a2) corresponding to video frame 1 is greater than or equal to Th_intra, the server side may determine that video frame 1 is the second target video frame.

[0096] For video frame 2: Take the prediction mode of coding block 21 corresponding to the above video frame 2 as Skip mode, the prediction mode of coding block 22 as Intra mode, and the prediction mode of coding block 23 as Skip mode as an example. For coding block 21, the server side can count the number of pixels b1 included in the area corresponding to coding block 21 in video frame 2. For coding block 22, the server side can count the number of pixels b2 included in the area corresponding to coding block 22 in video frame 2. For coding block 23, the server side can count the number of pixels b3 included in the area corresponding to coding block 23 in video frame 2. Afterwards, the server side can calculate the number of pixels included in the area corresponding to the coding block whose prediction mode is Skip mode in video frame 2 as (b1+b3), and can calculate the number of pixels included in the area corresponding to the coding block whose prediction mode is Intra mode in video frame 2 as b2. Then, the server may calculate the first ratio corresponding to the video frame 2 as (b1+b3) / (b1+b2+b3), and may calculate the second ratio corresponding to the video frame 2 as b2 / (b1+b2+b3).

[0097] In one example, when the first ratio (b1+b3) / (b1+b2+b3) corresponding to video frame 2 is greater than Th_skip and / or the second ratio b2 / (b1+b2+b3) corresponding to video frame 2 is less than Th_intra, the server side may determine that video frame 2 is the first target video frame. In another example, when the first ratio (b1+b3) / (b1+b2+b3) corresponding to video frame 2 is less than or equal to Th_skip, or the second ratio b2 / (b1+b2+b3) corresponding to video frame 2 is greater than or equal to Th_intra, or the first ratio (b1+b3) / (b1+b2+b3) corresponding to video frame 2 is greater than Th_skip and the second ratio b2 / (b1+b2+b3) corresponding to video frame 2 is greater than or equal to Th_intra, the server side may determine that video frame 2 is the second target video frame.

[0098] For video frame 3: Take the prediction mode of coding block 31 corresponding to the above video frame 3 as Intra mode, the prediction mode of coding block 32 as Intra mode, and the prediction mode of coding block 33 as Skip mode as an example. For coding block 31, the server side can count the number of pixels c1 included in the area corresponding to coding block 31 in video frame 3. For coding block 32, the server side can count the number of pixels c2 included in the area corresponding to coding block 32 in video frame 3. For coding block 33, the server side can count the number of pixels c3 included in the area corresponding to coding block 33 in video frame 3. Afterwards, the server side can calculate the number of pixels c3 included in the area corresponding to the coding block whose prediction mode is Skip mode in video frame 3, and can calculate the number of pixels (c1+c2) included in the area corresponding to the coding block whose prediction mode is Intra mode in video frame 3. Then, the server may calculate the first ratio corresponding to the video frame 3 as c3 / (c1+c2+c3), and may calculate the second ratio corresponding to the video frame 3 as (c1+c2) / (c1+c2+c3).

[0099] In one example, when the first ratio c3 / (c1+c2+c3) corresponding to video frame 3 is greater than Th_skip and / or the second ratio (c1+c2) / (c1+c2+c3) corresponding to video frame 3 is less than Th_intra, the server side may determine that video frame 3 is the first target video frame. In another example, when the first ratio c3 / (c1+c2+c3) corresponding to video frame 3 is less than or equal to Th_skip, or the second ratio (c1+c2) / (c1+c2+c3) corresponding to video frame 3 is greater than or equal to Th_intra, or the first ratio c3 / (c1+c2+c3) corresponding to video frame 3 is greater than Th_skip and the second ratio (c1+c2) / (c1+c2+c3) corresponding to video frame 3 is greater than or equal to Th_intra, the server side may determine that video frame 3 is the second target video frame.

[0100] For video frame 4: Take the prediction mode of coding block 41 corresponding to the above video frame 4 as Intra mode, and the prediction mode of coding block 42 as Skip mode as an example. For coding block 41, the server side can count the number of pixels d1 included in the area corresponding to coding block 41 in video frame 4. For coding block 42, the server side can count the number of pixels d2 included in the area corresponding to coding block 42 in video frame 4. Afterwards, the server side can calculate the first ratio corresponding to video frame 4 as d2 / (d1+d2), and can calculate the second ratio corresponding to video frame 4 as d1 / (d1+d2).

[0101] In one example, when the first ratio d2 / (d1+d2) corresponding to video frame 4 is greater than Th_skip and / or the second ratio d1 / (d1+d2) corresponding to video frame 4 is less than Th_intra, the server side may determine that video frame 4 is the first target video frame. In another example, when the first ratio d2 / (d1+d2) corresponding to video frame 4 is less than or equal to Th_skip, or the second ratio d1 / (d1+d2) corresponding to video frame 4 is greater than or equal to Th_intra, or the first ratio d2 / (d1+d2) corresponding to video frame 4 is greater than Th_skip and the second ratio d1 / (d1+d2) corresponding to video frame 4 is greater than or equal to Th_intra, the server side may determine that video frame 4 is the second target video frame.

[0102] Step 203: The server sends a second code stream to the client, and the client receives the second code stream.

[0103] Exemplarily, the second code stream may carry multiple second target video frames.

[0104] Optionally, after determining the attributes of each of the multiple video frames included in the first video, the server can perform a time domain downsampling operation on the first video according to the attributes of each of the multiple video frames, that is, discard at least one first target video frame included in the first video, and retain the multiple second target video frames included in the first video. The retained multiple second target video frames can be used as a new video sequence through normal encoding operations to form a code stream and transmit it to the server; the attribute information of the discarded at least one first target video frame can also be encoded to form another code stream and transmit it to the server.

[0105] In a possible implementation, after encoding multiple second target video frames to obtain a second bitstream, the server may send the second bitstream to the client.

[0106] In another possible implementation, in order to assist the client to generate the interpolated frames more accurately, so as to ensure the quality of the interpolated frames, the server can send a third stream to the client in addition to sending the second code stream to the client. The third code stream can carry the attribute information of at least one first target video frame. For example, the attribute information of any first target video frame may include but is not limited to the motion vector information of multiple coding blocks corresponding to the first target video frame and the position information of the first target video frame in the first video. Optionally, the attribute information of any first target video frame may also include the percentage of the number of pixels in the area where the coding block whose prediction mode is the inter-frame prediction mode in the first target video frame is located, the percentage of the number of pixels in the area where the coding block whose prediction mode is the intra-frame prediction mode in the first target video frame is located, the first threshold or the second threshold, etc. Optionally, the third code stream may also carry the server-side frame loss strategy for the first video, or may also carry the discarding rule (or discarding strategy) of at least one first target video frame. For example, the frame loss strategy may include information on how at least one first target video frame is discarded.

[0107] The following describes the implementation process of the server sending the second code stream to the client.

[0108] The server side can encode the retained multiple second target video frames to form a second code stream. Then, the server side can send the second code stream to the client. Exemplarily, the second code stream carries the encoded multiple second target video frames. Optionally, the second code stream can also carry the position information of the multiple second target video frames in the first video.

[0109] For example, let's continue to take the above-mentioned video 1 as an example. In one example, it is assumed that the video frame 1 included in the video 1 is the second target video frame, the video frame 2 is the first target video frame, the video frame 3 is the second target video frame, and the video frame 4 is the second target video frame. The server side can use the retained video frame 1, video frame 3, and video frame 4 as a new video sequence, and can use standard encoding technology to encode the new video sequence into a standard bit stream (i.e., the second bit stream). Then, the server side can send the standard bit stream to the client. Optionally, the standard bit stream carries the encoded video frame 1, the encoded video frame 3, and the encoded video frame 4.

[0110] In another example, it is assumed that the video frame 1 included in the video 1 is the second target video frame, the video frame 2 is the first target video frame, the video frame 3 is the first target video frame, and the video frame 4 is the second target video frame. The server can use the retained video frame 1 and the video frame 4 as a new video sequence, and can use standard encoding technology to encode the new video sequence into a standard bit stream (i.e., the second bit stream). Then, the server can send the standard bit stream to the client. Optionally, the standard bit stream carries the encoded video frame 1 and the encoded video frame 4.

[0111] Optionally, after executing step 203, the server may also execute step 204. By executing step 204, the quality of video frame insertion of the client can be further guaranteed, and the phenomenon that the quality of the generated frames to be inserted is poor due to inaccurate estimation of the motion vector information of some video frames with more intense motion after being discarded can be alleviated. For example, when the server executes step 203 but does not execute step 204, the client can generate the second video according to the multiple second target video frames carried by the second stream after receiving the second bitstream. When the server executes step 203 and also executes step 204, the client can generate the second video according to the attribute information of the multiple second target video frames carried by the second bitstream and the at least one first target video frame carried by the third bitstream after receiving the second bitstream and the third bitstream respectively.

[0112] Step 204: The server sends the third code stream to the client, and the client receives the third code stream.

[0113] The following describes the implementation process of the server sending the third code stream to the client:

[0114] The server may encode the attribute information of the at least one discarded first target video frame to form a third code stream, and then the server may send the third code stream to the client.

[0115] Exemplarily, the following describes an implementation process of determining the motion vector information of each coding block corresponding to any first target video frame on the server side.

[0116] For a third target video frame included in the first video, the server can identify a prediction mode of each coding block in the multiple coding blocks according to coding information of the multiple coding blocks corresponding to the third target video frame. The third target video frame can be any one of the at least one first target video frame.

[0117] When the prediction mode of the first coding block included in the multiple coding blocks is the inter-frame prediction mode, the server side can obtain the motion vector information of the first coding block from the coding information of the first coding block, and can scale the motion vector information of the first coding block to obtain the scaled motion vector information of the first coding block.

[0118] For example, the server side can obtain the motion vector information of the first coding block from the coding information of the first coding block, and can obtain the position information of the reference video frame of the first coding block from the coding information of the first coding block. Wherein, the first coding block can be any one of the multiple coding blocks corresponding to the third target video frame. Afterwards, the server side can calculate the playback interval between the reference video frame of the first coding block and the third target video frame according to the position information of the reference video frame of the first coding block and the position information of the third target video frame, and can calculate the playback interval between the third target video frame and the second target video frame adjacent to the third target video frame according to the position information of the third target video frame and the position information of the second target video frame adjacent to the third target video frame. Then, the server side can scale the motion vector information of the first coding block according to the playback interval between the reference video frame of the first coding block and the third target video frame and the playback interval between the third target video frame and the second target video frame adjacent to the third target video frame, and obtain the scaled motion vector information. Wherein, the scaled motion vector information can be used as the current motion vector information of the first coding block. Exemplarily, the play interval may be one of the following: a play order difference, a play index difference, or a play time difference, etc.

[0119] When the prediction mode of the first coding block is the intra-frame prediction mode, the server side cannot obtain the motion vector information of the first coding block from the coding information of the first coding block. Of course, the server side will not obtain the position information of the reference video frame of the first coding block from the coding information of the first coding block. At this time, the server side can determine whether there is a second coding block with an inter-frame prediction mode on the left, above, or upper left of the first coding block. When there is a second coding block with an inter-frame prediction mode on the left, above, or upper left of the first coding block, the server side can obtain the motion vector information of the second coding block from the coding information of the second coding block, and can scale the motion vector information of the second coding block to obtain the scaled motion vector information, which is used as the current motion vector information of the first coding block.

[0120] For example, the server side may first determine whether there is a coding block whose prediction mode is the inter-frame prediction mode in the three positions of the left side, the top and the upper left of the first coding block. If there is a coding block whose prediction mode is the inter-frame prediction mode in one of the three positions of the left side, the top and the upper left of the first coding block, the server side may obtain the motion vector information of the coding block at the position from the coding information of the coding block at the position, and may obtain the position information of the reference video frame of the coding block at the position from the coding information of the coding block at the position. Afterwards, the server side may calculate the playback interval between the reference video frame of the coding block at the position and the third target video frame based on the position information of the reference video frame of the coding block at the position and the position information of the third target video frame, and may calculate the playback interval between the third target video frame and a second target video frame located adjacent to the third target video frame based on the position information of the third target video frame and the position information of a second target video frame located adjacent to the third target video frame. Then, the server can scale the motion vector information of the second coding block according to the playback interval between the reference video frame of the coding block at the position and the third target video frame and the playback interval between the third target video frame and a second target video frame adjacent to the third target video frame, and obtain the scaled motion vector information corresponding to the coding block at the position. The scaled motion vector information corresponding to the coding block at the position can be used as the current motion vector information of the first coding block.

[0121] If there are multiple coding blocks whose prediction mode is the inter-frame prediction mode in the three positions of the left side, the top and the top left of the first coding block, the server side can select a coding block at one position from the coding blocks at the multiple positions, and can obtain the motion vector information of the coding block at the position from the coding information of the coding block at the position. After that, the server side can use the motion vector information of the coding block at the position to participate in determining the current motion vector information of the first coding block. After the server side selects a coding block at a certain position from the coding blocks at multiple positions, it can also obtain the position information of the reference video frame of the coding block at the position from the coding information of the coding block at the position. After that, the server side can calculate the playback interval between the reference video frame of the coding block at the position and the third target video frame based on the position information of the reference video frame of the coding block at the position and the position information of the third target video frame, and can calculate the playback interval between the third target video frame and the second target video frame located adjacent to the third target video frame based on the position information of the third target video frame and the position information of a second target video frame located adjacent to the third target video frame. Then, the server can scale the motion vector information of the second coding block according to the playback interval between the reference video frame of the coding block at the position and the third target video frame and the playback interval between the third target video frame and a second target video frame adjacent to the third target video frame, and obtain the scaled motion vector information corresponding to the coding block at the position. The scaled motion vector information corresponding to the coding block at the position can be used as the current motion vector information of the first coding block.

[0122] If there is no coding block whose prediction mode is the inter-frame prediction mode in the coding blocks at at least one position, the server side may use (0,0) as the current motion vector information of the first coding block.

[0123] When there are no coding blocks at the left side, above, and upper left of the first coding block, the server side may use (0,0) as the current motion vector information of the first coding block.

[0124] After obtaining the current motion vector information of the multiple coding blocks corresponding to each of the at least one first target video frame, the server can generate a third bitstream according to the current motion vector information of the multiple coding blocks corresponding to each of the at least one first target video frame and the position information of the at least one first target video frame in the first video. Then, the server can send the third bitstream to the client.

[0125] For example, let's continue to take the above video 1 as an example. In one example, it is assumed that the video frame 2 included in the video 1 is the first target video frame. The server side can encode the current motion vector information of the coding block 21, coding block 22, and coding block 23 corresponding to the video frame 2 and the position information of the video frame 2 in the video 1 into a private code stream (i.e., the third code stream). Afterwards, the server side can send the private code stream to the client. Among them, the private code stream can carry the current motion vector information of the encoded coding block 21, coding block 22, and coding block 23 and the position information of the encoded video frame 2 in the video 1.

[0126] In another example, it is assumed that the video frame 2 included in the video 1 is the first target video frame, and the video frame 3 is the first target video frame. The server side can encode the current motion vector information of the coding block 21, coding block 22, and coding block 23 corresponding to the video frame 2, the position information of the video frame 2 in the video 1, the current motion vector information of the coding block 31, coding block 32, and coding block 33 corresponding to the video frame 3, and the position information of the video frame 3 in the video 1 into a private bitstream. Afterwards, the server side can send the private bitstream to the client side. The private bitstream can carry the current motion vector information of the encoded coding block 21, coding block 22, and coding block 23, the position information of the encoded video frame 2 in the video 1, the current motion vector information of the encoded coding block 31, coding block 32, and coding block 33, and the position information of the encoded video frame 3 in the video 1.

[0127] For example, see below Figure 3 , taking the first target video frame as the video frame 2 included in the above video 1 as an example, the implementation process of determining the motion vector information of each coding block corresponding to the video frame 2 by the server is introduced. It is assumed that the multiple coding blocks corresponding to the video frame 2 are coding blocks 21, coding blocks 22, and coding blocks 23, the prediction mode of coding block 21 is the Skip mode, the prediction mode of coding block 22 is the Intra mode, and the prediction mode of coding block 23 is the Skip mode, and it is assumed that the video frame 1 is the second target video frame, the video frame 3 is the second target video frame, and the video frame 4 is the second target video frame. Figure 3 As shown, the method includes:

[0128] Step 301: The server side obtains the prediction mode of each coding block in multiple coding blocks corresponding to video frame 2.

[0129] Optionally, the server can obtain the prediction mode of coding block 21 as Skip mode, the prediction mode of coding block 22 as Intra mode, and the prediction mode of coding block 23 as Skip mode based on the coding information of coding block 21, the coding information of coding block 22, and the coding information of coding block 23 corresponding to video frame 2.

[0130] Step 302: The server determines whether the prediction mode of the first coding block included in the multiple coding blocks corresponding to the video frame 2 is the Skip mode. If yes, execute step 303; if no, execute step 304.

[0131] Step 303: The server side obtains the motion vector information of the first coding block, and scales the motion vector information of the first coding block according to the playback interval between the reference video frame of the first coding block and video frame 2 and the playback interval between video frame 2 and video frame 1 to obtain the scaled motion vector information.

[0132] Optionally, the video frame 1 is a second target video frame that is located before and adjacent to the video frame 2 .

[0133] For example, when the first coding block is coding block 21, the server side may determine that the prediction mode of coding block 21 is Skip mode. Afterwards, the server side may obtain the motion vector information of coding block 21 from the coding information of coding block 21, and may scale the motion vector information of coding block 21 according to the playback interval between the reference video frame of coding block 21 and video frame 2 and the playback interval between video frame 2 and video frame 1, to obtain scaled motion vector information, and the scaled motion vector information is used as the current motion vector information of coding block 21.

[0134] For another example, when the first coding block is coding block 22, the server may determine that the prediction mode of coding block 22 is Intra mode. In this case, the server does not execute the above step 303, but executes step 304.

[0135] For another example, when the first coding block is coding block 23, the server side may determine that the prediction mode of coding block 23 is Skip mode. Afterwards, the server side may obtain the motion vector information of coding block 23 from the coding information of coding block 23, and may scale the motion vector information of coding block 23 according to the playback interval between the reference video frame of coding block 23 and video frame 2 and the playback interval between video frame 2 and video frame 1, to obtain scaled motion vector information, and the scaled motion vector information is used as the current motion vector information of coding block 23.

[0136] Step 304: When the server determines that the prediction mode of the first coding block is the Intra mode, it determines whether there is a second coding block with a prediction mode of the Skip mode to the left, above, or above the left of the first coding block. If yes, execute step 305; if not, execute step 306.

[0137] Step 305: The server side obtains the motion vector information of the second coding block from the coding information of the second coding block, and scales the motion vector information of the second coding block according to the playback interval between the reference video frame of the second coding block and video frame 2 and the playback interval between video frame 2 and video frame 1 to obtain the scaled motion vector information corresponding to the second coding block.

[0138] Exemplarily, taking the first coding block as coding block 22 as an example, when the server determines that the prediction mode of coding block 22 is Intra mode, the server performs the above step 304. If there are one or more coding blocks whose prediction mode is inter-frame prediction mode in the coding blocks located on the left, above, or above the left of coding block 22, the server can perform step 305. If there is no coding block whose prediction mode is inter-frame prediction mode in the coding blocks located on the left, above, and above the left of coding block 22, the server can perform step 306.

[0139] The following describes the implementation process of the server side executing the above step 305 through the following possible examples.

[0140] Example 1: The coding block located on the left side (or above or upper left) of coding block 22 is coding block 21, and the coding block located on the right side (or upper right or lower right or lower or lower left) of coding block 22 is coding block 23. When the server determines that the prediction mode of coding block 22 is Intra mode, the server can determine whether there is a coding block whose prediction mode is inter-frame prediction mode on the left side, upper left or above of coding block 22. When the prediction mode of coding block 21 located on the left side (or above or upper left) of coding block 22 is inter-frame prediction mode, the server can obtain the motion vector information of coding block 21 from the coding information of coding block 21, and can scale the motion vector information of coding block 21 according to the playback interval between the reference video frame of coding block 21 and video frame 2 and the playback interval between video frame 2 and video frame 1, and obtain the scaled motion vector information corresponding to coding block 21, and the scaled motion vector information corresponding to coding block 21 is used as the current motion vector information of coding block 22.

[0141] Example 2: The coding block located on the left side of coding block 22 is coding block 21, and the coding block located above (or to the upper left) of coding block 22 is coding block 23. When the server determines that the prediction mode of coding block 22 is Intra mode, the server can determine whether there is a coding block whose prediction mode is inter-frame prediction mode on the left, upper left, or above coding block 22. When the prediction modes of coding block 21 located on the left side of coding block 22 and coding block 23 located above (or to the upper left) of coding block 22 are both inter-frame prediction modes, the server can obtain the motion vector information of coding block 21 from the coding information of coding block 21, and can obtain the motion vector information of coding block 23 from the coding information of coding block 23, and can use the motion vector information of any one of coding block 21 and coding block 23 to participate in determining the current motion vector information of coding block 22. Afterwards, the server can select the motion vector information of one coding block from the motion vector information of coding block 21 and the motion vector information of coding block 23 to participate in determining the current motion vector information of coding block 22. For example, taking the example that the server side selects the motion vector information of coding block 23 from the motion vector information of coding block 21 and the motion vector information of coding block 23 to participate in determining the current motion vector information of coding block 22, the server side can scale the motion vector information of coding block 23 according to the playback interval between the reference video frame of coding block 23 and video frame 2 and the playback interval between video frame 2 and video frame 1, and obtain the scaled motion vector information corresponding to coding block 23, and the scaled motion vector information corresponding to coding block 23 is used as the current motion vector information of coding block 22.

[0142] If only one of coding blocks 21 and coding block 23 has an inter-frame prediction mode as the prediction mode, the server side can obtain the motion vector information of the coding block from the coding information of the coding block whose prediction mode is the inter-frame prediction mode, and can use the motion vector information of the coding block to participate in determining the current motion vector information of coding block 22.

[0143] Example 3: The coding block located above coding block 22 is coding block 21, and the coding block located at the upper left (or left) of coding block 22 is coding block 23. When the server determines that the prediction mode of coding block 22 is Intra mode, the server can determine whether there is a coding block whose prediction mode is inter-frame prediction mode on the left, upper left or above of coding block 22. When the prediction modes of coding block 21 located above coding block 22 and coding block 23 located at the upper left (or left) of coding block 22 are both inter-frame prediction modes, the server can obtain the motion vector information of coding block 21 from the coding information of coding block 21, and can obtain the motion vector information of coding block 23 from the coding information of coding block 23, and can use the motion vector information of any one of coding block 21 and coding block 23 to participate in determining the current motion vector information of coding block 22. Afterwards, the server can select the motion vector information of one coding block from the motion vector information of coding block 21 and the motion vector information of coding block 23 to participate in determining the current motion vector information of coding block 22. For example, taking the example that the server side selects the motion vector information of coding block 23 from the motion vector information of coding block 21 and the motion vector information of coding block 23 to participate in determining the current motion vector information of coding block 22, the server side can scale the motion vector information of coding block 23 according to the playback interval between the reference video frame of coding block 23 and video frame 23 and the playback interval between video frame 2 and video frame 1, and obtain the scaled motion vector information corresponding to coding block 23, and the scaled motion vector information corresponding to coding block 23 is used as the current motion vector information of coding block 22.

[0144] If only one of coding blocks 21 and coding block 23 has an inter-frame prediction mode as the prediction mode, the server side can obtain the motion vector information of the coding block from the coding information of the coding block whose prediction mode is the inter-frame prediction mode, and can use the motion vector information of the coding block to participate in determining the current motion vector information of coding block 22.

[0145] Example 4: The coding block located at the upper left of coding block 22 is coding block 21, and the coding block located above (or on the left) of coding block 22 is coding block 23. When the server determines that the prediction mode of coding block 22 is Intra mode, the server can determine whether there is a coding block whose prediction mode is inter-frame prediction mode on the left, upper left, or above coding block 22. When the prediction modes of coding block 21 located at the upper left of coding block 22 and coding block 23 located above (or on the left) of coding block 22 are both inter-frame prediction modes, the server can obtain the motion vector information of coding block 21 from the coding information of coding block 21, and can obtain the motion vector information of coding block 23 from the coding information of coding block 23, and can use the motion vector information of any one of coding block 21 and coding block 23 to participate in determining the current motion vector information of coding block 22. Afterwards, the server can select the motion vector information of one coding block from the motion vector information of coding block 21 and the motion vector information of coding block 23 to participate in determining the current motion vector information of coding block 22. For example, taking the example that the server side selects the motion vector information of coding block 21 from the motion vector information of coding block 21 and the motion vector information of coding block 23 to participate in determining the current motion vector information of coding block 22, the server side can scale the motion vector information of coding block 21 according to the playback interval between the reference video frame of coding block 21 and video frame 23 and the playback interval between video frame 2 and video frame 1, and obtain the scaled motion vector information corresponding to coding block 21, and the scaled motion vector information corresponding to coding block 21 is used as the current motion vector information of coding block 22.

[0146] If only one of coding blocks 21 and coding block 23 has an inter-frame prediction mode as the prediction mode, the server side can obtain the motion vector information of the coding block from the coding information of the coding block whose prediction mode is the inter-frame prediction mode, and can use the motion vector information of the coding block to participate in determining the current motion vector information of coding block 22.

[0147] Step 306: The server uses (0,0) as the current motion vector information of the first coding block.

[0148] The following describes the implementation process of the server side executing the above step 306 through the following possible examples.

[0149] Example 1: Taking the coding block located at the lower left (or below) of coding block 22 as coding block 21, and the coding block located at the right side (or upper right or lower right) of coding block 22 as coding block 23, after the server determines that the prediction mode of coding block 22 is Intra mode, when the server determines that there are no coding blocks located at the left side, above, and upper left of coding block 22, or there are no coding blocks whose prediction mode is inter-frame prediction mode in the three positions of the left side, above, and upper left of coding block 22, the server can use (0,0) as the current motion vector information of coding block 22.

[0150] Example 2: Taking the coding block located on the right side (or upper right or lower right) of coding block 22 as coding block 21, and the coding block located on the lower left (or lower) of coding block 22 as coding block 23, after the server determines that the prediction mode of coding block 22 is Intra mode, when the server determines that there are no coding blocks located on the left side, above, and upper left of coding block 22, or there are no coding blocks whose prediction mode is inter-frame prediction mode in the three positions of the left side, above, and upper left of coding block 22, the server can use (0,0) as the current motion vector information of coding block 22.

[0151] It can be seen from the above steps 301 to 306 that for each first target video frame, after the server side identifies the prediction mode of each coding block in the multiple coding blocks corresponding to the first target video frame, it can select, for each coding block, a motion vector information calculation method suitable for the prediction mode of the coding block to calculate the current motion vector information of the coding block. This can make the calculation of the current motion vector information of the coding block more accurate, and can make the calculation of the current motion vector information for different coding blocks more in line with the actual encoding conditions of different coding blocks, thereby assisting in the subsequent accurate generation of the frames to be inserted, and helping to ensure the quality of the frames to be inserted.

[0152] Optionally, after performing a time domain downsampling operation on the first video, the server can also dynamically adjust the frame loss strategy by evaluating the frame insertion quality of the first video, so that the frame loss strategy has relatively high flexibility and can flexibly adapt to different video transcoding scenarios. Different frame loss strategies can be adopted for different video transcoding scenarios, thereby ensuring the quality of video frame insertion.

[0153] When the server decodes each video frame that meets the first number threshold (such as 50 or an integer multiple of 50 or other numbers, etc.), the server can generate a to-be-inserted frame corresponding to the first target video frame according to the motion vector information of a first target video frame (such as the fourth target video frame) included in the video frames of the first number threshold and two video frames that are adjacent to and located before and after the first target video frame and are normally coded frames. For example, the server can input the motion vector information of the first target video frame and the two video frames that are adjacent to and located before and after the first target video frame and are normally coded frames into a neural network model (such as a convolutional neural network model) to obtain the to-be-inserted frame corresponding to the first target video frame. The neural network model is a model trained by a large number of video frame training sample sets, and is used to generate the to-be-inserted frame corresponding to the first target video frame. Afterwards, the server can determine the peak signal-to-noise ratio corresponding to the to-be-inserted frame according to the to-be-inserted frame and the first target video frame. Optionally, the peak signal-to-noise ratio can be used as a quality indicator value of the to-be-inserted frame to evaluate the quality of the to-be-inserted frame. Then, the server side can determine how to adjust the frame loss strategy, that is, how to adjust the first threshold and / or the second threshold, based on the size relationship between the peak signal-to-noise ratio and the quality threshold. When the peak signal-to-noise ratio is less than the quality threshold, the server side can reduce the second threshold (such as Th_intra). When the peak signal-to-noise ratio is greater than or equal to the quality threshold, if the number of first target video frames included in the video frames of the first quantity threshold is less than or equal to the second quantity threshold, the server side can reduce the first threshold (such as Th_skip) and / or increase the second threshold (such as Th_intra). If the number of first target video frames included in the video frames of the first quantity threshold is greater than the second quantity threshold, the server side does not need to adjust the first threshold and / or the second threshold. Among them, the second quantity threshold is less than the first quantity threshold; the fourth target video frame is any first target video frame among the multiple first target video frames included in the video frames of the first quantity threshold.

[0154] When the server does not decode the video frames that meet the first number threshold, the server does not need to adjust the frame loss strategy, that is, does not need to adjust the first threshold and / or the second threshold.

[0155] Exemplarily, taking the first quantity threshold as 50 and the second quantity threshold as 20 as an example, when the server decodes 50 video frames each time, the server can select a first target video frame from at least one first target video frame (i.e., a video frame with a discarding characteristic) included in the 50 decoded video frames to participate in evaluating the interpolation quality of the video. For example, taking the 50 video frames decoded by the server as including the above-mentioned video frame 1, video frame 2, and video frame 3, video frame 2 is the first target video frame, and video frame 2 and video frame 3 are second target video frames, the server can select video frame 2 to participate in evaluating the interpolation quality of the video. Among them, video frame 1 is a second target video frame adjacent to video frame 2, and video frame 3 is a second target video frame adjacent to video frame 2. Afterwards, the server can input the motion vector information of multiple coding blocks corresponding to video frame 2, video frame 1, and video frame 3 into the convolutional neural network model to obtain the interpolation frame corresponding to video frame 2, such as interpolation frame a. Then, the server can calculate the peak signal-to-noise ratio corresponding to interpolation frame a based on video frame 2 and interpolation frame a. When the peak signal-to-noise ratio corresponding to the frame to be inserted a is less than the quality threshold, the server side adjusts down Th_intra. When the peak signal-to-noise ratio corresponding to the frame to be inserted a is greater than or equal to the quality threshold, if the server side determines that the number of first target video frames included in the 50 decoded video frames is less than or equal to 20, the server side can adjust down Th_skip and / or increase Th_intra. If the server side determines that the number of first target video frames included in the 50 decoded video frames is greater than 20, the server side does not need to adjust Th_skip and / or Th_intra.

[0156] Step 205: The client generates a second video according to the multiple second target video frames, or generates a second video according to the attribute information of the multiple second target video frames and at least one first target video frame.

[0157] In a possible implementation, after receiving the second code stream from the server, the client can decode the second code stream to obtain multiple second target video frames. Afterwards, the client can generate corresponding frames to be inserted between the multiple second target video frames in a traditional frame insertion manner. Then, the client can generate a second video based on the multiple second target video frames and the corresponding frames to be inserted.

[0158] In another possible implementation, after receiving the second code stream and the third code stream from the server, the client can decode the second code stream to obtain multiple second target video frames, and decode the third code stream to obtain the attribute information of at least one first target video frame. Afterwards, the server can generate the second video according to the attribute information of the multiple second target video frames and the at least one first target video frame. In this way, the implementation can use the attribute information of the video frame with the discarding characteristic and the video frame with the retaining characteristic to participate in the generation of the corresponding frame to be inserted, so as to further ensure the quality of the video insertion frame, thereby solving the problem of more artifacts in the generated video due to inaccurate estimation of the motion vector information of the frame to be inserted in the existing solution, and helping to further alleviate the phenomenon that after some video frames with more intense motion are discarded, the quality of the generated frames to be inserted is poor due to inaccurate estimation of the motion vector information of these video frames with more intense motion.

[0159] For example, for each first target video frame in at least one first target video frame, the server side may first determine the nearest second target video frame in the first video that is located before the playback position of the first target video frame according to the position information of the first target video frame in the first video, and determine the nearest second target video frame in the first video that is located after the playback position of the first target video frame. Afterwards, the server side may generate a to-be-inserted frame corresponding to the first target video frame according to the motion vector information of the multiple coding blocks corresponding to the first target video frame and the two second target video frames adjacent to each other before and after the first target video frame. Among them, the position information of the to-be-inserted frame corresponding to any first target video frame is the same as the position information of the first target video frame in the first video. In other words, the position (such as the playback position) of the first target video frame in the first video is the same as the position where the to-be-inserted frame corresponding to the first target video frame needs to be inserted (or placed or put in). Then, the server side may insert (or put in) the to-be-inserted frame corresponding to each first target video frame into multiple second target video frames according to the position information of each first target video frame in the first video, and generate a second video, and the second video has a complete video sequence.

[0160] Exemplarily, continue to take the above-mentioned video 1, and the server sends the second code stream and the third code stream to the client as an example. Assume that the video frame 1 included in the video 1 is the second target video frame, the video frame 2 is the first target video frame, the video frame 3 is the second target video frame, and the video frame 4 is the second target video frame. Assume that the playback sequence of the video frames in the video 1 is video frame 1, video frame 2, video frame 3, and video frame 4. The second code stream sent by the client to the server carries the encoded video frame 1, the encoded video frame 3, and the encoded video frame 4, and the third code stream sent by the client to the server carries the attribute information of the encoded video frame 2. Exemplarily, the attribute information of the video frame 2 may include the current motion vector information of the coding block 21, the coding block 22, and the coding block 23 corresponding to the video frame 2 and the position information of the video frame 2 in the video 1 (such as the playback position). After receiving the second and third bitstreams from the client, the server decodes the second bitstream to obtain video frame 1, video frame 3, and video frame 4, and decodes the third bitstream to obtain the attribute information of video frame 2. Afterwards, the server can first determine that the nearest second target video frame in video 1 before the playback position of video frame 2 is video frame 1, and determine that the nearest second target video frame in video 1 after the playback position of video frame 2 is video frame 3, according to the position information of video frame 2 in video 1, and then input the current motion vector information of coding block 21, coding block 22, and coding block 23, as well as video frame 1 and video frame 3 into the convolutional neural network model (i.e., the trained convolutional neural network model for generating the to-be-inserted frame), and obtain the to-be-inserted frame corresponding to video frame 2. Among them, the playback position of video frame 2 in video 1 is the same as the playback position where the to-be-inserted frame corresponding to video frame 2 needs to be inserted. Then, according to the position information of video frame 2 in video 1, the server can insert the to-be-inserted frame corresponding to video frame 2 into video frame 1, video frame 3, and video frame 4 to generate a complete video.

[0161] It can be seen from the above steps 201 to 205 that the server analyzes the properties of each video frame (i.e., whether each video frame has a discarding feature or a retaining feature) according to the encoding information of each video frame in the multiple video frames, and can more accurately decide which video frames are suitable for being discarded according to the properties of the multiple video frames. Afterwards, the server can encode the video frames with the retaining feature into a code stream and transmit it to the client. Then, after decoding the received code stream, the client obtains the video frames with the retaining feature. Since the server discards the video frames that are easier to interpolate, the client can generate the corresponding frames to be interpolated more easily according to the video frames with the retaining feature. This can ensure the quality of the video interpolation frames and help alleviate the phenomenon that the quality of the generated frames to be interpolated is poor after some video frames with more intense motion are discarded due to inaccurate estimation of the motion vector information of these video frames with more intense motion. In addition, since the server performs more accurate time domain downsampling operations on multiple video frames, some video frames can be discarded to the maximum extent, and the data transmitted from the server to the client is encoded, so the amount of data transmitted from the server to the client can be further reduced, thereby further reducing the transmission bandwidth between the server and the client, which helps to alleviate the phenomenon of poor video viewing experience for users in weak network environments. Optionally, when the server also encodes the attribute information of the video frame with the discarding characteristic into a code stream and transmits it to the client, the client can effectively and comprehensively utilize the attribute information of the video frame with the discarding characteristic and the video frame with the retention characteristic to generate the corresponding frame to be inserted, so that the quality of the video insertion frame can be further guaranteed, thereby solving the problem of more artifacts in the generated video due to inaccurate estimation of the motion vector information of the frame to be inserted in the existing solution, and can further effectively alleviate the phenomenon that after some video frames with more intense motion are discarded, the quality of the generated frame to be inserted is poor due to inaccurate estimation of the motion vector information of these video frames with more intense motion.

[0162] It should be noted that in the description of the present application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there may be three relationships, for example, A and / or B, which can represent: the situation where A exists alone, A and B exist at the same time, and B exists alone, wherein A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are an "or" relationship. "At least one of the following (individuals)" or its similar expression refers to any combination of these items, including any combination of single items (individuals) or plural items (individuals). For example, "at least one of A, B and C" includes A, B, C, AB, AC, BC or ABC. And, unless otherwise specified, the ordinal numbers such as "first", "second", and "third" mentioned in the embodiments of the present application are used to distinguish multiple objects, and are not used to limit the order, timing, priority or importance of multiple objects. In addition, the terms "including", "comprising", "having" and their variations appearing in the present application all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0163] In addition, it should be noted that each step involved in the above embodiments can be performed by a corresponding device, or by a chip, processor, or chip system in the device, and the embodiments of the present application do not limit them. The above embodiments are only described by taking the corresponding device as an example.

[0164] It should be noted that in the above embodiments, some steps can be selected for implementation, and the order of the steps in the diagram can be adjusted for implementation, and this application does not limit this. It should be understood that executing some steps in the diagram, adjusting the order of the steps, or combining them for specific implementation all fall within the scope of protection of this application.

[0165] It is understandable that, in order to implement the functions in the above embodiments, the various devices involved in the above embodiments include hardware structures and / or software modules corresponding to the execution of the various functions. It should be easily appreciated by those skilled in the art that, in combination with the units and method steps of the various examples described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0166] It should be noted that the "steps" in the embodiments of the present application are only for illustration, and are a method of expression used to better understand the embodiments, and do not constitute a substantial limitation on the execution of the scheme of the present application. For example, the "steps" can also be understood as "features". In addition, the steps do not constitute any limitation on the execution order of the scheme of the present application. Any changes in the order of steps, step merging, or step splitting made on this basis that do not affect the implementation of the overall scheme, and the resulting new technical solutions are also within the scope of the disclosure of the present application.

[0167] Based on the same concept, the embodiment of the present application also provides a possible video processing device, which is suitable for Figure 1 Optionally, the video processing device may be a computing device (such as a first video processing device or a second video processing device) or a component (such as a chip, a chip system or a circuit, etc.) that can support the computing device to implement the functions required by the video processing method. In one example, when the video processing device is a first video processing device (such as Figure 1 When the video processing device is used to implement the technical solution involved in the first video processing device in the above embodiment, or the module (such as a chip) of the video processing device is used to implement the technical solution involved in the first video processing device in the above embodiment, so the beneficial effects of the first video processing device in the above embodiment can also be achieved. Exemplarily, taking the video processing device as a chip set in a computing device as an example, when the video processing device is a chip, the video processing device includes a transceiver and a processor, but does not include a memory. Among them, the transceiver exists as an input and output interface, and the input and output interface is used for the chip to implement the transceiver of the computing device. The input and output interface may include an input interface and / or an output interface, the input interface can implement the reception of the computing device, and the output interface can be used to implement the sending of the computing device. The processor is used to read and execute the corresponding computer program or instruction, so that the corresponding function of the video processing device is implemented. Optionally, when the chip implements the corresponding function of the first video processing device in the above embodiment, the input and output interface can implement the transceiver operation performed by the first video processing device in the above embodiment; the processor can implement other operations except the transceiver operation performed by the first video processing device in the above embodiment. For specific related descriptions, please refer to the related descriptions in the above embodiments, which will not be described in detail here.

[0168] See also Figure 4a The video processing device 400a includes a decoding module 401, a video characteristic analysis module 402 and a transceiver module 403 (for sending and receiving data). Optionally, the video processing device 400a may also include a time domain downsampling module 404, an encoding module 405 and an interpolation frame quality feedback module 406.

[0169] Among them, the time domain downsampling module 404 is used to discard at least one first target video frame and retain multiple second target video frames. The interpolation frame quality feedback module 406 is used to provide feedback to the video characteristic analysis module 402. For example, the interpolation frame quality feedback module 406 can generate a frame to be interpolated, and determine whether the first threshold and the second threshold need to be adjusted according to the quality of the frame to be interpolated. The encoding module 405 is used to encode multiple second target video frames to obtain a second bitstream, or to encode the attribute information of at least one first target video frame to obtain a third bitstream. Among them, the third bitstream carries the attribute information of at least one first target video frame.

[0170] When the video processing device 400a is used to implement Figure 2 or Figure 3 The functions of the server side (such as a cloud server) in the illustrated method embodiment are as follows: a transceiver module 403 is used to obtain a first code stream. A decoding module 401 is used to decode the first code stream to obtain multiple video frames included in the first video and encoding information of multiple video frames. Among them, the encoding information may include one or more of block division information, motion vector information, prediction mode and block residual information of multiple video frames. A video characteristic analysis module 402 is used to determine at least one first target video frame included in multiple video frames based on the encoding information of multiple video frames. Among them, at least one first target video frame is a video frame that needs to be discarded among multiple video frames. The transceiver module 403 is also used to send a second code stream. Among them, the second code stream carries multiple second target video frames, and the multiple second target video frames are other video frames in multiple video frames except at least one first target video frame. The transceiver module 403 is also used to send a third code stream.

[0171] Wherein, when the video processing device 400a is used to implement Figure 2 or Figure 3 For more detailed description of the decoding module 401, the video characteristic analysis module 402, the transceiver module 403, the time domain downsampling module 404, the encoding module 405 and the interpolation frame quality feedback module 406, please refer to the above Figure 2 or Figure 3 The relevant description about the server side in the illustrated method embodiment will not be repeated here.

[0172] In another example, when the video processing device is a second video processing device (eg Figure 1When the client 100 is shown in the figure, the video processing device is used to implement the technical solution involved in the second video processing device in the above embodiment, or the module (such as a chip) of the video processing device is used to implement the technical solution involved in the second video processing device in the above embodiment, so the beneficial effects of the second video processing device in the above embodiment can also be achieved. Exemplarily, taking the video processing device as a chip set in a computing device as an example, when the video processing device is a chip, the video processing device includes a transceiver and a processor, but does not include a memory. Among them, the transceiver exists as an input and output interface, and the input and output interface is used for the chip to implement the transceiver of the computing device. The input and output interface may include an input interface and / or an output interface, the input interface can implement the reception of the computing device, and the output interface can be used to implement the sending of the computing device. The processor is used to read and execute the corresponding computer program or instruction, so that the corresponding function of the video processing device is implemented. Optionally, when the chip implements the corresponding function of the second video processing device in the above embodiment, the input and output interface can implement the transceiver operation performed by the second video processing device in the above embodiment; the processor can implement other operations except the transceiver operation performed by the second video processing device in the above embodiment. For specific related descriptions, please refer to the related descriptions in the above embodiments, which will not be described in detail here.

[0173] See also Figure 4b , the video processing device 400b includes a transceiver module 401 and a video interpolation module 402. Optionally, the video processing device 400b also includes a collection module 403, an encoding module 404 and a decoding module 405. Among them, the collection module 403 is used to collect relevant videos of the user, such as videos recorded by the user or videos generated by the user's live broadcast; the encoding module 404 is used to encode the relevant videos (such as the first video) collected by the collection module 403 to generate corresponding code streams (such as the first code stream corresponding to the first video). The transceiver module 405 is used to send the corresponding code stream (such as the first code stream) generated by the encoding module 404, or to receive the corresponding code stream (such as the second code stream and the third code stream) from the server.

[0174] When the video processing device 400b is used to implement Figure 2 The functions of the client in the illustrated method embodiment are: a transceiver module 401, used to receive a second bit stream from a server. The second bit stream carries a plurality of second target video frames. A decoding module 405, used to decode the second bit stream to obtain a plurality of second target video frames. A video frame insertion module 402, used to generate a second video according to the plurality of second target video frames.

[0175] Wherein, when the video processing device 400b is used to implement Figure 2For more detailed description of the transceiver module 401, the video frame insertion module 402, the acquisition module 403, the encoding module 404 and the decoding module 405, please refer to the above Figure 2 The relevant description about the client in the illustrated method embodiment will not be repeated here.

[0176] Optional, above Figure 4a The decoding module, video characteristic analysis module, transceiver module, time domain downsampling module, interpolation frame quality feedback module and encoding module shown in the figure can be implemented by software, or can be implemented by hardware; Figure 4b The decoding module, video frame insertion module, acquisition module, encoding module and transceiver module shown in the figure can be implemented by software, or can be implemented by hardware. Figure 4a Taking the video characteristic analysis module shown in FIG. 1 as an example, the implementation method of the video characteristic analysis module is introduced. Figure 4a The implementation of the decoding module, the transceiver module, the time domain downsampling module, the interpolation frame quality feedback module and the encoding module shown in the figure can refer to the implementation of the video characteristic analysis module. Figure 4b The decoding module, video frame insertion module, acquisition module, encoding module and transceiver module shown in the figure can also refer to the implementation method of the video characteristic analysis module, which will not be repeated here.

[0177] When implemented by software, the video feature analysis module is used as an example of a software functional unit, and the video feature analysis module can be an application or a code block running on a computer device. Among them, the computer device can be at least one of a physical host, a virtual machine, a container and other computing devices. Furthermore, the above-mentioned computer device can be one or more. For example, the video feature analysis module can be an application running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application can be distributed in the same availability zone (AZ) or in different AZs. The multiple hosts / virtual machines / containers used to run the application can be distributed in the same region (region) or in different regions. Among them, usually a region can include multiple AZs.

[0178] Similarly, multiple hosts / virtual machines / containers for running the application can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Typically, a region can include multiple VPCs, and a VPC can include multiple AZs.

[0179] When implemented by hardware, the video characteristic analysis module is used as an example of a hardware functional unit, and the video characteristic analysis module may include at least one computing device, such as a server, etc. Alternatively, the video characteristic analysis module may also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0180] The multiple computing devices included in the video characteristic analysis module can be distributed in the same AZ or in different AZs. The multiple computing devices included in the video characteristic analysis module can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the video characteristic analysis module can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0181] It should be noted that the above Figure 4a The steps implemented by the decoding module, video characteristic analysis module, transceiver module, time domain downsampling module, interpolation frame quality feedback module and encoding module can be specified as needed, and the decoding module, video characteristic analysis module, transceiver module, time domain downsampling module, interpolation frame quality feedback module and encoding module respectively implement different execution steps of the video processing method to realize all functions of the first video processing device. Figure 4b The steps implemented by the decoding module, video interpolation module, acquisition module, encoding module and transceiver module shown in the figure can be specified as needed. The full functions of the second video processing device can be realized by respectively implementing different execution steps of the video processing method through the decoding module, video interpolation module, acquisition module, encoding module and transceiver module.

[0182] In addition, it should be noted that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. The functional modules in the embodiments of the present application may be integrated into one module, or each module may exist physically separately, or two or more modules may be integrated into one module. For example, Figure 4a Taking the multiple modules shown as an example, the decoding module, video characteristic analysis module, transceiver module, time domain downsampling module, interpolation frame quality feedback module and encoding module can be integrated into one module, or the decoding module, video characteristic analysis module, transceiver module, time domain downsampling module, interpolation frame quality feedback module and encoding module are the same module. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0183] Based on the same concept, the present application embodiment also provides a possible computing device for executing the video processing method shown in the above method embodiment. The relevant features can be found in the above method embodiment and will not be described in detail here. Figure 5a As shown, the computing device 500 includes: a bus 501, a processor 502 and a memory 504. Optionally, the computing device 500 may also include a communication interface 503. The processor 502, the communication interface 503 and the memory 504 communicate with each other through the bus 501. The computing device 500 may be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 500.

[0184] The bus 501 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5a The bus 501 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 501 may include a path for transmitting information between various components of the computing device 500 (for example, the memory 504, the processor 502, and the communication interface 503).

[0185] The processor 502 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0186] The memory 504 may include a volatile memory, such as a random access memory (RAM). The processor 502 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0187] The memory 504 stores executable program codes, and the processor 502 executes the executable program codes to implement the functions of the aforementioned first video processing device (such as a decoding module, a video characteristic analysis module, and a transceiver module, or one or more of a time domain downsampling module, an interpolation frame quality feedback module, or an encoding module), thereby implementing the video processing method provided in the embodiment of the present application. That is, the memory 504 stores computer program instructions for executing the video processing method.

[0188] Alternatively, the memory 504 stores executable code, and the processor 502 executes the executable code to implement the functions of the aforementioned first video processing device (such as a decoding module, a video characteristic analysis module, and a transceiver module, or one or more of a time domain downsampling module, an interpolation frame quality feedback module, or an encoding module), thereby implementing the video processing method provided in the embodiment of the present application. That is, the memory 504 stores computer program instructions for the first video processing device to execute the video processing method provided in the embodiment of the present application.

[0189] The communication interface 503 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 500 and other devices or a communication network.

[0190] like Figure 5b As shown, the computing device 500 includes: a bus 501, a processor 502 and a memory 504. Optionally, the computing device 500 may also include a communication interface 503. The processor 502, the communication interface 503 and the memory 504 communicate with each other through the bus 501. The computing device 500 may be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 500.

[0191] The bus 501 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5b The bus 501 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 501 may include a path for transmitting information between various components of the computing device 500 (for example, the memory 504, the processor 502, and the communication interface 503).

[0192] The processor 502 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0193] The memory 504 may include a volatile memory, such as a random access memory (RAM). The processor 502 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0194] The memory 504 stores executable program codes, and the processor 502 executes the executable program codes to implement the functions of the aforementioned second video processing device (such as the transceiver module and the video interpolation module and one or more of the acquisition module, encoding module or decoding module), thereby implementing the video processing method provided in the embodiment of the present application. That is, the memory 504 stores computer program instructions for executing the video processing method.

[0195] Alternatively, the memory 504 stores executable code, and the processor 502 executes the executable code to implement the functions of the aforementioned second video processing device (such as the transceiver module and the video interpolation module and one or more of the acquisition module, encoding module or decoding module), thereby implementing the video processing method provided in the embodiment of the present application. That is, the memory 504 stores computer program instructions for the second video processing device to execute the video processing method provided in the embodiment of the present application.

[0196] The communication interface 503 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 500 and other devices or a communication network.

[0197] Based on the same concept, the embodiment of the present application also provides a possible computing device cluster for executing the method shown in the above method embodiment. The relevant features can be found in the above method embodiment, which will not be repeated here. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop, a laptop, or a smart phone.

[0198] For example, Figure 5a The instructions stored in the memory 504 of the computing device 500 shown in the figure can implement the functions of the first video processing device as an example. Figure 6a As shown, the computing device cluster 600 includes at least one computing device 500. The memory 504 in one or more computing devices 500 in the computing device cluster 600 may store the same instructions for executing the video processing method provided in the embodiment of the present application.

[0199] In some possible implementations, the memory 504 of one or more computing devices 500 in the computing device cluster 600 may also store partial instructions for executing the video processing method. In other words, the combination of one or more computing devices 500 may jointly execute instructions for executing the video processing method.

[0200] It should be noted that the memory 504 in different computing devices 500 in the computing device cluster 600 may store different instructions, which are respectively used to execute part of the functions of the first video processing apparatus. That is, the instructions stored in the memory 504 in different computing devices 500 may implement the functions of one or more modules among the decoding module, the video characteristic analysis module, the transceiver module, the time domain downsampling module, the interpolation frame quality feedback module or the encoding module.

[0201] In some possible implementations, one or more computing devices in computing device cluster 600 may be connected via a network. The network may be a wide area network or a local area network. Figure 5a For example, the instructions stored in the memory 504 of the computing device 500 can implement the functions of the aforementioned first video processing device. Figure 7a A possible implementation is shown. Figure 7a As shown, two computing devices 500A and 500B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 504 in the computing device 500A stores instructions for executing the functions of a decoding module, a video characteristic analysis module, and a transceiver module. At the same time, the memory 504 in the computing device 500B stores instructions for executing the functions of a time domain downsampling module, an interpolation quality feedback module, and an encoding module. In some possible implementations, the memory 504 of one or more computing devices 500 in the computing device cluster may also respectively store partial instructions for executing the video processing method. In other words, a combination of one or more computing devices 500 can jointly execute instructions for executing the video processing method.

[0202] It should be understood that Figure 7a The functionality of the computing device 500A shown in FIG. 5 may also be provided by multiple Figure 5a Similarly, the functions of computing device 500B can also be performed by multiple Figure 5a The illustrated computing device 500 is completed.

[0203] For example, Figure 5b The instructions stored in the memory 504 of the computing device 500 shown in the figure can implement the functions of the aforementioned second video processing device as an example. Figure 6b As shown, the computing device cluster 600 includes at least one computing device 500. The memory 504 in one or more computing devices 500 in the computing device cluster 600 may store the same instructions for executing the video processing method provided in the embodiment of the present application.

[0204] In some possible implementations, the memory 504 of one or more computing devices 500 in the computing device cluster 600 may also store partial instructions for executing the video processing method. In other words, the combination of one or more computing devices 500 may jointly execute instructions for executing the video processing method.

[0205] It should be noted that the memory 504 in different computing devices 500 in the computing device cluster 600 may store different instructions, which are respectively used to execute part of the functions of the second video processing apparatus. That is, the instructions stored in the memory 504 in different computing devices 500 may implement the functions of one or more modules among the decoding module, the video interpolation module, the acquisition module, the encoding module or the transceiver module.

[0206] In some possible implementations, one or more computing devices in computing device cluster 600 may be connected via a network. The network may be a wide area network or a local area network. Figure 5b For example, the instructions stored in the memory 504 of the computing device 500 can implement the functions of the aforementioned second video processing device. Figure 7b A possible implementation is shown. Figure 7b As shown, two computing devices 500A' and 500B' are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 504 in the computing device 500A' stores instructions for executing the functions of the transceiver module and the video interpolation module. At the same time, the memory 504 in the computing device 500B' stores instructions for executing the functions of the acquisition module, the encoding module, and the decoding module. In some possible implementations, the memory 504 of one or more computing devices 500 in the computing device cluster may also respectively store some instructions for executing the video processing method. In other words, a combination of one or more computing devices 500 can jointly execute instructions for executing the video processing method.

[0207] It should be understood that Figure 7b The functionality of the computing device 500A' shown in FIG. 5 may also be provided by multiple Figure 5b Similarly, the functions of computing device 500B' can also be performed by multiple Figure 5b The illustrated computing device 500 is completed.

[0208] It is understandable that the computing device storing the functions of the multiple modules included in the first video processing device can also be combined with the computing device storing the functions of the multiple modules included in the second video processing device to form a computing device such as Figure 7a or Figure 7b A cluster of computing devices is shown.

[0209] Based on the same concept, the embodiment of the present application also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the video processing method provided in the embodiment of the present application.

[0210] Based on the same concept, an embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the video processing method provided in the embodiment of the present application.

[0211] Based on the same concept, an embodiment of the present application further provides a computer chip, which is coupled to a memory and is used to read a computer program stored in the memory to execute a video processing method provided by an embodiment of the present application.

[0212] Based on the same concept, an embodiment of the present application also provides a chip system, which includes a processor for supporting a computer device to implement the video processing method provided in an embodiment of the present application. In a possible design, the chip system also includes a memory, which is used to store the necessary programs and data of the computer device. The chip system can be composed of a computer chip, or it can include a computer chip and other discrete devices.

[0213] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A video processing method, characterized in that: include: Decoding the first bitstream to obtain a plurality of video frames included in the first video and encoding information of the plurality of video frames, wherein the encoding information includes one or more of block division information, motion vector information, prediction mode, and block residual information of the plurality of video frames; Determine, according to the encoding information of the multiple video frames, at least one first target video frame included in the multiple video frames, where the at least one first target video frame is a video frame that needs to be discarded among the multiple video frames; A second code stream is sent, where the second code stream carries a plurality of second target video frames, where the plurality of second target video frames are other video frames in the plurality of video frames except the at least one first target video frame.

2. The method according to claim 1, characterized in that Determining, according to the encoding information of the multiple video frames, at least one first target video frame included in the multiple video frames, comprising: Determine, according to encoding information of a first video frame, a texture complexity index value of the first video frame, where the first video frame is any one of the multiple video frames; When the texture complexity index value of the first video frame meets the texture complexity threshold, the first video frame is determined to be the first target video frame.

3. The method according to claim 2, characterized in that Each of the multiple video frames includes multiple coding blocks, the coding information of each video frame includes prediction modes of the multiple coding blocks of the video frame, the texture complexity index value includes a first ratio and / or a second ratio, and the texture complexity threshold includes a first threshold and / or a second threshold; When the texture complexity index value of the first video frame meets the texture complexity threshold, determining that the first video frame is the first target video frame includes: When a first ratio corresponding to the first video frame is greater than a first threshold and / or a second ratio corresponding to the first video frame is less than a second threshold, determining that the first video frame is the first target video frame; Among them, the first ratio is used to indicate the proportion of the number of pixels in the area where the coding block whose prediction mode is the inter-frame prediction mode in the first video frame is located, and the second ratio is used to indicate the proportion of the number of pixels in the area where the coding block whose prediction mode is the intra-frame prediction mode in the first video frame is located.

4. The method according to claim 3, characterized in that The method further comprises: When the first ratio is less than or equal to the first threshold, or the second ratio is greater than or equal to the second threshold, the first video frame is determined to be the second target video frame.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: A third code stream is sent, where the third code stream carries attribute information of the at least one first target video frame, where the attribute information is used to indicate that a to-be-inserted frame corresponding to the first target video frame is generated from among the multiple second target video frames according to the attribute information.

6. The method according to claim 5, characterized in that The method further comprises: When the prediction mode of the first coding block corresponding to the first target video frame is the inter-frame prediction mode, scaling the motion vector information of the first coding block to obtain scaled motion vector information of the first coding block; or When the prediction mode of the first coding block is the intra-frame prediction mode, if there is a second coding block whose prediction mode is the inter-frame prediction mode on the left side, above or above the left of the first coding block, the motion vector information of the second coding block is scaled to obtain the motion vector information of the first coding block; if there is no second coding block whose prediction mode is the inter-frame prediction mode in the first coding block, (0,0) is used as the motion vector information of the first coding block; Attribute information of the first target video frame is determined according to motion vector information of a plurality of coding blocks included in the first target video frame and position information of the first target video frame in the first video.

7. The method according to claim 5 or 6, characterized in that The method further comprises: Whenever decoding video frames of a first number threshold, generating a to-be-interpolated frame corresponding to the first target video frame according to motion vector information of one of the first target video frames included in the video frames of the first number threshold and two second target video frames located before and after the first target video frame; Determine that the quality indicator value of the frame to be inserted is less than or equal to the quality threshold, and reduce the second threshold; or Determine that the number of the first target video frames included in the video frames of the first number threshold is less than or equal to a second number threshold, reduce the first threshold and / or increase the second threshold.

8. The method according to claim 7, characterized in that The method further comprises: A peak signal-to-noise ratio corresponding to the frame to be inserted is determined according to the first target video frame and the frame to be inserted, and the peak signal-to-noise ratio is used as a quality indicator value of the frame to be inserted.

9. A video processing method, characterized in that: The method is applied to a client, the client is used to process a code stream from a server, the server is used to process a first code stream, the first code stream carries multiple video frames included in a first video and encoding information of the multiple video frames, wherein at least one first target video frame among the multiple video frames is a video frame that needs to be discarded, and multiple second target video frames among the multiple video frames are other video frames among the multiple video frames except the at least one first target video frame, the encoding information includes one or more of block division information, motion vector information, prediction mode and block residual information of the multiple video frames, and the method includes: receiving a second code stream from the server, wherein the second code stream carries the plurality of second target video frames; A second video is generated according to the plurality of second target video frames.

10. The method according to claim 9, characterized in that Generating a second video according to the plurality of second target video frames comprises: receiving a third bitstream from the server, the third bitstream carrying attribute information of the at least one first target video frame, the attribute information being used to indicate generating a to-be-inserted frame corresponding to the first target video frame from among the plurality of second target video frames according to the attribute information; The second video is generated according to the attribute information of the plurality of second target video frames and the at least one first target video frame.

11. The method according to claim 10, characterized in that The attribute information of each first target video frame in the at least one first target video frame includes position information of the first target video frame in the first video and motion vector information of a plurality of coding blocks included in the first target video frame; Generating the second video according to the attribute information of the plurality of second target video frames and the at least one first target video frame includes: Generate a to-be-interpolated frame corresponding to each first target video frame according to motion vector information of multiple coding blocks corresponding to each first target video frame and two second target video frames located before and after each first target video frame, wherein position information of the to-be-interpolated frame corresponding to each first target video frame is the same as position information of the first target video frame in the first video; According to the position information of each first target video frame in the first video, the to-be-inserted frame corresponding to each first target video frame is inserted into the plurality of second target video frames to obtain the second video.

12. A video processing device, characterized in that: include: A decoding module, configured to decode the first bitstream to obtain a plurality of video frames included in the first video and encoding information of the plurality of video frames, wherein the encoding information includes one or more of block division information, motion vector information, prediction mode, and block residual information of the plurality of video frames; A video characteristic analysis module, configured to determine, according to encoding information of the plurality of video frames, at least one first target video frame included in the plurality of video frames, wherein the at least one first target video frame is a video frame that needs to be discarded among the plurality of video frames; The transceiver module is used to send a second code stream, where the second code stream carries multiple second target video frames, and the multiple second target video frames are other video frames in the multiple video frames except the at least one first target video frame.

13. The device according to claim 12, characterized in that When the video characteristic analysis module determines at least one first target video frame included in the plurality of video frames according to the encoding information of the plurality of video frames, it is specifically used to: Determine, according to encoding information of a first video frame, a texture complexity index value of the first video frame, where the first video frame is any one of the multiple video frames; When the texture complexity index value of the first video frame meets the texture complexity threshold, the first video frame is determined to be the first target video frame.

14. The device according to claim 13, characterized in that Each of the multiple video frames includes multiple coding blocks, the coding information of each video frame includes prediction modes of the multiple coding blocks of the video frame, the texture complexity index value includes a first ratio and / or a second ratio, and the texture complexity threshold includes a first threshold and / or a second threshold; The video characteristic analysis module determines that the first video frame is the first target video frame when the texture complexity index value of the first video frame meets the texture complexity threshold, specifically for: When a first ratio corresponding to the first video frame is greater than a first threshold and / or a second ratio corresponding to the first video frame is less than a second threshold, determining that the first video frame is the first target video frame; Among them, the first ratio is used to indicate the proportion of the number of pixels in the area where the coding block whose prediction mode is the inter-frame prediction mode in the first video frame is located, and the second ratio is used to indicate the proportion of the number of pixels in the area where the coding block whose prediction mode is the intra-frame prediction mode in the first video frame is located.

15. The device according to claim 14, characterized in that The video characteristic analysis module is also used for: When the first ratio is less than or equal to the first threshold, or the second ratio is greater than or equal to the second threshold, the first video frame is determined to be the second target video frame.

16. The device according to any one of claims 12 to 15, characterized in that: The transceiver module is also used for: A third code stream is sent, where the third code stream carries attribute information of the at least one first target video frame, where the attribute information is used to indicate that a to-be-inserted frame corresponding to the first target video frame is generated from among the multiple second target video frames according to the attribute information.

17. The device according to claim 16, characterized in that The video characteristic analysis module is also used for: When the prediction mode of the first coding block corresponding to the first target video frame is the inter-frame prediction mode, scaling the motion vector information of the first coding block to obtain scaled motion vector information of the first coding block; or When the prediction mode of the first coding block is the intra-frame prediction mode, if there is a second coding block whose prediction mode is the inter-frame prediction mode on the left side, above or above the left of the first coding block, the motion vector information of the second coding block is scaled to obtain the motion vector information of the first coding block; if there is no second coding block whose prediction mode is the inter-frame prediction mode in the first coding block, (0,0) is used as the motion vector information of the first coding block; Attribute information of the first target video frame is determined according to motion vector information of a plurality of coding blocks included in the first target video frame and position information of the first target video frame in the first video.

18. The device according to claim 16 or 17, characterized in that The device also includes a frame insertion quality feedback module; The interpolation frame quality feedback module is used to generate, when decoding video frames of a first number threshold, a to-be-interpolated frame corresponding to the first target video frame according to motion vector information of a first target video frame included in the video frames of the first number threshold and two second target video frames located before and after the first target video frame; determine that the quality index value of the to-be-interpolated frame is less than or equal to the quality threshold, and reduce the second threshold; or determine that the number of the first target video frames included in the video frames of the first number threshold is less than or equal to the second number threshold, and reduce the first threshold or increase the second threshold.

19. The device according to claim 18, characterized in that The interpolation frame quality feedback module is further used to determine a peak signal-to-noise ratio corresponding to the frame to be interpolated according to the first target video frame and the frame to be interpolated, and the peak signal-to-noise ratio is used as a quality indicator value of the frame to be interpolated.

20. A video processing device, characterized in that: The device is applied to a client, the client is used to process a code stream from a server, the server is used to process a first code stream, the first code stream carries multiple video frames included in a first video and encoding information of the multiple video frames, wherein at least one first target video frame among the multiple video frames is a video frame that needs to be discarded, and multiple second target video frames among the multiple video frames are other video frames among the multiple video frames except the at least one first target video frame, the encoding information includes one or more of block division information, motion vector information, prediction mode and block residual information of the multiple video frames, and the device includes: A transceiver module, configured to receive a second code stream from the server, wherein the second code stream carries the plurality of second target video frames; The video frame insertion module is used to generate a second video according to the multiple second target video frames.

21. The device according to claim 20, characterized in that When the video frame insertion module generates the second video according to the plurality of second target video frames, it is specifically used to: receiving a third bitstream from the server, the third bitstream carrying attribute information of the at least one first target video frame, the attribute information being used to indicate generating a to-be-inserted frame corresponding to the first target video frame from among the plurality of second target video frames according to the attribute information; The second video is generated according to the attribute information of the plurality of second target video frames and the at least one first target video frame.

22. The device according to claim 20, characterized in that The attribute information of each first target video frame in the at least one first target video frame includes position information of the first target video frame in the first video and motion vector information of a plurality of coding blocks included in the first target video frame; When the video frame insertion module generates the second video according to the attribute information of the plurality of second target video frames and the at least one first target video frame, it is specifically used to: Generate a to-be-interpolated frame corresponding to each first target video frame according to motion vector information of multiple coding blocks corresponding to each first target video frame and two second target video frames located before and after each first target video frame, wherein position information of the to-be-interpolated frame corresponding to each first target video frame is the same as position information of the first target video frame in the first video; According to the position information of each first target video frame in the first video, the to-be-inserted frame corresponding to each first target video frame is inserted into the plurality of second target video frames to obtain the second video.

23. A computing device, characterized in that The computing device includes a communication interface, a processor, and a memory; The communication interface is used to receive and send data; The memory is used to store computer program instructions; The processor calls the computer program instructions stored in the memory to execute the method according to any one of claims 1 to 8 or the method according to any one of claims 9 to 11.

24. A computing device cluster, characterized in that: The computing device cluster includes at least one computing device, each computing device includes a processor and a memory; the memory in the at least one computing device is used to store computer program instructions; The processor in the at least one computing device calls the computer program instructions stored in the memory to execute the method according to any one of claims 1 to 8 or the method according to any one of claims 9 to 11.

25. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device, the computing device performs the method according to any one of claims 1 to 8 or the method according to any one of claims 9 to 11.

26. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device, the computing device is caused to perform the method according to any one of claims 1 to 8 or the method according to any one of claims 9 to 11.

Images