Video processing method, device, electronic device and storage medium

Through the combination of multiple motion prediction modes and weight information, the target prediction frame is generated, which solves the problem of difficult to predict complex video time domain motion relationships in the prior art, and improves the performance and accuracy of video encoding and decoding.

CN114339216BActive Publication Date: 2024-11-08ALIBABA DAMO (HANGZHOU) TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011080641.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-10-10
Publication Date
2024-11-08
Estimated Expiration
2040-10-10

AI Technical Summary

Technical Problem

Existing video encoding and decoding technologies based on deep learning are difficult to fully predict complex time-domain motion relationships in videos, and relying solely on a single time-domain prediction mode such as optical flow cannot effectively reflect the motion relationships between video frames.

Method used

Using a variety of motion prediction modes, including optical flow motion prediction information and adaptive convolution kernel, a variety of motion prediction information between the current frame and the reference frame are obtained through the motion prediction model, and the target prediction frame is generated by combining the weight information, and the target prediction frame is generated by weighting. Summarize to get the final predicted frame.

Benefits of technology

It improves the motion prediction performance of video frames, can more accurately reflect the complex time-domain motion relationships in the video, and enhances the efficiency and accuracy of video encoding and decoding.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114339216B_ABST
    Figure CN114339216B_ABST
Patent Text Reader

Abstract

The disclosed embodiment discloses a video processing method, device, electronic device and storage medium, the method comprising: obtaining a reconstructed reference frame of a current frame and at least one reference frame in a video to be processed; obtaining multiple motion prediction information of the current frame and the reconstructed reference frame using a motion prediction model; determining a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information. The technical solution can improve the motion prediction performance of video frames through diversified motion prediction modes during motion prediction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a video processing method, device, electronic device and storage medium. Background Art

[0002] In the field of video coding and decoding, international coding standards such as H.264, H.265, H.266, etc. have achieved significant technical progress and market promotion. However, the technical framework of the above-mentioned international coding standards is no different from that of decades ago. In recent years, image and / or video compression technology based on deep learning has attracted extensive research and achieved certain results. However, the inventors of the present disclosure have found that the current image and / or video compression technology based on deep learning is basically limited to a single time domain prediction mode such as optical flow to predict the time domain motion relationship between video frames, and this prediction mode alone cannot fully reflect the complex time domain motion relationship in a video. Therefore, how to fully predict the complex time domain motion relationship in a video has become one of the technical problems that need to be solved in the field of deep learning video coding and decoding technology. Summary of the invention

[0003] Embodiments of the present disclosure provide a video processing method, device, electronic device, and computer-readable storage medium.

[0004] In a first aspect, an embodiment of the present disclosure provides a video processing method, including:

[0005] Obtaining a reconstructed reference frame of a current frame and at least one reference frame in a video to be processed;

[0006] Acquire multiple motion prediction information of the current frame and the reconstructed reference frame using a motion prediction model;

[0007] A target prediction frame corresponding to the current frame is determined according to the reconstructed reference frame and the multiple motion prediction information.

[0008] Further, determining a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information includes:

[0009] Determine a plurality of candidate prediction frames corresponding to the current frame according to the plurality of motion prediction information;

[0010] The target prediction frame is obtained by using weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames.

[0011] Furthermore, the weight information is obtained by the motion prediction model based on the current frame and the reconstructed reference frame; and / or, the weight information corresponding to each type of motion prediction information is represented as a matrix of the same size as the current frame, and the sum of the elements in the same position in multiple groups of weight information corresponding to the multiple motion prediction information is equal to 1.

[0012] Furthermore, the multiple motion prediction information includes optical flow motion prediction information and an adaptive convolution kernel for representing the motion prediction information; and determining multiple candidate prediction frames corresponding to the current frame according to the multiple motion prediction information includes:

[0013] Acquire a first candidate prediction frame corresponding to the optical flow motion prediction information according to the optical flow motion prediction information and the reconstructed reference frame;

[0014] A second candidate prediction frame corresponding to the adaptive convolution kernel is obtained according to the adaptive convolution kernel and the reconstructed reference frame.

[0015] Furthermore, the weight information further includes weight information corresponding to at least one preset prediction frame; and obtaining the target prediction frame by using the weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames includes:

[0016] The target prediction frame is obtained by weighted summing the first candidate prediction frame, the second candidate prediction frame, the preset prediction frame and the corresponding weight information.

[0017] Furthermore, the preset prediction frame includes a reconstructed reference frame and / or a prediction frame whose elements are all zero.

[0018] Furthermore, a motion prediction model is used to obtain a variety of motion prediction information of the current frame and the reconstructed reference frame, including:

[0019] Processing the current frame and the reconstructed reference frame using the motion prediction model to obtain a first encoding stream of the multiple motion prediction information;

[0020] The first encoded bitstream is processed using the motion prediction model to obtain the multiple motion prediction information.

[0021] Furthermore, the method further comprises:

[0022] Acquire residual data of the current frame using the current frame and the target predicted frame;

[0023] The residual data is encoded to obtain a second encoded code stream.

[0024] Furthermore, the reference frame includes a plurality of frames, and a motion prediction model is used to obtain a plurality of motion prediction information of the current frame and the reconstructed reference frame, including:

[0025] For each reference reconstructed frame of the reference frame, using the motion prediction model to obtain a variety of motion prediction information of the current frame and each of the reconstructed reference frames;

[0026] Determining a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information includes:

[0027] Determine a branch prediction frame corresponding to the current frame according to the reconstructed reference frame and multiple motion prediction information of each of the reconstructed reference frames;

[0028] The target prediction frame is determined according to the multiple branch prediction frames corresponding to the multiple reconstructed reference frames.

[0029] In a second aspect, an embodiment of the present disclosure provides a video processing method, including:

[0030] Acquire a first coded code stream corresponding to the current frame and a reconstructed reference frame of at least one reference frame; the first coded code stream includes a coded code stream of motion prediction information between the current frame and the reconstructed reference frame of at least one reference frame;

[0031] Processing the first coded bitstream using a motion prediction model to obtain a variety of motion prediction information between the current frame and the reconstructed reference frame;

[0032] A target prediction frame corresponding to the current frame is determined according to the reconstructed reference frame and the multiple motion prediction information.

[0033] Further, determining a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information includes:

[0034] Determine a plurality of candidate prediction frames corresponding to the current frame according to the plurality of motion prediction information;

[0035] The target prediction frame is obtained by using weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames.

[0036] Furthermore, the weight information is obtained by the motion prediction model based on the current frame and the reconstructed reference frame; and / or, the weight information corresponding to each type of motion prediction information is represented as a matrix of the same size as the current frame, and the sum of the elements in the same position in multiple groups of weight information corresponding to the multiple motion prediction information is equal to 1.

[0037] Furthermore, the multiple motion prediction information includes optical flow motion prediction information and an adaptive convolution kernel for representing the motion prediction information; and obtaining the target prediction frame using weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames includes:

[0038] Acquire a first candidate prediction frame corresponding to the optical flow motion prediction information according to the optical flow motion prediction information and the reconstructed reference frame;

[0039] A second candidate prediction frame corresponding to the adaptive convolution kernel is obtained according to the adaptive convolution kernel and the reconstructed reference frame.

[0040] Furthermore, the weight information further includes weight information corresponding to at least one preset prediction frame; and determining a plurality of candidate prediction frames corresponding to the current frame according to the plurality of motion prediction information further includes:

[0041] The target prediction frame is obtained by weighted summing the first candidate prediction frame, the second candidate prediction frame, the preset prediction frame and the corresponding weight information.

[0042] Furthermore, the preset prediction frame includes a reconstructed reference frame and / or a prediction frame whose elements are all zero.

[0043] Furthermore, the method further comprises:

[0044] Decoding the second encoded code stream to obtain residual data of the current frame;

[0045] The current frame is obtained according to the target prediction frame and the residual data.

[0046] Furthermore, the first coded bitstream includes a plurality of groups of coded bitstreams corresponding to the plurality of reconstructed reference frames; the first coded bitstream is processed by using a motion prediction model to obtain a plurality of motion prediction information between the current frame and the reconstructed reference frame, including:

[0047] For each group of the first encoded bitstreams, using the motion prediction model to obtain a plurality of motion prediction information of the current frame and each of the reconstructed reference frames;

[0048] Determining a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information includes:

[0049] Determine a branch prediction frame corresponding to the current frame according to the reconstructed reference frame and multiple motion prediction information of each of the reconstructed reference frames;

[0050] The target prediction frame is determined according to a plurality of the branch candidate prediction frames corresponding to a plurality of the reconstructed reference frames.

[0051] In a third aspect, an embodiment of the present disclosure provides a video processing method, including:

[0052] Get the video to be processed;

[0053] Calling a preset service interface so that the preset service interface obtains a reconstructed reference frame of a current frame and at least one reference frame in a video to be processed, processes the current frame and the reconstructed reference frame using a motion prediction model to obtain multiple motion prediction information of the current frame, and determines a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information;

[0054] The target prediction frame is output.

[0055] In a fourth aspect, an embodiment of the present disclosure provides a video processing method, including:

[0056] Get the video to be processed;

[0057] Calling a preset service interface so that the preset service interface obtains a reconstructed reference frame of a current frame and at least one reference frame in a to-be-processed video, obtains a plurality of motion prediction information between the current frame and the reconstructed reference frame and a first coded bitstream corresponding to the plurality of motion prediction information by using a motion prediction model, determines a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the plurality of motion prediction information, and obtains a second coded bitstream of residual data between the current frame and the target prediction frame according to the target prediction frame and the motion prediction model;

[0058] The first encoded code stream and the second encoded code stream are output.

[0059] In a fifth aspect, an embodiment of the present disclosure provides a video processing method, including:

[0060] Acquire a first coded bitstream and a second coded bitstream; the first coded bitstream includes a coded bitstream of multiple motion prediction information between a current frame and a reconstructed reference frame of at least one reference frame;

[0061] calling a preset service interface so that the preset service interface processes the first coded bitstream using a motion prediction model to obtain a plurality of motion prediction information between the current frame and the reconstructed reference frame, and determines a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the plurality of motion prediction information;

[0062] The target prediction frame is output.

[0063] In a sixth aspect, an embodiment of the present disclosure provides a video processing method, including:

[0064] Acquire a first coded bitstream and a second coded bitstream; the first coded bitstream includes a coded bitstream of multiple motion prediction information between a current frame and a reconstructed reference frame of at least one reference frame; the second coded bitstream includes a coded bitstream of residual data between the current frame and a target prediction frame;

[0065] calling a preset service interface so that the preset service interface processes the first coded bitstream using a motion prediction model to obtain a plurality of motion prediction information between the current frame and the reconstructed reference frame, and decodes the second coded bitstream to obtain residual data between the current frame and the target prediction frame, determines a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the plurality of motion prediction information, and obtains the current frame according to the target prediction frame and the residual data;

[0066] The current frame is output.

[0067] In a seventh aspect, an embodiment of the present disclosure provides a video processing method, including:

[0068] Obtaining a reconstructed reference frame of a current frame and at least one reference frame in a video to be encoded;

[0069] Acquire a plurality of motion prediction information between the current frame and the reconstructed reference frame and a first encoding bitstream corresponding to the plurality of motion prediction information by using a motion prediction model;

[0070] Determine a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information;

[0071] Obtaining a second coded stream of residual data between the current frame and the target predicted frame according to the target predicted frame and the motion prediction model;

[0072] The first encoded code stream and the second encoded code stream are output.

[0073] In an eighth aspect, an embodiment of the present disclosure provides a video processing method, including:

[0074] Acquire a first coded bitstream and a second coded bitstream; the first coded bitstream includes a coded bitstream of multiple motion prediction information between a current frame and a reconstructed reference frame of at least one reference frame; the second coded bitstream includes a coded bitstream of residual data between the current frame and a target prediction frame;

[0075] Processing the first coded bitstream using a motion prediction model to obtain a variety of motion prediction information between the current frame and the reconstructed reference frame, and decoding the second coded bitstream to obtain residual data between the current frame and the target prediction frame;

[0076] Determine a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information;

[0077] Obtaining the current frame according to the target prediction frame and the residual data;

[0078] The current frame is output.

[0079] In a ninth aspect, an embodiment of the present disclosure provides a video processing device, including:

[0080] A first acquisition module is configured to acquire a current frame in the video to be processed and a reconstructed reference frame of at least one reference frame;

[0081] A second acquisition module is configured to acquire multiple motion prediction information of the current frame and the reconstructed reference frame using a motion prediction model;

[0082] The first determination module is configured to determine a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information.

[0083] In a tenth aspect, an embodiment of the present disclosure provides a video processing device, including:

[0084] A fourth acquisition module is configured to receive a first coded stream corresponding to the current frame and a reconstructed reference frame of at least one reference frame; the first coded stream includes a coded stream of motion prediction information between the current frame and the reconstructed reference frame of at least one reference frame;

[0085] A first processing module is configured to process the first coded bitstream using a motion prediction model to obtain a variety of motion prediction information between the current frame and the reconstructed reference frame;

[0086] The second determination module is configured to determine a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information.

[0087] In a ninth aspect, an embodiment of the present disclosure provides a video processing device, including:

[0088] A sixth acquisition module is configured to acquire a video to be processed;

[0089] A first calling module is configured to call a preset service interface so that the preset service interface obtains a reconstructed reference frame of a current frame and at least one reference frame in a video to be processed, processes the current frame and the reconstructed reference frame using a motion prediction model to obtain multiple motion prediction information of the current frame, and determines a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information;

[0090] The first output module is configured to output the target prediction frame.

[0091] In an eleventh aspect, an embodiment of the present disclosure provides a video processing device, including:

[0092] A seventh acquisition module is configured to acquire a video to be processed;

[0093] A second calling module is configured to call a preset service interface so that the preset service interface obtains a reconstructed reference frame of a current frame and at least one reference frame in a to-be-processed video, obtains a plurality of motion prediction information between the current frame and the reconstructed reference frame and a first coded bitstream corresponding to the plurality of motion prediction information by using a motion prediction model, determines a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the plurality of motion prediction information, and obtains a second coded bitstream of residual data between the current frame and the target prediction frame according to the target prediction frame and the motion prediction model;

[0094] The second output module is configured to output the first encoded code stream and the second encoded code stream.

[0095] In a twelfth aspect, an embodiment of the present disclosure provides a video processing device, including:

[0096] A seventh acquisition module is configured to acquire a first encoding stream and a second encoding stream; the first encoding stream includes an encoding stream of multiple motion prediction information between a current frame and a reconstructed reference frame of at least one reference frame;

[0097] a third calling module configured to call a preset service interface so that the preset service interface processes the first coded bitstream using a motion prediction model to obtain a plurality of motion prediction information between the current frame and the reconstructed reference frame, and determines a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the plurality of motion prediction information;

[0098] The third output module is configured to output the target prediction frame.

[0099] In a thirteenth aspect, an embodiment of the present disclosure provides a video processing device, including:

[0100] An eighth acquisition module is configured to acquire a first encoding stream and a second encoding stream; the first encoding stream includes an encoding stream of multiple motion prediction information between a current frame and a reconstructed reference frame of at least one reference frame; the second encoding stream includes an encoding stream of residual data between the current frame and a target prediction frame;

[0101] a fourth calling module, configured to call a preset service interface, so that the preset service interface processes the first coded stream using a motion prediction model to obtain a plurality of motion prediction information between the current frame and the reconstructed reference frame, and decodes the second coded stream to obtain residual data between the current frame and the target prediction frame, determines a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the plurality of motion prediction information, and obtains the current frame according to the target prediction frame and the residual data;

[0102] The fourth output module is configured to output the current frame.

[0103] In a thirteenth aspect, an embodiment of the present disclosure provides a video processing device, including:

[0104] A ninth acquisition module, configured to acquire a reconstructed reference frame of a current frame and at least one reference frame in the video to be encoded;

[0105] a tenth acquisition module, configured to acquire, by using a motion prediction model, a plurality of motion prediction information between the current frame and the reconstructed reference frame and a first encoding bitstream corresponding to the plurality of motion prediction information;

[0106] A third determination module is configured to determine a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information;

[0107] an eleventh acquisition module, configured to obtain a second encoded bit stream of residual data between the current frame and the target prediction frame according to the target prediction frame and the motion prediction model;

[0108] The fifth output module is configured to output the first encoded code stream and the second encoded code stream.

[0109] In a fourteenth aspect, an embodiment of the present disclosure provides a video processing device, including:

[0110] A twelfth acquisition module is configured to acquire a first encoding stream and a second encoding stream; the first encoding stream includes an encoding stream of multiple motion prediction information between a current frame and a reconstructed reference frame of at least one reference frame; the second encoding stream includes an encoding stream of residual data between the current frame and a target prediction frame;

[0111] A thirteenth acquisition module is configured to process the first coded bitstream using a motion prediction model to obtain a variety of motion prediction information between the current frame and the reconstructed reference frame, and decode the second coded bitstream to obtain residual data between the current frame and the target prediction frame;

[0112] A fourth determination module is configured to determine a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information;

[0113] a fifth determination module, configured to obtain the current frame according to the target prediction frame and the residual data;

[0114] The sixth output module is configured to output the current frame.

[0115] The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the functions.

[0116] In one possible design, the structure of the above-mentioned device includes a memory and a processor, the memory is used to store one or more computer instructions that support the above-mentioned device to execute the above-mentioned corresponding method, and the processor is configured to execute the computer instructions stored in the memory. The above-mentioned device may also include a communication interface for the above-mentioned device to communicate with other devices or communication networks.

[0117] In the thirteenth aspect, an embodiment of the present disclosure provides an electronic device, comprising a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method described in any one of the above aspects.

[0118] In a fourteenth aspect, an embodiment of the present disclosure provides a computer-readable storage medium for storing computer instructions used by any of the above-mentioned devices, which includes computer instructions involved in executing the method described in any of the above-mentioned aspects.

[0119] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0120] The disclosed embodiment obtains a plurality of different motion prediction information corresponding to a plurality of motion prediction modes for the current frame through a motion prediction model, and finally obtains a target prediction frame of the current frame by integrating the plurality of different motion prediction information. The technical solution of the disclosed embodiment improves the motion prediction performance of the video frame through diversified motion prediction modes during the motion prediction process.

[0121] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0122] Other features, objectives and advantages of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0123] Figure 1A flowchart showing a video processing method according to an embodiment of the present disclosure is shown;

[0124] Figure 2 A flowchart showing a video processing method according to another embodiment of the present disclosure;

[0125] Figure 3 A flowchart showing a video processing method according to another embodiment of the present disclosure;

[0126] Figure 4 A flowchart showing a video processing method according to another embodiment of the present disclosure;

[0127] Figure 5 A flowchart showing a video processing method according to another embodiment of the present disclosure;

[0128] Figure 6 A flowchart showing a video processing method according to another embodiment of the present disclosure;

[0129] Figure 7 A schematic diagram of an implementation flow of obtaining a predicted frame and residual data of a current frame by using a motion prediction model in a video encoding and decoding scenario according to an embodiment of the present disclosure is shown;

[0130] Figure 8 It is a structural schematic diagram of an electronic device suitable for implementing a video processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0131] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.

[0132] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate the presence of features, numbers, steps, behaviors, components, parts, or a combination thereof disclosed in the present specification, and are not intended to exclude the possibility that one or more other features, numbers, steps, behaviors, components, parts, or a combination thereof exist or are added.

[0133] It should also be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0134] The details of the embodiments of the present disclosure are described in detail below through specific examples.

[0135] Figure 1 FIG. 1 is a flow chart showing a video processing method according to an embodiment of the present disclosure. Figure 1 As shown, the video processing method includes the following steps:

[0136] In step S101, a reconstructed reference frame of a current frame and at least one reference frame in a video to be processed is obtained;

[0137] In step S102, a motion prediction model is used to obtain a variety of motion prediction information of the current frame and the reconstructed reference frame;

[0138] In step S103, a target prediction frame corresponding to the current frame is determined according to the reconstructed reference frame and the multiple motion prediction information.

[0139] In this embodiment, the video processing method can be executed at the encoding end, and the encoding end can be located at a user device, a server device, a cloud, etc. The video to be processed may include a series of video frames with a sequential relationship, the current frame may be the current video frame to be encoded, and the reference frame may be the video frame referenced when the current frame is encoded using the current encoding mode, the reference frame may be the previous frame, the next frame, etc. of the current frame, and the reference frame may include one or more, which may vary based on different encoding modes and are not limited here. It should be noted that when encoding the current frame, the reference frame is a video frame that has been encoded, and the reconstructed reference frame is a video frame obtained by decoding according to the encoding information of the reference frame.

[0140] The motion prediction model can be a machine self-learning model based on deep learning, for example, it can be a convolutional neural network model. The motion prediction model can be pre-trained, and during the training process, the motion prediction model can learn the motion relationship between the current sample frame and the reconstructed reference sample frame in the sample data, and then predict a variety of different motion prediction information between the current frame and the reconstructed reference frame based on the learned motion relationship. In some embodiments, a variety of different motion prediction information can be understood as motion prediction information obtained using different motion prediction modes, for example, it can include but is not limited to motion prediction information obtained by optical flow mode prediction and motion prediction information obtained by adaptive convolution kernel prediction, etc. It should be noted that the motion prediction mode here is only for example, and any motion prediction mode can be applied to the motion prediction model in the embodiment of the present disclosure.

[0141] In some embodiments, the motion prediction model may include an encoding part and a decoding part. The encoding part may encode and compress the input information after processing, and the decoding part may decode the encoded information obtained by encoding and compression. In the process of processing the input current frame and the reconstructed reference frame, the motion prediction model in the disclosed embodiment can not only obtain a variety of motion prediction information between the current frame and the reconstructed reference frame, but also compress and encode the various motion prediction information obtained by processing. It should be noted that the various motion prediction information output by the motion prediction model of the video encoding end is obtained after compression encoding by the encoding part and then decoding by the decoding part. In this way, the target prediction frame obtained by the video encoding end can be consistent with the target prediction frame obtained by the video decoding end after decoding the encoded code stream of the current frame.

[0142] After obtaining a variety of different motion prediction information in a variety of different motion prediction modes between the current frame and the reconstructed reference frame of the current frame using a motion prediction model, a target prediction frame corresponding to the current frame can be determined according to the variety of different motion prediction information.

[0143] The disclosed embodiment obtains a plurality of different motion prediction information corresponding to a plurality of motion prediction modes for the current frame through a motion prediction model, and finally obtains a target prediction frame of the current frame by integrating the plurality of different motion prediction information. The technical solution of the disclosed embodiment improves the motion prediction performance of the video frame through diversified motion prediction modes during the motion prediction process.

[0144] In an optional implementation of this embodiment, step S103, i.e., the step of determining the target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information, further includes the following steps:

[0145] Determine a plurality of candidate prediction frames corresponding to the current frame according to the plurality of motion prediction information;

[0146] The target prediction frame is obtained by using weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames.

[0147] In this optional implementation, after obtaining a variety of different motion prediction information using a motion prediction model, candidate prediction frames corresponding to the current frame can be obtained according to different motion prediction information. The method of obtaining the candidate prediction frames varies based on the different motion prediction modes corresponding to different motion prediction information. For example, in the optical flow mode, the corresponding motion prediction information, that is, the optical flow information, can be used to perform a warp operation (distortion operation) on the reconstructed reference frame to obtain the corresponding candidate prediction frame. In other modes, the corresponding motion prediction information can also be used to perform corresponding operations on the reconstructed reference frame to obtain the corresponding candidate prediction frame. The specific method varies according to the different motion prediction modes and is not limited here.

[0148] After obtaining the candidate prediction frames corresponding to each different motion prediction information, the weight information corresponding to the motion prediction information in different motion prediction modes can be used to perform weighted operations on the candidate prediction frames to obtain the target prediction frame. The weight information can be used to indicate the prediction accuracy of the corresponding motion prediction information on the corresponding image block of the current frame. The higher the prediction accuracy, the larger the weight value on the image block can be, and the lower the prediction accuracy, the smaller the corresponding weight value on the image block can be. The weight information can be pre-set using empirical values, or can be predicted using a motion prediction model.

[0149] In some embodiments, the weight information can be obtained by the motion prediction model according to the current frame and the reconstructed reference frame. In this embodiment, the motion prediction model processes the current frame and the reconstructed reference frame to obtain compressed multiple motion prediction information, and can also obtain weight information corresponding to the multiple motion prediction information.

[0150] In some embodiments, the weight information corresponding to each motion prediction information can be represented as a matrix of the same size as the current frame, and the sum of the elements at the same position in the multiple sets of weight information corresponding to the multiple motion prediction information is equal to 1. In this embodiment, the weight information can be represented as a matrix, the size of the matrix is ​​the same as the size of the current frame, and each bit in the matrix represents the prediction accuracy of each pixel in the current frame under the corresponding motion prediction mode, and the sum of the values ​​of the elements at the same position in the multiple weight information corresponding to the multiple motion prediction modes is equal to 1. In this way, different weights can be used for different image areas of the current frame to weight different motion prediction information to obtain the corresponding area information in the target prediction frame. For example, when two types of motion prediction information A and B are used, for the current image area of ​​the current frame, the weight value corresponding to the motion prediction information A is a, the weight value corresponding to the motion prediction information B is b, and a+b=1, then the information corresponding to the current image area block in the target prediction frame is aA+bB.

[0151] In an optional implementation of this embodiment, the multiple motion prediction information includes optical flow motion prediction information and an adaptive convolution kernel for representing the motion prediction information; and the step of determining multiple candidate prediction frames corresponding to the current frame according to the multiple motion prediction information further includes the following steps:

[0152] Acquire a first candidate prediction frame corresponding to the optical flow motion prediction information according to the optical flow motion prediction information and the reconstructed reference frame;

[0153] A second candidate prediction frame corresponding to the adaptive convolution kernel is obtained according to the adaptive convolution kernel and the reconstructed reference frame.

[0154] In this optional implementation, the motion prediction model can predict the optical flow motion prediction information between the current frame and the reconstructed reference frame based on the current frame and the reconstructed reference frame, and the optical flow motion prediction information includes the motion change information between the reconstructed reference frame and the current frame in the time domain. During the training process, the motion prediction model learns the optical flow changes between the current sample frame and the reconstructed reference sample frame in the sample data, and then outputs the optical flow motion prediction information for the input current frame and the reconstructed reference frame.

[0155] In some embodiments, the known optical flow information between the current frame and the reconstructed reference frame can also be used as the input of the motion prediction model, so that the motion prediction model can further process the current frame, the reconstructed reference value and the optical flow information to obtain a variety of motion prediction information, including the reconstructed optical flow information output by the motion prediction model. The known optical flow information can be determined using an existing method, and the reconstructed optical flow information is the optical flow information obtained by further optimizing the known optical flow information.

[0156] In addition, the motion prediction model can also predict an adaptive convolution kernel that can represent motion prediction information based on the current frame and the reconstructed reference frame. The predicted frame of the current frame can be obtained through the adaptive convolution kernel and the reconstructed reference frame in determining the residual data between the current frame and the reconstructed reference frame and / or in the decoding stage.

[0157] It is understandable that, in addition to the optical flow motion prediction information and the adaptive convolution kernel, the motion prediction model can also predict other motion prediction information corresponding to other motion prediction modes, and no limitation is made here.

[0158] For each type of motion prediction information, the corresponding candidate prediction frame can be obtained by using the motion prediction information and the reconstructed reference frame. For example, the first candidate prediction frame in the optical flow motion prediction mode can be obtained by performing a warp operation on the reconstructed reference frame using the optical flow motion prediction information, and the second candidate prediction frame can be obtained by performing a convolution operation on the reconstructed reference frame using an adaptive convolution kernel representing the motion prediction information; and when other motion prediction information in other motion modes is predicted using the motion prediction mode, other candidate prediction frames can be obtained using the other motion prediction information and the reconstructed reference frame.

[0159] In an optional implementation of this embodiment, the weight information further includes weight information corresponding to at least one preset prediction frame; and the step of obtaining the target prediction frame by using the weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames further includes the following steps:

[0160] The target prediction frame is obtained by weighted summing the first candidate prediction frame, the second candidate prediction frame, the preset prediction frame and the corresponding weight information.

[0161] In this optional implementation, one or more preset prediction frames may be pre-set. The preset prediction frames may be preset prediction frames set for special situations. The weight information may also include weights of the preset prediction frames.

[0162] In some embodiments, the preset prediction frame may include but is not limited to a prediction frame using a reconstructed reference frame as a prediction frame, a prediction frame in which all elements are set to 0, and the like.

[0163] For example, for a certain image area block in the current frame, when multiple motion prediction information is not accurate enough (for example, the predicted frame obtained using the multiple motion prediction information is not as similar to the current frame as the reconstructed reference frame), the reconstructed reference frame can be directly used as the prediction frame, and by setting weight information, when determining the information of the image area block in the target prediction frame, the proportion of the reconstructed reference frame information is larger, while the proportion of other multiple motion prediction information is smaller. For another example, for a certain image area block of the current frame, when multiple motion prediction information is not accurate enough (for example, the information in the predicted frame obtained using multiple motion prediction information and the reconstructed reference frame are significantly different from the information in the current frame), a prediction frame whose elements are all 0 can be used, and by setting weight information, when determining the information of the image area block in the target prediction frame, the proportion of the information of the predicted frame whose elements are all 0 is larger, while the proportion of other multiple motion prediction information is smaller. In this case, when calculating the residual data between the current frame and the target prediction frame, the information in the image area block in the current frame can be retained in the residual data as much as possible, so that the decoding end can obtain more accurate image information for the image area block when decoding the current frame.

[0164] The target prediction frame can be obtained by multiplying each candidate prediction frame (including the first candidate prediction frame, the second candidate prediction frame and the preset prediction frame) with the corresponding weight information and then adding them together.

[0165] In an optional implementation of this embodiment, the step of obtaining multiple motion prediction information of the current frame and the reconstructed reference frame by using a motion prediction model further includes the following steps:

[0166] Processing the current frame and the reconstructed reference frame using the motion prediction model to obtain a first encoding stream of the multiple motion prediction information;

[0167] The first encoded bitstream is processed using the motion prediction model to obtain the multiple motion prediction information.

[0168] In this optional implementation, the motion prediction model may include but is not limited to a coding part and a decoding part. Taking the motion prediction model using a convolutional neural network as an example, the decoding part in the motion prediction model may include but is not limited to a coding network and a code stream coding part. The coding network may be a downsampling network, that is, a network for compressing image information. The coding network may include multiple convolutional layers, and the stride of each convolutional layer may be set according to actual conditions, for example, it may be set to 2, that is, the coding network may be a convolutional neural network with 2 times downsampling. After the current frame and the reconstructed reference frame are downsampled by the coding network, a variety of motion prediction information in a variety of different motion modes for representing the motion change information between the current frame and the reconstructed reference frame may be obtained through the code stream coding part. The multiple motion prediction information may also obtain a first coded code stream after, for example, an information entropy coding process, and the first coded code stream is the compressed motion prediction information. For example, the code stream coding part may include a code word estimation network and an arithmetic coding part, and the implicit tensor obtained after downsampling by the coding network may be input into the code word estimation network for estimating the occurrence probability of each feature point in the implicit tensor. The codeword estimation network may also include an autoencoder structure consisting of a downsampling network and an upsampling network. After downsampling and upsampling the input implicit tensor, the context information may be used to estimate the occurrence probability of feature points in the implicit tensor, and then the implicit tensor may be quantized according to the estimated occurrence probability, and the quantized implicit tensor may be losslessly encoded using arithmetic coding to obtain a first encoded code stream.

[0169] The decoding part in the motion prediction model may include but is not limited to a decoding network and a bitstream decoding part. The first encoded bitstream is passed through the bitstream decoding part corresponding to the bitstream encoding part to obtain decoding information, and the decoding information is passed through the decoding network to obtain a variety of decoded motion prediction information. The decoding network can be an upsampling network, which can correspond to the encoding network, including multiple deconvolution layers, and the stride of each deconvolution layer is the same as the stride in the encoding network, depending on actual needs, and is not limited here. After sampling on the decoding network, a variety of motion prediction information is obtained. It should be noted that the purpose of decoding the first encoded bitstream through the decoding part of the motion prediction model at the video encoding end is to ensure that the target prediction frame and residual data obtained based on the various motion prediction information are consistent with the target prediction frame and residual data obtained at the video decoding end, so that the video decoding end can decode and obtain the current frame.

[0170] In an optional implementation of this embodiment, the method further includes the following steps:

[0171] Acquire residual data of the current frame using the current frame and the target predicted frame;

[0172] The residual data is encoded to obtain a second encoded code stream.

[0173] In this optional implementation, the target prediction frame obtained according to the motion prediction information between the current frame and the reconstructed reference frame can be understood as a video frame that can be predicted from the reconstructed reference frame and is relatively close to the current frame. The video decoding end can obtain the target prediction frame by decoding the first coded code stream corresponding to the motion prediction information and the reconstructed reference frame. However, in order to finally restore the information of the current frame, it is also necessary to obtain the residual data between the current frame and the target prediction frame. Therefore, after obtaining the target prediction frame, the video encoding end can obtain the residual data through the current frame and the target prediction frame, and then encode the residual data to obtain the second coded code stream. The compression coding of the residual data can be implemented using a residual data coding network, and the network structure of the residual data coding network can be the same as the coding network in the motion prediction network, but the model parameters can be different. The residual data can be obtained, for example, by directly subtracting the target prediction frame from the current frame. The first coded code stream and the second coded code stream can be sent to the video decoding end, and the video decoding end restores the target prediction frame according to the first coded code stream, and restores the residual data from the second coded code stream, and finally the decoded current frame can be obtained according to the target prediction frame and the residual data.

[0174] In an optional implementation of this embodiment, the reference frame includes multiple reference frames, and step S102, i.e., the step of obtaining multiple motion prediction information of the current frame and the reconstructed reference frame by using a motion prediction model, further includes the following steps:

[0175] For each reference reconstructed frame of the reference frame, using the motion prediction model to obtain a variety of motion prediction information of the current frame and each of the reconstructed reference frames;

[0176] Step S103, i.e., the step of determining the target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information, further comprises the following steps:

[0177] Determine a branch prediction frame corresponding to the current frame according to the reconstructed reference frame and multiple motion prediction information of each of the reconstructed reference frames;

[0178] The target prediction frame is determined according to the multiple branch prediction frames corresponding to the multiple reconstructed reference frames.

[0179] In this optional implementation, during the video encoding process, one reference frame or multiple reference frames can be selected for the current frame based on different encoding modes. In the case of selecting multiple reference frames to encode the current frame, the video processing method in the disclosed embodiment can be used to obtain a corresponding prediction frame for each reference frame. In order to distinguish it from the final target prediction frame, it is called a branch prediction frame in the disclosed embodiment, that is, one reference frame or one reconstructed reference frame corresponds to one branch prediction frame. For each reconstructed reference frame, the motion prediction model can obtain corresponding multiple motion prediction information, and then obtain the branch prediction frame corresponding to the reconstructed reference frame based on the multiple motion prediction information. Finally, the target prediction frame of the current frame can be obtained based on the multiple branch prediction frames corresponding to the multiple reconstructed reference frames. For example, the target prediction frame can be obtained by adding and averaging the multiple branch prediction frames.

[0180] Figure 2 FIG. 2 is a flowchart showing a video processing method according to another embodiment of the present disclosure. Figure 2 As shown, the video processing method includes the following steps:

[0181] In step S201, a first coded bitstream corresponding to the current frame and a reconstructed reference frame of at least one reference frame are obtained; the first coded bitstream includes a coded bitstream of motion prediction information between the current frame and the reconstructed reference frame of at least one reference frame;

[0182] In step S202, the first coded bitstream is processed using a motion prediction model to obtain a variety of motion prediction information between the current frame and the reconstructed reference frame;

[0183] In step S203, a target prediction frame corresponding to the current frame is determined according to the reconstructed reference frame and the multiple motion prediction information.

[0184] In this embodiment, the video processing method is executed at the video decoding end, and the video decoding end can be located at a user terminal, a server end, a cloud end, etc. The current frame can be the video frame to be decoded in the video to be decoded, the reference frame can be the video frame referenced when the current frame is encoded by the adopted encoding mode at the video encoding end, the reference frame can be the previous frame, the next frame, etc. of the current frame, and the reference frame can include one or more, which varies based on different encoding modes and is not limited here. It should be noted that when decoding the current frame, the reference frame is a video frame that has been decoded, and the reconstructed reference frame is a video frame obtained by decoding the encoded code stream of the reference frame.

[0185] The motion prediction model can be a machine self-learning model based on deep learning, for example, it can be a convolutional neural network model. The motion prediction model can be pre-trained, and during the training process, the motion prediction model can learn the motion relationship between the current sample frame and the reconstructed reference sample frame in the sample data, and then predict a variety of different motion prediction information between the current frame and the reconstructed reference frame based on the learned motion relationship. In some embodiments, a variety of different motion prediction information can be understood as motion prediction information obtained using different motion prediction modes, for example, it can include but is not limited to motion prediction information obtained by optical flow mode prediction and motion prediction information obtained by adaptive convolution kernel prediction, etc. It should be noted that the motion prediction mode here is only for example, and any motion prediction mode can be applied to the motion prediction model in the embodiment of the present disclosure.

[0186] In some embodiments, the motion prediction model may include an encoding part and a decoding part. The encoding part may encode and compress the input information after processing, and the decoding part may decode the encoded information obtained by encoding and compressing. The motion prediction model in the disclosed embodiment uses the decoding part to decode the first encoded code stream to obtain a variety of motion prediction information between the current frame and the reconstructed reference frame corresponding to the current frame. It should be noted that the motion prediction model is used at the video encoding end to process the input current frame and the reconstructed reference frame to obtain a variety of motion prediction information between the current frame and the reconstructed reference frame, and the obtained multiple motion prediction information is compressed and encoded to obtain the first encoded code stream.

[0187] After the video decoding end uses the motion prediction model to decode the first coded bitstream to obtain a variety of motion prediction information between the current frame and the reconstructed reference frame, the target prediction frame of the current frame can be obtained according to the reconstructed reference frame and the multiple motion prediction information.

[0188] The disclosed embodiment obtains a plurality of different motion prediction information corresponding to a plurality of motion prediction modes for the current frame through a motion prediction model, and finally obtains a target prediction frame of the current frame by integrating the plurality of different motion prediction information. The technical solution of the disclosed embodiment improves the motion prediction performance of the video frame through diversified motion prediction modes during the motion prediction process.

[0189] In an optional implementation of this embodiment, step S203, i.e., the step of determining the target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information, further includes the following steps:

[0190] Determine a plurality of candidate prediction frames corresponding to the current frame according to the plurality of motion prediction information;

[0191] The target prediction frame is obtained by using weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames.

[0192] In this optional implementation, after obtaining a variety of different motion prediction information using a motion prediction model, candidate prediction frames corresponding to the current frame can be obtained according to different motion prediction information. The method of obtaining the candidate prediction frames varies based on the different motion prediction modes corresponding to different motion prediction information. For example, in the optical flow mode, the corresponding motion prediction information, that is, the optical flow information, can be used to perform a warp operation (distortion operation) on the reconstructed reference frame to obtain the corresponding candidate prediction frame. In other modes, the corresponding motion prediction information can also be used to perform corresponding operations on the reconstructed reference frame to obtain the corresponding candidate prediction frame. The specific method varies according to the different motion prediction modes and is not limited here.

[0193] After obtaining the candidate prediction frames corresponding to each different motion prediction information, the weight information corresponding to the motion prediction information in different motion prediction modes can be used to perform weighted operations on the candidate prediction frames to obtain the target prediction frame. The weight information can be used to indicate the prediction accuracy of the corresponding motion prediction information on the corresponding image block of the current frame. The higher the prediction accuracy, the larger the weight value on the image block can be, and the lower the prediction accuracy, the smaller the corresponding weight value on the image block can be. At the video encoding end, the weight information can be pre-set using empirical values, or it can be predicted using a motion prediction model.

[0194] In some embodiments, the video encoding end may obtain weight information from the motion prediction model according to the current frame and the reconstructed reference frame. In this embodiment, the motion prediction model obtains multiple motion prediction information from the first encoding bitstream and also obtains weight information corresponding to the multiple motion prediction information.

[0195] In some embodiments, the weight information corresponding to each motion prediction information can be represented as a matrix of the same size as the current frame, and the sum of the elements at the same position in the multiple sets of weight information corresponding to the multiple motion prediction information is equal to 1. In this embodiment, the weight information can be represented as a matrix, the size of the matrix is ​​the same as the size of the current frame, and each bit in the matrix represents the prediction accuracy of each pixel in the current frame under the corresponding motion prediction mode, and the sum of the values ​​of the elements at the same position in the multiple weight information corresponding to the multiple motion prediction modes is equal to 1. In this way, different weights can be used for different image areas of the current frame to weight different motion prediction information to obtain the corresponding area information in the target prediction frame. For example, when two types of motion prediction information A and B are used, for the current image area of ​​the current frame, the weight value corresponding to the motion prediction information A is a, the weight value corresponding to the motion prediction information B is b, and a+b=1, then the information corresponding to the current image area block in the target prediction frame is aA+bB.

[0196] In an optional implementation of this embodiment, the multiple motion prediction information includes optical flow motion prediction information and an adaptive convolution kernel for representing the motion prediction information; and the step of determining multiple candidate prediction frames corresponding to the current frame according to the multiple motion prediction information further includes the following steps:

[0197] Acquire a first candidate prediction frame corresponding to the optical flow motion prediction information according to the optical flow motion prediction information and the reconstructed reference frame;

[0198] A second candidate prediction frame corresponding to the adaptive convolution kernel is obtained according to the adaptive convolution kernel and the reconstructed reference frame.

[0199] In this optional implementation, the motion prediction model at the video encoding end can predict the optical flow motion prediction information between the current frame and the reconstructed reference frame based on the current frame and the reconstructed reference frame. The optical flow motion prediction information includes the motion change information between the reconstructed reference frame and the current frame in the time domain. During the training process, the motion prediction model learns the optical flow changes between the current sample frame and the reconstructed reference sample frame in the sample data, and then outputs the optical flow motion prediction information for the input current frame and the reconstructed reference frame. Therefore, the optical flow motion prediction information can be obtained by decoding the first encoded bitstream at the video decoding end.

[0200] In addition, the motion prediction model at the video encoding end can also predict an adaptive convolution kernel that can represent motion prediction information based on the current frame and the reconstructed reference frame, so the adaptive convolution kernel can be obtained by decoding the first encoded bitstream at the video decoding end. The residual data between the current frame and the reconstructed reference frame and the target prediction frame of the current frame can be obtained through the adaptive convolution kernel and the reconstructed reference frame.

[0201] It is understandable that, in addition to the optical flow motion prediction information and the adaptive convolution kernel, the motion prediction model can also predict other motion prediction information corresponding to other motion prediction modes, and no limitation is made here.

[0202] For each type of motion prediction information, the corresponding candidate prediction frame can be obtained by using the motion prediction information and the reconstructed reference frame. For example, the first candidate prediction frame in the optical flow motion prediction mode can be obtained by performing a warp operation on the reconstructed reference frame using the optical flow motion prediction information, and the second candidate prediction frame can be obtained by performing a convolution operation on the reconstructed reference frame using an adaptive convolution kernel representing the motion prediction information; and when other motion prediction information in other motion modes is predicted using the motion prediction mode, other candidate prediction frames can be obtained using the other motion prediction information and the reconstructed reference frame.

[0203] In an optional implementation of this embodiment, the weight information further includes weight information corresponding to at least one preset prediction frame; and the step of obtaining the target prediction frame by using the weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames further includes the following steps:

[0204] The target prediction frame is obtained by weighted summing the first candidate prediction frame, the second candidate prediction frame, the preset prediction frame and the corresponding weight information.

[0205] In this optional implementation, one or more preset prediction frames may be pre-set. The preset prediction frames may be preset prediction frames set for special situations. The weight information may also include weights of the preset prediction frames.

[0206] In some embodiments, the preset prediction frame may include but is not limited to a prediction frame using a reconstructed reference frame as a prediction frame, a prediction frame in which all elements are set to 0, and the like.

[0207] For example, for a certain image area block in the current frame, when multiple motion prediction information is not accurate enough (for example, the predicted frame obtained using the multiple motion prediction information is not as similar to the current frame as the reconstructed reference frame), the reconstructed reference frame can be directly used as the prediction frame, and by setting weight information, when determining the information of the image area block in the target prediction frame, the proportion of the reconstructed reference frame information is larger, while the proportion of other multiple motion prediction information is smaller. For another example, for a certain image area block of the current frame, when multiple motion prediction information is not accurate enough (for example, the information in the predicted frame obtained using multiple motion prediction information and the reconstructed reference frame are significantly different from the information in the current frame), a prediction frame whose elements are all 0 can be used, and by setting weight information, when determining the information of the image area block in the target prediction frame, the proportion of the information of the predicted frame whose elements are all 0 is larger, while the proportion of other multiple motion prediction information is smaller. In this case, during the encoding process at the video encoding end, when calculating the residual data between the current frame and the target prediction frame, the information in the image area block in the current frame can be retained in the residual data as much as possible, so that the video decoding end can obtain more accurate image information for the image area block when decoding and obtaining the current frame.

[0208] The target prediction frame can be obtained by multiplying each candidate prediction frame (including the first candidate prediction frame, the second candidate prediction frame and the preset prediction frame) with the corresponding weight information and then adding them together.

[0209] In an optional implementation of this embodiment, the method further includes the following steps:

[0210] Decoding the second encoded code stream to obtain residual data of the current frame;

[0211] The current frame is obtained according to the target prediction frame and the residual data.

[0212] In this optional implementation, the target prediction frame can be understood as a video frame that can be predicted from the reconstructed reference frame and is relatively close to the current frame. The video decoding end can obtain the target prediction frame by decoding the first coded code stream corresponding to the motion prediction information and the reconstructed reference frame. In order to decode the current frame, it is also necessary to decode the second coded code stream to obtain the residual data between the current frame and the target prediction frame. The second coded code stream can be decoded using a residual data decoding network, and the network structure of the residual data decoding network can be the same as the decoding network in the motion prediction network, but the model parameters can be different. After decoding to obtain the target prediction frame and the residual data, the current frame can be obtained. For example, the current frame can be obtained by adding the target prediction frame and the residual data.

[0213] In an optional implementation of this embodiment, the first coded bitstream includes multiple groups of coded bitstreams corresponding to the multiple reconstructed reference frames; in step S202, the step of using a motion prediction model to process the first coded bitstream to obtain multiple motion prediction information between the current frame and the reconstructed reference frame further includes the following steps:

[0214] For each group of the first encoded bitstreams, using the motion prediction model to obtain a plurality of motion prediction information of the current frame and each of the reconstructed reference frames;

[0215] In step S203, the step of determining the target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information further includes the following steps:

[0216] Determine a branch prediction frame corresponding to the current frame according to the reconstructed reference frame and multiple motion prediction information of each of the reconstructed reference frames;

[0217] The target prediction frame is determined according to a plurality of the branch candidate prediction frames corresponding to a plurality of the reconstructed reference frames.

[0218] In this optional implementation, at the video encoding end, when encoding the current frame, one reference frame or multiple reference frames can be selected for encoding due to different encoding modes. In the case where multiple reference frames are selected to encode the current frame, the video encoding end can use the video processing method in the embodiment of the present disclosure to obtain a corresponding prediction frame for each reference frame. In order to distinguish it from the final target prediction frame, it is called a branch prediction frame in the embodiment of the present disclosure, that is, one reference frame or one reconstructed reference frame corresponds to one branch prediction frame. The first coded stream obtained by the video decoding end includes multiple groups of motion prediction information corresponding to each branch prediction frame, each group of motion prediction information corresponds to a reconstructed reference frame, and includes multiple motion prediction information. The video decoding end obtains multiple groups of motion prediction information by decoding the first coded stream, and then can obtain the branch prediction frame corresponding to the corresponding reconstructed reference frame according to the multiple motion prediction information in each group. Finally, multiple branch prediction frames corresponding to multiple reconstructed reference frames can be obtained, and the target prediction frame of the current frame can be obtained according to the multiple branch prediction frames corresponding to the multiple reconstructed reference frames. For example, the target prediction frame can be obtained by adding and averaging multiple branch prediction frames.

[0219] Figure 3 FIG. 4 is a flowchart of a video processing method according to another embodiment of the present disclosure. Figure 3 As shown, the video processing method includes the following steps:

[0220] In step S301, a video to be processed is obtained;

[0221] In step S302, a preset service interface is called so that the preset service interface obtains a reconstructed reference frame of a current frame and at least one reference frame in the video to be processed, processes the current frame and the reconstructed reference frame using a motion prediction model to obtain multiple motion prediction information of the current frame, and determines a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information;

[0222] In step S303, the target prediction frame is output.

[0223] In this embodiment, the video processing method can be executed in the cloud. The preset service interface can be pre-deployed in the cloud, and the preset service interface can be a SaaS (Software-as-a-service) interface. The demander can obtain the right to use the preset service interface in advance, and can process the video to be processed by calling the preset service when necessary, so as to obtain the target prediction frame of the current frame in the video to be processed.

[0224] The video to be processed may include a series of video frames with a sequential relationship, the current frame may be the current video frame to be encoded, and the reference frame may be the video frame referenced when the current frame is encoded using the current encoding mode, the reference frame may be the previous frame, the next frame, etc. of the current frame, and the reference frame may include one or more, which may vary based on different encoding modes and are not limited here. It should be noted that when encoding the current frame, the reference frame is a video frame that has been encoded, and the reconstructed reference frame is a video frame obtained by decoding according to the encoding information of the reference frame.

[0225] The motion prediction model can be a machine self-learning model based on deep learning, for example, it can be a convolutional neural network model. The motion prediction model can be pre-trained, and during the training process, the motion prediction model can learn the motion relationship between the current sample frame and the reconstructed reference sample frame in the sample data, and then predict a variety of different motion prediction information between the current frame and the reconstructed reference frame based on the learned motion relationship. In some embodiments, a variety of different motion prediction information can be understood as motion prediction information obtained using different motion prediction modes, for example, it can include but is not limited to motion prediction information obtained by optical flow mode prediction and motion prediction information obtained by adaptive convolution kernel prediction, etc. It should be noted that the motion prediction mode here is only for example, and any motion prediction mode can be applied to the motion prediction model in the embodiment of the present disclosure.

[0226] In some embodiments, the motion prediction model may include an encoding part and a decoding part. The encoding part may encode and compress the input information after processing, and the decoding part may decode the encoded information obtained by encoding and compression. In the process of processing the input current frame and the reconstructed reference frame, the motion prediction model in the disclosed embodiment can not only obtain a variety of motion prediction information between the current frame and the reconstructed reference frame, but also compress and encode the various motion prediction information obtained by processing. It should be noted that the various motion prediction information output by the motion prediction model of the video encoding end is obtained after compression encoding by the encoding part and then decoding by the decoding part. In this way, the target prediction frame obtained by the video encoding end can be consistent with the target prediction frame obtained by the video decoding end after decoding the encoded code stream of the current frame.

[0227] After obtaining a variety of different motion prediction information in a variety of different motion prediction modes between the current frame and the reconstructed reference frame of the current frame using a motion prediction model, a target prediction frame corresponding to the current frame can be determined according to the variety of different motion prediction information.

[0228] The disclosed embodiment can call a preset service interface in response to a request from a demander, so that the preset service interface obtains a plurality of different motion prediction information corresponding to a plurality of motion prediction modes for the current frame through a motion prediction model, and finally obtains a target prediction frame of the current frame by integrating the plurality of different motion prediction information. The technical solution of the disclosed embodiment improves the motion prediction performance of the video frame through diversified motion prediction modes during the motion prediction process.

[0229] Figure 4 FIG. 4 is a flowchart of a video processing method according to another embodiment of the present disclosure. Figure 4 As shown, the video processing method includes the following steps:

[0230] In step S401, a video to be processed is obtained;

[0231] In step S402, a preset service interface is called so that the preset service interface obtains a reconstructed reference frame of a current frame and at least one reference frame in a video to be processed, a plurality of motion prediction information between the current frame and the reconstructed reference frame and a first coded bitstream corresponding to the plurality of motion prediction information are obtained by using a motion prediction model, a target prediction frame corresponding to the current frame is determined according to the reconstructed reference frame and the plurality of motion prediction information, and a second coded bitstream of residual data between the current frame and the target prediction frame is obtained according to the target prediction frame and the motion prediction model;

[0232] In step S403, the first encoded code stream and the second encoded code stream are output.

[0233] In this embodiment, the video processing method can be executed in the cloud. The preset service interface can be pre-deployed in the cloud, and the preset service interface can be a SaaS (Software-as-a-service) interface. The demander can obtain the right to use the preset service interface in advance, and can process the video to be processed by calling the preset service when necessary, so as to obtain the encoded code stream of the current frame in the video to be processed.

[0234] The video to be processed may include a series of video frames with a sequential relationship, the current frame may be the current video frame to be encoded, and the reference frame may be the video frame referenced when the current frame is encoded using the current encoding mode, the reference frame may be the previous frame, the next frame, etc. of the current frame, and the reference frame may include one or more, which may vary based on different encoding modes and are not limited here. It should be noted that when encoding the current frame, the reference frame is a video frame that has been encoded, and the reconstructed reference frame is a video frame obtained by decoding according to the encoding information of the reference frame.

[0235] The motion prediction model can be a machine self-learning model based on deep learning, for example, it can be a convolutional neural network model. The motion prediction model can be pre-trained, and during the training process, the motion prediction model can learn the motion relationship between the current sample frame and the reconstructed reference sample frame in the sample data, and then predict a variety of different motion prediction information between the current frame and the reconstructed reference frame based on the learned motion relationship. In some embodiments, a variety of different motion prediction information can be understood as motion prediction information obtained using different motion prediction modes, for example, it can include but is not limited to motion prediction information obtained by optical flow mode prediction and motion prediction information obtained by adaptive convolution kernel prediction, etc. It should be noted that the motion prediction mode here is only for example, and any motion prediction mode can be applied to the motion prediction model in the embodiment of the present disclosure.

[0236] In some embodiments, the motion prediction model may include an encoding part and a decoding part. The encoding part may encode and compress the input information after processing, and the decoding part may decode the encoded information obtained by encoding and compression. In the process of processing the input current frame and the reconstructed reference frame, the motion prediction model in the disclosed embodiment can not only obtain a variety of motion prediction information between the current frame and the reconstructed reference frame, but also compress and encode the various motion prediction information obtained by processing. It should be noted that the various motion prediction information output by the motion prediction model of the video encoding end is obtained after compression encoding by the encoding part and then decoding by the decoding part. In this way, the target prediction frame obtained by the video encoding end can be consistent with the target prediction frame obtained by the video decoding end after decoding the encoded code stream of the current frame.

[0237] Taking the motion prediction model using a convolutional neural network as an example, the decoding part in the motion prediction model may include but is not limited to a coding network and a bitstream coding part. The coding network may be a downsampling network, that is, a network for compressing image information. The coding network may include multiple convolutional layers, and the stride of each convolutional layer may be set according to actual conditions, for example, it may be set to 2, that is, the coding network may be a 2-fold downsampling convolutional neural network. After the current frame and the reconstructed reference frame are downsampled by the coding network, a variety of motion prediction information in a variety of different motion modes for representing the motion change information between the current frame and the reconstructed reference frame may be obtained through the bitstream coding part. The multiple motion prediction information may also be obtained through, for example, an information entropy coding process to obtain a first coded bitstream, and the first coded bitstream is the compressed motion prediction information. For example, the bitstream coding part may include a codeword estimation network and an arithmetic coding part, and the implicit tensor obtained after downsampling by the coding network may be input into the codeword estimation network to estimate the probability of occurrence of each feature point in the implicit tensor. The codeword estimation network may also include an autoencoder structure consisting of a downsampling network and an upsampling network. After downsampling and upsampling the input implicit tensor, the context information may be used to estimate the occurrence probability of feature points in the implicit tensor, and then the implicit tensor may be quantized according to the estimated occurrence probability, and the quantized implicit tensor may be losslessly encoded using arithmetic coding to obtain a first encoded code stream.

[0238] The decoding part in the motion prediction model may include but is not limited to a decoding network and a bitstream decoding part. The first encoded bitstream is passed through the bitstream decoding part corresponding to the bitstream encoding part to obtain decoding information, and the decoding information is passed through the decoding network to obtain a variety of decoded motion prediction information. The decoding network can be an upsampling network, which can correspond to the encoding network, including multiple deconvolution layers, and the stride of each deconvolution layer is the same as the stride in the encoding network, depending on actual needs, and is not limited here. After sampling on the decoding network, a variety of motion prediction information is obtained. It should be noted that the purpose of decoding the first encoded bitstream through the decoding part of the motion prediction model at the video encoding end is to ensure that the target prediction frame and residual data obtained based on the various motion prediction information are consistent with the target prediction frame and residual data obtained at the video decoding end, so that the video decoding end can decode and obtain the current frame.

[0239] After obtaining a variety of different motion prediction information in a variety of different motion prediction modes between the current frame and the reconstructed reference frame of the current frame using a motion prediction model, a target prediction frame corresponding to the current frame can be determined according to the variety of different motion prediction information.

[0240] The target prediction frame obtained according to the motion prediction information between the current frame and the reconstructed reference frame can be understood as a video frame that can be predicted from the reconstructed reference frame and is relatively close to the current frame. The video decoding end can obtain the target prediction frame according to the first coded code stream corresponding to the motion prediction information and the reconstructed reference frame. However, in order to finally restore the information of the current frame, it is also necessary to obtain the residual data between the current frame and the target prediction frame. Therefore, after obtaining the target prediction frame, the video encoding end can obtain the residual data through the current frame and the target prediction frame, and then encode the residual data to obtain the second coded code stream. The residual data can be obtained, for example, by directly subtracting the target prediction frame from the current frame. Both the first coded code stream and the second coded code stream can be sent to the video decoding end, and the video decoding end restores the target prediction frame according to the first coded code stream, and restores the residual data from the second coded code stream. Finally, the decoded current frame can be obtained according to the target prediction frame and the residual data.

[0241] The disclosed embodiment can call a preset service interface in response to a request from a demander, so that the preset service interface obtains a plurality of different motion prediction information corresponding to a plurality of motion prediction modes and their corresponding first coded bitstreams for the current frame through a motion prediction model, and finally obtains a target prediction frame of the current frame by integrating a plurality of different motion prediction information, and then obtains a second coded bitstream of residual data between the current frame and the target prediction value according to the target prediction frame and the current frame, and finally obtains the coded bitstream of the current frame. The technical solution of the disclosed embodiment improves the motion prediction performance of the video frame through a variety of motion prediction modes during the motion prediction process, thereby improving the coding accuracy of the video frame.

[0242] Figure 5 FIG. 2 is a flowchart showing a video processing method according to another embodiment of the present disclosure. Figure 5 As shown, the video processing method includes the following steps:

[0243] In step S501, a first coded bitstream is obtained; the first coded bitstream includes a coded bitstream of multiple motion prediction information between a current frame and a reconstructed reference frame of at least one reference frame;

[0244] In step S502, a preset service interface is called so that the preset service interface processes the first coded bitstream using a motion prediction model to obtain a plurality of motion prediction information between the current frame and the reconstructed reference frame, and determines a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the plurality of motion prediction information;

[0245] In step S503, the target prediction frame is output.

[0246] In this embodiment, the video processing method can be executed in the cloud. The preset service interface can be pre-deployed in the cloud, and the preset service interface can be a SaaS (Software-as-a-service) interface. The demander can obtain the right to use the preset service interface in advance, and can process the encoded bit stream by calling the preset service when necessary, so as to obtain the target prediction frame of the current frame in the video to be processed.

[0247] The current frame may be a video frame to be decoded in the video to be decoded, the reference frame may be a video frame referenced when the current frame is encoded using the encoding mode adopted at the video encoding end, the reference frame may be a previous frame, a next frame, etc. of the current frame, and the reference frame may include one or more, which may vary based on different encoding modes and are not limited here. It should be noted that when decoding the current frame, the reference frame is a video frame that has been decoded, and the reconstructed reference frame is a video frame obtained by decoding the encoded code stream of the reference frame.

[0248] The motion prediction model can be a machine self-learning model based on deep learning, for example, it can be a convolutional neural network model. The motion prediction model can be pre-trained, and during the training process, the motion prediction model can learn the motion relationship between the current sample frame and the reconstructed reference sample frame in the sample data, and then predict a variety of different motion prediction information between the current frame and the reconstructed reference frame based on the learned motion relationship. In some embodiments, a variety of different motion prediction information can be understood as motion prediction information obtained using different motion prediction modes, for example, it can include but is not limited to motion prediction information obtained by optical flow mode prediction and motion prediction information obtained by adaptive convolution kernel prediction, etc. It should be noted that the motion prediction mode here is only for example, and any motion prediction mode can be applied to the motion prediction model in the embodiment of the present disclosure.

[0249] In some embodiments, the motion prediction model may include an encoding part and a decoding part. The encoding part may encode and compress the input information after processing, and the decoding part may decode the encoded information obtained by encoding and compressing. The motion prediction model in the disclosed embodiment uses the decoding part to decode the first encoded code stream to obtain a variety of motion prediction information between the current frame and the reconstructed reference frame corresponding to the current frame. It should be noted that the motion prediction model is used at the video encoding end to process the input current frame and the reconstructed reference frame to obtain a variety of motion prediction information between the current frame and the reconstructed reference frame, and the obtained multiple motion prediction information is compressed and encoded to obtain the first encoded code stream.

[0250] After the video decoding end uses the motion prediction model to decode the first coded bitstream to obtain a variety of motion prediction information between the current frame and the reconstructed reference frame, the target prediction frame of the current frame can be obtained according to the reconstructed reference frame and the multiple motion prediction information.

[0251] The disclosed embodiment obtains a plurality of different motion prediction information corresponding to a plurality of motion prediction modes for the current frame through a motion prediction model, and finally obtains a target prediction frame of the current frame by integrating the plurality of different motion prediction information. The technical solution of the disclosed embodiment improves the motion prediction performance of the video frame through diversified motion prediction modes during the motion prediction process.

[0252] Figure 6 FIG. 4 is a flowchart of a video processing method according to another embodiment of the present disclosure. Figure 6 As shown, the video processing method includes the following steps:

[0253] In step S601, a first coded bitstream and a second coded bitstream are obtained; the first coded bitstream includes a coded bitstream of multiple motion prediction information between a current frame and a reconstructed reference frame of at least one reference frame; the second coded bitstream includes a coded bitstream of residual data between the current frame and a target prediction frame;

[0254] In step S602, a preset service interface is called so that the preset service interface processes the first coded bitstream using a motion prediction model to obtain a plurality of motion prediction information between the current frame and the reconstructed reference frame, and decodes the second coded bitstream to obtain residual data between the current frame and the target prediction frame, determines a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the plurality of motion prediction information, and obtains the current frame according to the target prediction frame and the residual data;

[0255] In step S603, the current frame is output.

[0256] In this embodiment, the video processing method can be executed in the cloud. The preset service interface can be pre-deployed in the cloud, and the preset service interface can be a SaaS (Software-as-a-service) interface. The demander can obtain the right to use the preset service interface in advance, and can process the encoded bit stream by calling the preset service when necessary, so as to decode and obtain the current frame in the video to be processed.

[0257] The current frame may be a video frame to be decoded in the video to be decoded, the reference frame may be a video frame referenced when the current frame is encoded using the encoding mode adopted at the video encoding end, the reference frame may be a previous frame, a next frame, etc. of the current frame, and the reference frame may include one or more, which may vary based on different encoding modes and are not limited here. It should be noted that when decoding the current frame, the reference frame is a video frame that has been decoded, and the reconstructed reference frame is a video frame obtained by decoding the encoded code stream of the reference frame.

[0258] The motion prediction model can be a machine self-learning model based on deep learning, for example, it can be a convolutional neural network model. The motion prediction model can be pre-trained, and during the training process, the motion prediction model can learn the motion relationship between the current sample frame and the reconstructed reference sample frame in the sample data, and then predict a variety of different motion prediction information between the current frame and the reconstructed reference frame based on the learned motion relationship. In some embodiments, a variety of different motion prediction information can be understood as motion prediction information obtained using different motion prediction modes, for example, it can include but is not limited to motion prediction information obtained by optical flow mode prediction and motion prediction information obtained by adaptive convolution kernel prediction, etc. It should be noted that the motion prediction mode here is only for example, and any motion prediction mode can be applied to the motion prediction model in the embodiment of the present disclosure.

[0259] In some embodiments, the motion prediction model may include an encoding part and a decoding part. The encoding part may encode and compress the input information after processing, and the decoding part may decode the encoded information obtained by encoding and compressing. The motion prediction model in the disclosed embodiment uses the decoding part to decode the first encoded code stream to obtain a variety of motion prediction information between the current frame and the reconstructed reference frame corresponding to the current frame. It should be noted that the motion prediction model is used at the video encoding end to process the input current frame and the reconstructed reference frame to obtain a variety of motion prediction information between the current frame and the reconstructed reference frame, and the obtained multiple motion prediction information is compressed and encoded to obtain the first encoded code stream.

[0260] After the video decoding end uses the motion prediction model to decode the first coded bitstream to obtain a variety of motion prediction information between the current frame and the reconstructed reference frame, the target prediction frame of the current frame can be obtained according to the reconstructed reference frame and the multiple motion prediction information.

[0261] The target prediction frame can be understood as a video frame that can be predicted from the reconstructed reference frame and is relatively close to the current frame. The video decoding end can obtain the target prediction frame by decoding the first coded code stream corresponding to the motion prediction information and the reconstructed reference frame. In order to decode the current frame, it is also necessary to decode the residual data between the current frame and the target prediction frame according to the second coded code stream. After decoding the target prediction frame and the residual data, the current frame can be obtained. For example, the current frame can be obtained by adding the target prediction frame and the residual data.

[0262] The disclosed embodiment can call a preset service interface in response to a request from a demander, so that the preset service interface can decode the first coded stream through a motion prediction model for the current frame to obtain a variety of different motion prediction information corresponding to a variety of motion prediction modes, and finally obtain a target prediction frame of the current frame by integrating a variety of different motion prediction information, and then use the motion prediction model to decode the second coded stream to obtain residual data between the current frame and the target prediction frame, and obtain the decoded current frame based on the target prediction frame and the residual data, and finally output the decoded current frame. The technical solution of the disclosed embodiment improves the motion prediction performance of the video frame through a variety of motion prediction modes during the motion prediction process, thereby improving the decoding accuracy of the video frame.

[0263] According to another embodiment of the present invention, the video processing method comprises the following steps:

[0264] Obtaining a reconstructed reference frame of a current frame and at least one reference frame in a video to be encoded;

[0265] Acquire a plurality of motion prediction information between the current frame and the reconstructed reference frame and a first encoding bitstream corresponding to the plurality of motion prediction information by using a motion prediction model;

[0266] Determine a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information;

[0267] Obtaining a second coded stream of residual data between the current frame and the target predicted frame according to the target predicted frame and the motion prediction model;

[0268] The first encoded code stream and the second encoded code stream are output.

[0269] In this embodiment, the video to be processed may include a series of video frames with a sequential relationship, the current frame may be the current video frame to be encoded, and the reference frame may be the video frame referenced when the current frame is encoded using the current encoding mode, the reference frame may be the previous frame, the next frame, etc. of the current frame, and the reference frame may include one or more, which may vary based on different encoding modes and are not limited here. It should be noted that when encoding the current frame, the reference frame is a video frame that has been encoded, and the reconstructed reference frame is a video frame obtained by decoding according to the encoding information of the reference frame.

[0270] The video processing method proposed in the embodiment of the present disclosure can be executed at the video encoding end. After receiving the video to be processed provided by the user or other relevant parties, the video to be processed is encoded and processed, and finally a first encoded code stream and a second encoded code stream are output. The first encoded code stream and the second encoded code stream can be stored in a storage medium or sent to a video decoding end through a network for decoding. For specific details of the embodiment of the present disclosure, please refer to the above Figure 4 The descriptions in the illustrated embodiment and related embodiments will not be repeated here.

[0271] According to another embodiment of the present invention, the video processing method comprises the following steps:

[0272] Acquire a first coded bitstream and a second coded bitstream; the first coded bitstream includes a coded bitstream of multiple motion prediction information between a current frame and a reconstructed reference frame of at least one reference frame; the second coded bitstream includes a coded bitstream of residual data between the current frame and a target prediction frame;

[0273] Processing the first coded bitstream using a motion prediction model to obtain a variety of motion prediction information between the current frame and the reconstructed reference frame, and decoding the second coded bitstream to obtain residual data between the current frame and the target prediction frame;

[0274] Determine a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information;

[0275] Obtaining the current frame according to the target prediction frame and the residual data;

[0276] The current frame is output.

[0277] In this embodiment, the current frame may be a video frame to be decoded in the video to be decoded, the reference frame may be a video frame referenced when the current frame is encoded using the adopted encoding mode at the video encoding end, the reference frame may be a previous frame, a next frame, etc. of the current frame, and the reference frame may include one or more, which may vary based on different encoding modes and are not limited here. It should be noted that when decoding the current frame, the reference frame is a video frame that has been decoded, and the reconstructed reference frame is a video frame obtained by decoding the encoded code stream of the reference frame.

[0278] The video processing method proposed in the embodiment of the present disclosure can be executed at the video decoding end. After receiving the first encoded code stream and the second encoded code stream provided by the user or other relevant parties, the first encoded code stream and the second encoded code stream are decoded and processed, and finally a video frame is output. The decoded video frame can be output through the display screen of the electronic device. For specific details of the embodiment of the present disclosure, please refer to the above Figure 6 The descriptions in the illustrated embodiment and related embodiments will not be repeated here.

[0279] The following example illustrates the process of obtaining target prediction frames and residual data during video encoding and decoding using the embodiments of the present disclosure.

[0280] Figure 7 A schematic diagram of an implementation flow of obtaining a predicted frame and residual data of a current frame by using a motion prediction model in a video encoding and decoding scenario according to an embodiment of the present disclosure is shown. Figure 7 As shown, the motion prediction model may include but is not limited to an encoding network and a decoding network. At the video encoding end, the video encoding of the current frame can be divided into two stages: the first stage includes the prediction and compression encoding of various motion prediction information; the second stage includes the compression encoding of the residual data of the current frame.

[0281] In the first stage, the current frame x t and reconstruct the reference frame The encoding network input to the motion prediction model, in this embodiment, also includes the current frame x t and reconstruct the reference frame The optical flow information flow between can be obtained in a known manner. Reconstruct the reference frame It can be obtained based on the information obtained during the encoding process of the reference frame, that is, it can be obtained based on the predicted frame and residual data of the reference frame. The encoding network can be composed of 4 convolutional layers with a stride of 2. The current frame x t , reconstruct the reference frame And the optical flow information is transformed into an implicit tensor after passing through the encoding network, and the implicit tensor enters the codeword estimation network; the codeword estimation network Figure 7Not shown in the figure, its structure is an autoencoder structure including a downsampling network and an upsampling network, which simultaneously uses context information to estimate the occurrence probability of various feature points in the above-mentioned implicit tensor and outputs probability model parameters; the above-mentioned implicit tensor is quantized using the above-mentioned probability model parameters, and the quantized implicit tensor is losslessly encoded using an arithmetic encoder to obtain a first encoded bitstream, which is a partial bitstream output to the video decoding end.

[0282] In the second stage, the first encoded bitstream is input into the motion prediction model, and after being decoded by the arithmetic decoder corresponding to the arithmetic encoder in the encoding network, a quantized implicit tensor is obtained, which corresponds to the implicit tensor quantized in the first stage. After the implicit tensor enters the decoding network of the motion prediction network, multiple motion prediction information and multiple sets of weight information corresponding to the multiple motion prediction information are output. The multiple motion prediction information includes reconstructed optical flow information. And adaptive convolution kernel K. After decoding and obtaining various motion prediction information and corresponding weight information, the candidate prediction frames corresponding to various motion prediction information can be obtained to reconstruct the optical flow information. After multiplying the corresponding weight information, the first candidate prediction frame is obtained. After multiplying the adaptive convolution kernel with the corresponding weight information, the second candidate prediction frame is obtained. The reconstructed reference frame The third candidate prediction frame is obtained by multiplying itself with the corresponding weight information. The fourth candidate prediction frame is obtained by multiplying the image with the same size as the current frame and the element is 0 with the corresponding weight information. The above four candidate prediction frames are added to get the current frame x t The target prediction frame Current frame x t and target prediction frame The difference between them is the current frame x t The residual data R of the current frame x t Subtract the target prediction frame to get the current frame x t The residual data R is further processed by the residual coding network to obtain a second coded code stream; the residual coding network here can be the same as the coding network structure in the above-mentioned motion prediction model (the model parameters can be different), and the conversion process of the residual data R to the second coded code stream is similar to the encoding process of the above-mentioned first coded code stream. For details, please refer to the above description and will not be repeated here.

[0283] At the video decoding end, the current frame x t The video decoding includes decoding of multiple motion prediction information, obtaining a target prediction frame according to the multiple prediction information obtained by decoding Current frame x tThe decoding process of the residual data R is the same as the process of obtaining the multiple motion prediction information by decoding the first coded bitstream in the video encoding end. For details, please refer to the above description and no further description is given here. t The decoding part of the residual data R uses the residual data decoding network to decode the second encoded code stream to obtain the reconstructed residual data The network structure of the residual data decoding network can be the same as the decoding network in the motion prediction network (the model parameters can be different). And reconstruct the residual data Afterwards, the target prediction frame can be And reconstruct the residual data Add to reconstruct the current frame

[0284] The following are embodiments of the apparatus of the present disclosure, which can be used to execute embodiments of the method of the present disclosure.

[0285] According to a video processing device in one embodiment of the present disclosure, the device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. The video processing device includes:

[0286] A first acquisition module is configured to acquire a current frame in the video to be processed and a reconstructed reference frame of at least one reference frame;

[0287] A second acquisition module is configured to acquire multiple motion prediction information of the current frame and the reconstructed reference frame using a motion prediction model;

[0288] The first determination module is configured to determine a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information.

[0289] In an optional implementation of this embodiment, the first determining module includes:

[0290] A first determination submodule is configured to determine a plurality of candidate prediction frames corresponding to the current frame according to the plurality of motion prediction information;

[0291] The first acquisition submodule is configured to obtain the target prediction frame by using weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames.

[0292] In an optional implementation of the present embodiment, the weight information is obtained by the motion prediction model based on the current frame and the reconstructed reference frame; and / or the weight information corresponding to each type of motion prediction information is represented as a matrix of the same size as the current frame, and the sum of the elements in the same position in multiple groups of weight information corresponding to the multiple motion prediction information is equal to 1.

[0293] In an optional implementation of this embodiment, the multiple motion prediction information includes optical flow motion prediction information and an adaptive convolution kernel for representing the motion prediction information; and the first determination submodule includes:

[0294] A second acquisition submodule is configured to acquire a first candidate prediction frame corresponding to the optical flow motion prediction information according to the optical flow motion prediction information and the reconstructed reference frame;

[0295] The third acquisition submodule is configured to acquire a second candidate prediction frame corresponding to the adaptive convolution kernel according to the adaptive convolution kernel and the reconstructed reference frame.

[0296] In an optional implementation of this embodiment, the weight information further includes weight information corresponding to at least one preset prediction frame; the first acquisition submodule includes:

[0297] The first summing submodule is configured to obtain the target prediction frame by weighted summing the first candidate prediction frame, the second candidate prediction frame, the preset prediction frame and the corresponding weight information.

[0298] In an optional implementation of this embodiment, the preset prediction frame includes a reconstructed reference frame and / or a prediction frame whose elements are all zero.

[0299] In an optional implementation of this embodiment, the second acquisition module includes:

[0300] A first processing submodule is configured to process the current frame and the reconstructed reference frame using the motion prediction model to obtain a first encoding stream of the multiple motion prediction information;

[0301] The second processing submodule is configured to process the first encoded bitstream using the motion prediction model to obtain the multiple motion prediction information.

[0302] In an optional implementation of this embodiment, the device further includes:

[0303] A third acquisition module is configured to acquire residual data of the current frame using the current frame and the target prediction frame;

[0304] The first encoding module is configured to encode the residual data to obtain a second encoding code stream.

[0305] In an optional implementation of this embodiment, the reference frame includes multiple reference frames, and the second acquisition module includes:

[0306] A fifth acquisition submodule is configured to acquire, for each reference reconstructed frame of the reference frame, a plurality of motion prediction information of the current frame and each of the reconstructed reference frames using the motion prediction model;

[0307] The first determining module includes:

[0308] A second determination submodule is configured to determine a branch prediction frame corresponding to the current frame according to the reconstructed reference frame and multiple motion prediction information of each of the reconstructed reference frames;

[0309] The third determination submodule is configured to determine the target prediction frame according to the multiple branch prediction frames corresponding to the multiple reconstructed reference frames.

[0310] The video processing device in this embodiment Figure 1 The video processing methods in the illustrated embodiment and the related embodiments correspond to each other. For details, please refer to the above description. Figure 1 The description of the video processing method in the illustrated embodiment and related embodiments will not be repeated here.

[0311] According to a video processing device in one embodiment of the present disclosure, the device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. The video processing device includes:

[0312] A fourth acquisition module is configured to receive a first coded stream corresponding to the current frame and a reconstructed reference frame of at least one reference frame; the first coded stream includes a coded stream of motion prediction information between the current frame and the reconstructed reference frame of at least one reference frame;

[0313] A first processing module is configured to process the first coded bitstream using a motion prediction model to obtain a variety of motion prediction information between the current frame and the reconstructed reference frame;

[0314] The second determination module is configured to determine a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information.

[0315] In an optional implementation of this embodiment, the second determining module includes:

[0316] A fourth determination submodule is configured to determine a plurality of candidate prediction frames corresponding to the current frame according to the plurality of motion prediction information;

[0317] The sixth acquisition submodule is configured to obtain the target prediction frame by using the weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames.

[0318] In an optional implementation of the present embodiment, the weight information is obtained by the motion prediction model based on the current frame and the reconstructed reference frame; and / or the weight information corresponding to each type of motion prediction information is represented as a matrix of the same size as the current frame, and the sum of the elements in the same position in multiple groups of weight information corresponding to the multiple motion prediction information is equal to 1.

[0319] In an optional implementation of this embodiment, the multiple motion prediction information includes optical flow motion prediction information and an adaptive convolution kernel for representing the motion prediction information; the fourth determination submodule includes:

[0320] a seventh acquisition submodule, configured to acquire a first candidate prediction frame corresponding to the optical flow motion prediction information according to the optical flow motion prediction information and the reconstructed reference frame;

[0321] An eighth acquisition submodule is configured to acquire a second candidate prediction frame corresponding to the adaptive convolution kernel according to the adaptive convolution kernel and the reconstructed reference frame.

[0322] In an optional implementation of this embodiment, the weight information further includes weight information corresponding to at least one preset prediction frame; and the fourth determination submodule further includes:

[0323] The second summing submodule is configured to obtain the target prediction frame by weighted summing the first candidate prediction frame, the second candidate prediction frame and the preset prediction frame and the corresponding weight information.

[0324] In an optional implementation of this embodiment, the preset prediction frame includes a reconstructed reference frame and / or a prediction frame whose elements are all zero.

[0325] In an optional implementation of this embodiment, the device further includes:

[0326] A second processing module is configured to decode the second encoded bitstream to obtain residual data of the current frame;

[0327] The fifth acquisition module is configured to obtain the current frame according to the target prediction frame and the residual data.

[0328] In an optional implementation of this embodiment, the first encoding code stream includes multiple groups of encoding code streams corresponding to multiple reconstructed reference frames; the first processing module includes:

[0329] a ninth acquisition submodule, configured to acquire, for each group of the first encoded bitstreams, a plurality of motion prediction information of the current frame and each of the reconstructed reference frames using the motion prediction model;

[0330] The second determining module includes:

[0331] a fifth determination submodule, configured to determine a branch prediction frame corresponding to the current frame according to the reconstructed reference frame and multiple motion prediction information of each of the reconstructed reference frames;

[0332] The sixth determination submodule is configured to determine the target prediction frame according to the multiple branch candidate prediction frames corresponding to the multiple reconstructed reference frames.

[0333] The video processing device in this embodiment Figure 2 The video processing methods in the illustrated embodiment and the related embodiments correspond to each other. For details, please refer to the above description. Figure 2 The description of the video processing method in the illustrated embodiment and related embodiments will not be repeated here.

[0334] According to a video processing device in one embodiment of the present disclosure, the device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. The video processing device includes:

[0335] A sixth acquisition module is configured to acquire a video to be processed;

[0336] A first calling module is configured to call a preset service interface so that the preset service interface obtains a reconstructed reference frame of a current frame and at least one reference frame in a video to be processed, processes the current frame and the reconstructed reference frame using a motion prediction model to obtain multiple motion prediction information of the current frame, and determines a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information;

[0337] The first output module is configured to output the target prediction frame.

[0338] The video processing device in this embodiment Figure 3 The video processing methods in the embodiment shown and the related embodiments correspond to each other. For details, please refer to the above description. Figure 3 The description of the video processing method in the illustrated embodiment and related embodiments will not be repeated here.

[0339] According to a video processing device in one embodiment of the present disclosure, the device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. The video processing device includes:

[0340] A seventh acquisition module is configured to acquire a video to be processed;

[0341] A second calling module is configured to call a preset service interface so that the preset service interface obtains a reconstructed reference frame of a current frame and at least one reference frame in a to-be-processed video, obtains a plurality of motion prediction information between the current frame and the reconstructed reference frame and a first coded bitstream corresponding to the plurality of motion prediction information by using a motion prediction model, determines a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the plurality of motion prediction information, and obtains a second coded bitstream of residual data between the current frame and the target prediction frame according to the target prediction frame and the motion prediction model;

[0342] The second output module is configured to output the first encoded code stream and the second encoded code stream.

[0343] The video processing device in this embodiment Figure 4 The video processing methods in the illustrated embodiment and the related embodiments correspond to each other. For details, please refer to the above description. Figure 4 The description of the video processing method in the illustrated embodiment and related embodiments will not be repeated here.

[0344] According to a video processing device in one embodiment of the present disclosure, the device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. The video processing device includes:

[0345] A seventh acquisition module is configured to acquire a first encoding stream and a second encoding stream; the first encoding stream includes an encoding stream of multiple motion prediction information between a current frame and a reconstructed reference frame of at least one reference frame;

[0346] a third calling module configured to call a preset service interface so that the preset service interface processes the first coded bitstream using a motion prediction model to obtain a plurality of motion prediction information between the current frame and the reconstructed reference frame, and determines a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the plurality of motion prediction information;

[0347] The third output module is configured to output the target prediction frame.

[0348] The video processing device in this embodiment Figure 5 The video processing methods in the illustrated embodiment and the related embodiments correspond to each other. For details, please refer to the above description. Figure 5 The description of the video processing method in the illustrated embodiment and related embodiments will not be repeated here.

[0349] According to a video processing device in one embodiment of the present disclosure, the device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. The video processing device includes:

[0350] An eighth acquisition module is configured to acquire a first encoding stream and a second encoding stream; the first encoding stream includes an encoding stream of multiple motion prediction information between a current frame and a reconstructed reference frame of at least one reference frame; the second encoding stream includes an encoding stream of residual data between the current frame and a target prediction frame;

[0351] a fourth calling module, configured to call a preset service interface, so that the preset service interface processes the first coded stream using a motion prediction model to obtain a plurality of motion prediction information between the current frame and the reconstructed reference frame, and decodes the second coded stream to obtain residual data between the current frame and the target prediction frame, determines a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the plurality of motion prediction information, and obtains the current frame according to the target prediction frame and the residual data;

[0352] The fourth output module is configured to output the current frame.

[0353] The video processing device in this embodiment Figure 6 The video processing methods in the embodiment shown and the related embodiments correspond to each other. For details, please refer to the above description. Figure 6 The description of the video processing method in the illustrated embodiment and related embodiments will not be repeated here.

[0354] Figure 8 It is a structural schematic diagram of an electronic device suitable for implementing the video processing method according to an embodiment of the present disclosure.

[0355] like Figure 8 As shown, the electronic device 800 includes a processing unit 801, which can be implemented as a processing unit such as a CPU, a GPU, an FPGA, and an NPU. The processing unit 801 can perform various processes in the implementation of any of the above methods of the present disclosure according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 to a random access memory (RAM) 803. In RAM803, various programs and data required for the operation of the electronic device 800 are also stored. The processing unit 801, ROM802, and RAM803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0356] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.

[0357] In particular, according to an embodiment of the present disclosure, any method in the above referenced embodiment of the present disclosure may be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly contained on a readable medium thereof, and the computer program includes a program code for executing any method in the embodiment of the present disclosure. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 809, and / or installed from a removable medium 811.

[0358] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the road map or block diagram can represent a part of a module, program segment or code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a different order from the order marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0359] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The units or modules described may also be set in a processor, and the names of these units or modules do not constitute limitations on the units or modules themselves in some cases.

[0360] As another aspect, the present disclosure further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the device described in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into the device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the method described in the present disclosure.

[0361] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.

Claims

1. A video processing method, wherein: include: Obtaining a reconstructed reference frame of a current frame and at least one reference frame in a video to be processed; Acquire multiple motion prediction information of the current frame and the reconstructed reference frame using a motion prediction model; the multiple motion prediction information includes optical flow motion prediction information and an adaptive convolution kernel for representing the motion prediction information; the motion prediction model is a model based on deep learning; Determine a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information; Wherein, determining the target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information includes: Determine a plurality of candidate prediction frames corresponding to the current frame according to the plurality of motion prediction information; Obtaining the target prediction frame using weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames; The determining, according to the multiple motion prediction information, a plurality of candidate prediction frames corresponding to the current frame comprises: Acquire a first candidate prediction frame corresponding to the optical flow motion prediction information according to the optical flow motion prediction information and the reconstructed reference frame; A second candidate prediction frame corresponding to the adaptive convolution kernel is obtained according to the adaptive convolution kernel and the reconstructed reference frame.

2. The method according to claim 1, wherein: The weight information is obtained by the motion prediction model based on the current frame and the reconstructed reference frame; and / or, the weight information corresponding to each type of motion prediction information is represented as a matrix of the same size as the current frame, and the sum of the elements in the same position in multiple groups of weight information corresponding to the multiple motion prediction information is equal to 1.

3. The method according to claim 1, wherein: The weight information also includes weight information corresponding to at least one preset prediction frame; Obtaining the target prediction frame by using weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames includes: The target prediction frame is obtained by weighted summing the first candidate prediction frame, the second candidate prediction frame, the preset prediction frame and the corresponding weight information.

4. The method according to claim 3, wherein: The preset prediction frame includes a reconstructed reference frame and / or a prediction frame whose elements are all zero.

5. The method according to any one of claims 1 to 4, wherein: The motion prediction model is used to obtain a variety of motion prediction information of the current frame and the reconstructed reference frame, including: Processing the current frame and the reconstructed reference frame using the motion prediction model to obtain a first encoding stream of the multiple motion prediction information; The first encoded bitstream is processed using the motion prediction model to obtain the multiple motion prediction information.

6. The method according to any one of claims 1 to 4, wherein: The method further comprises: Acquire residual data of the current frame using the current frame and the target predicted frame; The residual data is encoded to obtain a second encoded code stream.

7. The method according to any one of claims 1 to 4, wherein: The reference frame includes a plurality of frames, and a motion prediction model is used to obtain a plurality of motion prediction information of the current frame and the reconstructed reference frame, including: For each reconstructed reference frame of each reference frame, using the motion prediction model to obtain a variety of motion prediction information of the current frame and each of the reconstructed reference frames; Determining a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information includes: Determine a branch prediction frame corresponding to the current frame according to the reconstructed reference frame and multiple motion prediction information of each of the reconstructed reference frames; The target prediction frame is determined according to the multiple branch prediction frames corresponding to the multiple reconstructed reference frames.

8. A video processing method, wherein: include: Obtaining a first encoded code stream corresponding to the current frame and a reconstructed reference frame of at least one reference frame; The first coded bitstream includes a coded bitstream of motion prediction information between the current frame and a reconstructed reference frame of at least one reference frame; Processing the first coded bitstream using a motion prediction model to obtain a variety of motion prediction information between the current frame and the reconstructed reference frame; the multiple motion prediction information includes optical flow motion prediction information and an adaptive convolution kernel for representing motion prediction information; the motion prediction model is a model based on deep learning; Determine a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information; Wherein, determining the target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information includes: Determine a plurality of candidate prediction frames corresponding to the current frame according to the plurality of motion prediction information; Obtaining the target prediction frame using weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames; The step of obtaining the target prediction frame by using the weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames includes: Acquire a first candidate prediction frame corresponding to the optical flow motion prediction information according to the optical flow motion prediction information and the reconstructed reference frame; A second candidate prediction frame corresponding to the adaptive convolution kernel is obtained according to the adaptive convolution kernel and the reconstructed reference frame.

9. The method according to claim 8, wherein: The weight information is obtained by the motion prediction model based on the current frame and the reconstructed reference frame; and / or, the weight information corresponding to each type of motion prediction information is represented as a matrix of the same size as the current frame, and the sum of the elements in the same position in multiple groups of weight information corresponding to the multiple motion prediction information is equal to 1.

10. The method according to claim 8, wherein: The weight information further includes weight information corresponding to at least one preset prediction frame; determining multiple candidate prediction frames corresponding to the current frame according to the multiple motion prediction information also includes: The target prediction frame is obtained by weighted summing the first candidate prediction frame, the second candidate prediction frame, the preset prediction frame and the corresponding weight information.

11. The method according to claim 10, wherein: The preset prediction frame includes a reconstructed reference frame and / or a prediction frame whose elements are all zero.

12. The method according to any one of claims 8 to 11, wherein: The method further comprises: Decoding the second encoded code stream to obtain residual data of the current frame; The current frame is obtained according to the target prediction frame and the residual data.

13. The method according to any one of claims 8 to 11, wherein: The first coded bitstream includes a plurality of groups of coded bitstreams corresponding to the plurality of reconstructed reference frames; the first coded bitstream is processed by using a motion prediction model to obtain a plurality of motion prediction information between the current frame and the reconstructed reference frame, including: For each group of the first encoded bitstreams, using the motion prediction model to obtain a plurality of motion prediction information of the current frame and each of the reconstructed reference frames; Determining a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information includes: Determine a branch prediction frame corresponding to the current frame according to the reconstructed reference frame and multiple motion prediction information of each of the reconstructed reference frames; The target prediction frame is determined according to the multiple branch prediction frames corresponding to the multiple reconstructed reference frames.

14. A video processing method, wherein: include: Get the video to be processed; Calling a preset service interface so that the preset service interface obtains a reconstructed reference frame of a current frame and at least one reference frame in a video to be processed, processes the current frame and the reconstructed reference frame using a motion prediction model to obtain multiple motion prediction information of the current frame, and determines a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information; the multiple motion prediction information includes optical flow motion prediction information and an adaptive convolution kernel for representing motion prediction information; the motion prediction model is a model based on deep learning; Outputting the target prediction frame; Wherein, determining the target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information includes: Determine a plurality of candidate prediction frames corresponding to the current frame according to the plurality of motion prediction information; Obtaining the target prediction frame using weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames; The determining, according to the multiple motion prediction information, a plurality of candidate prediction frames corresponding to the current frame comprises: Acquire a first candidate prediction frame corresponding to the optical flow motion prediction information according to the optical flow motion prediction information and the reconstructed reference frame; A second candidate prediction frame corresponding to the adaptive convolution kernel is obtained according to the adaptive convolution kernel and the reconstructed reference frame.

15. A video processing method, wherein: include: Get the video to be processed; Calling a preset service interface so that the preset service interface obtains a reconstructed reference frame of a current frame and at least one reference frame in a video to be processed, uses a motion prediction model to obtain a plurality of motion prediction information between the current frame and the reconstructed reference frame and a first coded code stream corresponding to the plurality of motion prediction information, determines a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the plurality of motion prediction information, and obtains a second coded code stream of residual data between the current frame and the target prediction frame according to the target prediction frame and the motion prediction model; the plurality of motion prediction information includes optical flow motion prediction information and an adaptive convolution kernel for representing motion prediction information; and the motion prediction model is a model based on deep learning; Outputting the first encoded code stream and the second encoded code stream; Wherein, determining the target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information includes: Determine a plurality of candidate prediction frames corresponding to the current frame according to the plurality of motion prediction information; Obtaining the target prediction frame using weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames; The determining, according to the multiple motion prediction information, a plurality of candidate prediction frames corresponding to the current frame comprises: Acquire a first candidate prediction frame corresponding to the optical flow motion prediction information according to the optical flow motion prediction information and the reconstructed reference frame; A second candidate prediction frame corresponding to the adaptive convolution kernel is obtained according to the adaptive convolution kernel and the reconstructed reference frame.

16. A video processing method, wherein: include: Obtaining a first encoding stream and a second encoding stream; The first coded bitstream includes a coded bitstream of multiple motion prediction information between the current frame and a reconstructed reference frame of at least one reference frame; Calling a preset service interface so that the preset service interface processes the first coded bitstream using a motion prediction model to obtain a variety of motion prediction information between the current frame and the reconstructed reference frame, and determining a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information; the multiple motion prediction information includes optical flow motion prediction information and an adaptive convolution kernel for representing motion prediction information; the motion prediction model is a model based on deep learning; Outputting the target prediction frame; Wherein, determining the target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information includes: Determine a plurality of candidate prediction frames corresponding to the current frame according to the plurality of motion prediction information; Obtaining the target prediction frame using weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames; The determining, according to the multiple motion prediction information, a plurality of candidate prediction frames corresponding to the current frame comprises: Acquire a first candidate prediction frame corresponding to the optical flow motion prediction information according to the optical flow motion prediction information and the reconstructed reference frame; A second candidate prediction frame corresponding to the adaptive convolution kernel is obtained according to the adaptive convolution kernel and the reconstructed reference frame.

17. A video processing method, wherein: include: Acquire a first coded bitstream and a second coded bitstream; the first coded bitstream includes a coded bitstream of multiple motion prediction information between a current frame and a reconstructed reference frame of at least one reference frame; the second coded bitstream includes a coded bitstream of residual data between the current frame and a target prediction frame; Calling a preset service interface so that the preset service interface processes the first coded bitstream using a motion prediction model to obtain a variety of motion prediction information between the current frame and the reconstructed reference frame, and decodes the second coded bitstream to obtain residual data between the current frame and the target prediction frame, determines a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information, and obtains the current frame according to the target prediction frame and the residual data; the multiple motion prediction information includes optical flow motion prediction information and an adaptive convolution kernel for representing motion prediction information; and the motion prediction model is a model based on deep learning; Outputting the current frame; Wherein, determining the target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information includes: Determine a plurality of candidate prediction frames corresponding to the current frame according to the plurality of motion prediction information; Obtaining the target prediction frame using weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames; The determining, according to the multiple motion prediction information, a plurality of candidate prediction frames corresponding to the current frame comprises: Acquire a first candidate prediction frame corresponding to the optical flow motion prediction information according to the optical flow motion prediction information and the reconstructed reference frame; A second candidate prediction frame corresponding to the adaptive convolution kernel is obtained according to the adaptive convolution kernel and the reconstructed reference frame.

18. A video processing method, wherein: include: Obtaining a reconstructed reference frame of a current frame and at least one reference frame in a video to be encoded; Using a motion prediction model to obtain a plurality of motion prediction information between the current frame and the reconstructed reference frame and a first coded bitstream corresponding to the plurality of motion prediction information; the plurality of motion prediction information includes optical flow motion prediction information and an adaptive convolution kernel for representing the motion prediction information; the motion prediction model is a model based on deep learning; Determine a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information; Obtaining a second coded stream of residual data between the current frame and the target predicted frame according to the target predicted frame and the motion prediction model; Outputting the first encoded code stream and the second encoded code stream; Wherein, determining the target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information includes: Determine a plurality of candidate prediction frames corresponding to the current frame according to the plurality of motion prediction information; Obtaining the target prediction frame using weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames; The determining, according to the multiple motion prediction information, a plurality of candidate prediction frames corresponding to the current frame comprises: Acquire a first candidate prediction frame corresponding to the optical flow motion prediction information according to the optical flow motion prediction information and the reconstructed reference frame; A second candidate prediction frame corresponding to the adaptive convolution kernel is obtained according to the adaptive convolution kernel and the reconstructed reference frame.

19. A video processing method, wherein: include: Acquire a first coded bitstream and a second coded bitstream; the first coded bitstream includes a coded bitstream of multiple motion prediction information between a current frame and a reconstructed reference frame of at least one reference frame; the second coded bitstream includes a coded bitstream of residual data between the current frame and a target prediction frame; The first coded bitstream is processed by using a motion prediction model to obtain a plurality of motion prediction information between the current frame and the reconstructed reference frame, and the second coded bitstream is decoded to obtain residual data between the current frame and the target prediction frame; the plurality of motion prediction information includes optical flow motion prediction information and an adaptive convolution kernel for representing motion prediction information; the motion prediction model is a model based on deep learning; Determine a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information; Obtaining the current frame according to the target prediction frame and the residual data; Outputting the current frame; Wherein, determining the target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information includes: Determine a plurality of candidate prediction frames corresponding to the current frame according to the plurality of motion prediction information; Obtaining the target prediction frame using weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames; The determining, according to the multiple motion prediction information, a plurality of candidate prediction frames corresponding to the current frame comprises: Acquire a first candidate prediction frame corresponding to the optical flow motion prediction information according to the optical flow motion prediction information and the reconstructed reference frame; A second candidate prediction frame corresponding to the adaptive convolution kernel is obtained according to the adaptive convolution kernel and the reconstructed reference frame.

20. A video processing device, wherein: include: A first acquisition module is configured to acquire a current frame in the video to be processed and a reconstructed reference frame of at least one reference frame; A second acquisition module is configured to acquire multiple motion prediction information of the current frame and the reconstructed reference frame using a motion prediction model; the multiple motion prediction information includes optical flow motion prediction information and an adaptive convolution kernel for representing motion prediction information; the motion prediction model is a model based on deep learning; A first determination module is configured to determine a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information; Wherein, the first determining module includes: A first determination submodule is configured to determine a plurality of candidate prediction frames corresponding to the current frame according to the plurality of motion prediction information; A first acquisition submodule is configured to obtain the target prediction frame by using weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames; The first determining submodule includes: A second acquisition submodule is configured to acquire a first candidate prediction frame corresponding to the optical flow motion prediction information according to the optical flow motion prediction information and the reconstructed reference frame; The third acquisition submodule is configured to acquire a second candidate prediction frame corresponding to the adaptive convolution kernel according to the adaptive convolution kernel and the reconstructed reference frame.

21. A video processing device, wherein: include: A fourth acquisition module is configured to receive a first coded code stream corresponding to the current frame and a reconstructed reference frame of at least one reference frame; The first coded bitstream includes a coded bitstream of motion prediction information between the current frame and a reconstructed reference frame of at least one reference frame; A first processing module is configured to process the first coded bitstream using a motion prediction model to obtain a variety of motion prediction information between the current frame and the reconstructed reference frame; the multiple motion prediction information includes optical flow motion prediction information and an adaptive convolution kernel for representing motion prediction information; the motion prediction model is a model based on deep learning; A second determination module is configured to determine a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information; Wherein, the second determining module includes: A fourth determination submodule is configured to determine a plurality of candidate prediction frames corresponding to the current frame according to the plurality of motion prediction information; A sixth acquisition submodule is configured to obtain the target prediction frame by using the weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames; The fourth determining submodule includes: a seventh acquisition submodule, configured to acquire a first candidate prediction frame corresponding to the optical flow motion prediction information according to the optical flow motion prediction information and the reconstructed reference frame; An eighth acquisition submodule is configured to acquire a second candidate prediction frame corresponding to the adaptive convolution kernel according to the adaptive convolution kernel and the reconstructed reference frame.

22. A video processing device, wherein: include: A sixth acquisition module is configured to acquire a video to be processed; A first calling module is configured to call a preset service interface so that the preset service interface obtains a reconstructed reference frame of a current frame and at least one reference frame in a video to be processed, processes the current frame and the reconstructed reference frame using a motion prediction model to obtain a plurality of motion prediction information of the current frame, and determines a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the plurality of motion prediction information; the plurality of motion prediction information includes optical flow motion prediction information and an adaptive convolution kernel for representing motion prediction information; the motion prediction model is a model based on deep learning; A first output module, configured to output the target prediction frame; The part of the first calling module that determines the target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information is configured as follows: Determine a plurality of candidate prediction frames corresponding to the current frame according to the plurality of motion prediction information; Obtaining the target prediction frame using weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames; The part of determining the multiple candidate prediction frames corresponding to the current frame according to the multiple motion prediction information is configured as follows: Acquire a first candidate prediction frame corresponding to the optical flow motion prediction information according to the optical flow motion prediction information and the reconstructed reference frame; A second candidate prediction frame corresponding to the adaptive convolution kernel is obtained according to the adaptive convolution kernel and the reconstructed reference frame.

23. A video processing device, wherein: include: A seventh acquisition module is configured to acquire a video to be processed; The second calling module is configured to call a preset service interface so that the preset service interface obtains a reconstructed reference frame of a current frame and at least one reference frame in a video to be processed, obtains a plurality of motion prediction information between the current frame and the reconstructed reference frame and a first coded code stream corresponding to the plurality of motion prediction information by using a motion prediction model, determines a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the plurality of motion prediction information, and obtains a second coded code stream of residual data between the current frame and the target prediction frame according to the target prediction frame and the motion prediction model; the plurality of motion prediction information includes optical flow motion prediction information and an adaptive convolution kernel for representing motion prediction information; and the motion prediction model is a model based on deep learning; A second output module is configured to output the first encoded code stream and the second encoded code stream; The part of the second calling module that determines the target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information is configured as follows: Determine a plurality of candidate prediction frames corresponding to the current frame according to the plurality of motion prediction information; Obtaining the target prediction frame using weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames; The part of determining the multiple candidate prediction frames corresponding to the current frame according to the multiple motion prediction information is configured as follows: Acquire a first candidate prediction frame corresponding to the optical flow motion prediction information according to the optical flow motion prediction information and the reconstructed reference frame; A second candidate prediction frame corresponding to the adaptive convolution kernel is obtained according to the adaptive convolution kernel and the reconstructed reference frame.

24. A video processing device, wherein: include: A seventh acquisition module is configured to acquire the first encoded code stream and the second encoded code stream; The first coded bitstream includes a coded bitstream of multiple motion prediction information between the current frame and a reconstructed reference frame of at least one reference frame; The third calling module is configured to call a preset service interface so that the preset service interface processes the first coded bitstream using a motion prediction model to obtain a variety of motion prediction information between the current frame and the reconstructed reference frame, and determines a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information; the multiple motion prediction information includes optical flow motion prediction information and an adaptive convolution kernel for representing motion prediction information; the motion prediction model is a model based on deep learning; A third output module is configured to output the target prediction frame; The part of the third calling module that determines the target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information is configured as follows: Determine a plurality of candidate prediction frames corresponding to the current frame according to the plurality of motion prediction information; Obtaining the target prediction frame using weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames; The part of determining the multiple candidate prediction frames corresponding to the current frame according to the multiple motion prediction information is configured as follows: Acquire a first candidate prediction frame corresponding to the optical flow motion prediction information according to the optical flow motion prediction information and the reconstructed reference frame; A second candidate prediction frame corresponding to the adaptive convolution kernel is obtained according to the adaptive convolution kernel and the reconstructed reference frame.

25. A video processing device, wherein: include: An eighth acquisition module is configured to acquire the first encoded code stream and the second encoded code stream; The first coded bitstream includes a coded bitstream of multiple motion prediction information between the current frame and a reconstructed reference frame of at least one reference frame; the second coded bitstream includes a coded bitstream of residual data between the current frame and a target prediction frame; a fourth calling module configured to call a preset service interface so that the preset service interface processes the first coded stream using a motion prediction model to obtain a plurality of motion prediction information between the current frame and the reconstructed reference frame, and decodes the second coded stream to obtain residual data between the current frame and the target prediction frame, determines a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the plurality of motion prediction information, and obtains the current frame according to the target prediction frame and the residual data; the plurality of motion prediction information includes optical flow motion prediction information and an adaptive convolution kernel for representing motion prediction information; and the motion prediction model is a model based on deep learning; A fourth output module, configured to output the current frame; The part of the fourth calling module that determines the target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information is configured as follows: Determine a plurality of candidate prediction frames corresponding to the current frame according to the plurality of motion prediction information; Obtaining the target prediction frame using weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames; The part of determining the multiple candidate prediction frames corresponding to the current frame according to the multiple motion prediction information is configured as follows: Acquire a first candidate prediction frame corresponding to the optical flow motion prediction information according to the optical flow motion prediction information and the reconstructed reference frame; A second candidate prediction frame corresponding to the adaptive convolution kernel is obtained according to the adaptive convolution kernel and the reconstructed reference frame.

26. A video processing device, wherein: include: A ninth acquisition module, configured to acquire a reconstructed reference frame of a current frame and at least one reference frame in the video to be encoded; a tenth acquisition module, configured to acquire, by using a motion prediction model, a plurality of motion prediction information between the current frame and the reconstructed reference frame and a first coded bitstream corresponding to the plurality of motion prediction information; the plurality of motion prediction information comprises optical flow motion prediction information and an adaptive convolution kernel for representing the motion prediction information; the motion prediction model is a model based on deep learning; A third determination module is configured to determine a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information; an eleventh acquisition module, configured to obtain a second encoded bit stream of residual data between the current frame and the target prediction frame according to the target prediction frame and the motion prediction model; a fifth output module, configured to output the first encoded code stream and the second encoded code stream; The part of the third determination module that determines the target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information is configured as follows: Determine a plurality of candidate prediction frames corresponding to the current frame according to the plurality of motion prediction information; Obtaining the target prediction frame using weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames; The part of determining the multiple candidate prediction frames corresponding to the current frame according to the multiple motion prediction information is configured as follows: Acquire a first candidate prediction frame corresponding to the optical flow motion prediction information according to the optical flow motion prediction information and the reconstructed reference frame; A second candidate prediction frame corresponding to the adaptive convolution kernel is obtained according to the adaptive convolution kernel and the reconstructed reference frame.

27. A video processing device, wherein: include: A twelfth acquisition module is configured to acquire the first encoded code stream and the second encoded code stream; The first coded bitstream includes a coded bitstream of multiple motion prediction information between the current frame and a reconstructed reference frame of at least one reference frame; the second coded bitstream includes a coded bitstream of residual data between the current frame and a target prediction frame; A thirteenth acquisition module is configured to process the first coded bitstream using a motion prediction model to obtain a variety of motion prediction information between the current frame and the reconstructed reference frame, and decode the second coded bitstream to obtain residual data between the current frame and the target prediction frame; the multiple motion prediction information includes optical flow motion prediction information and an adaptive convolution kernel for representing motion prediction information; the motion prediction model is a model based on deep learning; A fourth determination module is configured to determine a target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information; a fifth determination module, configured to obtain the current frame according to the target prediction frame and the residual data; A sixth output module, configured to output the current frame; The part of the fourth determination module that determines the target prediction frame corresponding to the current frame according to the reconstructed reference frame and the multiple motion prediction information is configured as follows: Determine a plurality of candidate prediction frames corresponding to the current frame according to the plurality of motion prediction information; Obtaining the target prediction frame using weight information corresponding to the multiple motion prediction information and the multiple candidate prediction frames; The part of determining the multiple candidate prediction frames corresponding to the current frame according to the multiple motion prediction information is configured as follows: Acquire a first candidate prediction frame corresponding to the optical flow motion prediction information according to the optical flow motion prediction information and the reconstructed reference frame; A second candidate prediction frame corresponding to the adaptive convolution kernel is obtained according to the adaptive convolution kernel and the reconstructed reference frame.

28. An electronic device, wherein: comprising a memory and a processor; wherein, The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to any one of claims 1 to 19.

29. A computer-readable storage medium having computer instructions stored thereon, wherein: When the computer instructions are executed by a processor, the method according to any one of claims 1 to 19 is implemented.