Adaptive time domain variable resolution coding method and apparatus, terminal and storage medium

Abstract

The application discloses an adaptive time domain variable resolution coding and decoding method, device, terminal and storage medium, and the method comprises the following steps: dividing an input video into a plurality of equal-length subsequences; acquiring scene motion features of the subsequences, and determining a plurality of target subsequences according to the scene motion features; performing downsampling on the target subsequences to obtain a plurality of frame rate reduction sequences; performing coding and decoding on the frame rate reduction sequences to obtain a plurality of decoded sequences; and performing upsampling on the decoded sequences to obtain the coding and decoding sequences of the target subsequences. The application screens out target subsequences through scene motion features, performs coding and decoding on the target subsequences after downsampling, and then performs upsampling on the frame rate reduction decoded sequences at a decoding end, so that the frame rate of specific regions in the time domain is adaptively adjusted, and the coding rate is saved. The application solves the problem that, in the prior art, because the scene motion of some time domain regions in a video sequence is gentle, coding and decoding are performed by using a fixed frame rate, and the coding rate is wasted.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video coding, in particular to an adaptive temporal variable resolution coding method and device, a terminal and a storage medium. BACKGROUND

[0002] The existing AVS3 coding framework only receives a fixed frame rate video as an input sequence. For some sequences in certain temporal regions, the motion of the scene can be relatively flat, so the temporal redundancy of this period is very large, and using a fixed frame rate to code it will waste a lot of coding rate.

[0003] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0004] The technical problem to be solved by the present application is to provide an adaptive temporal variable resolution coding method and device, a terminal and a storage medium to solve the problem that the existing technology uses a fixed frame rate to code a video sequence, which wastes a lot of coding rate, due to the flat motion of the scene in certain temporal regions of the video sequence.

[0005] The technical scheme adopted by the present application to solve the problem is as follows:

[0006] In a first aspect, the present application provides an adaptive temporal variable resolution coding method, wherein the method comprises:

[0007] obtaining an input video and dividing the input video into a plurality of equal-length sub-sequences;

[0008] obtaining scene motion features corresponding to each of the sub-sequences, and determining a plurality of target sub-sequences from each of the sub-sequences according to the scene motion features;

[0009] performing a downsampling operation on each of the target sub-sequences in the time domain to obtain a plurality of downsampled sequences corresponding to each of the target sub-sequences;

[0010] performing a coding operation on each of the downsampled sequences to obtain a plurality of decoded sequences corresponding to each of the target sub-sequences;

[0011] performing an upsampling operation on each of the decoded sequences in the time domain to obtain a plurality of coded sequences corresponding to each of the target sub-sequences.

[0012] In one embodiment, the length of each of the sub-sequences is equal to the length of a time domain basic unit, and the length of the time domain basic unit is 2s.

[0013] In one embodiment, the downsampling operation is a frame rate doubling downsampling operation.

[0014] In an embodiment, the acquiring the scene motion feature corresponding to each of the sub-sequences comprises:

[0015] performing the downsampling operation on each of the sub-sequences to obtain a reduced-frame-rate sequence corresponding to the sub-sequence;

[0016] performing frame rate recovery on the reduced-frame-rate sequence of the sub-sequence by frame duplication to obtain a recovered sequence of the sub-sequence;

[0017] acquiring a similarity between the sub-sequence and the recovered sequence of the sub-sequence, and a preset similarity threshold value, and determining that the scene motion feature is smooth motion when the similarity value is higher than the similarity threshold value;

[0018] taking the sub-sequence with the scene motion feature being smooth motion as the target sub-sequence.

[0019] In an embodiment, the upsampling operation comprises generating an interpolated frame between two adjacent video frames of each of the decoded sequences, and the method for generating the interpolated frame comprises:

[0020] acquiring spatiotemporal domain information corresponding to a first video frame and a second video frame, respectively;

[0021] performing forward and reverse prediction operations according to two of the spatiotemporal domain information to obtain two interpolated reference frames corresponding to the forward and reverse prediction operations, respectively, wherein the forward prediction starts at the first video frame and ends at the second video frame, and the reverse prediction starts at the second video frame and ends at the first video frame;

[0022] fusing the two interpolated reference frames, the first video frame, and the second video frame to obtain the interpolated frame corresponding to the first video frame and the second video frame.

[0023] In an embodiment, the spatiotemporal domain information comprises spatial domain information and temporal domain features, and the prediction operation comprises:

[0024] determining attention weights corresponding to the prediction start point and the prediction end point according to the spatial domain information corresponding to the prediction start point and the prediction end point, respectively;

[0025] determining fused temporal domain features according to the temporal domain features corresponding to the prediction start point and the prediction end point and the attention weights;

[0026] determining enhanced temporal domain features according to the temporal domain features of the prediction end point and the fused temporal domain features;

[0027] determine the interpolation reference frame according to the spatial information of the prediction start point and the prediction end point and the enhanced temporal feature of the prediction end point.

[0028] In an embodiment, the determining the attention weight corresponding to the prediction start point and the prediction end point respectively according to the spatial information corresponding to the prediction start point and the prediction end point respectively comprises:

[0029] obtaining the spatial feature of the prediction end point according to the spatial information of the prediction end point;

[0030] performing matrix element-wise multiplication operation according to the spatial information of the prediction start point and the spatial feature to obtain the attention score of the prediction start point;

[0031] performing matrix element-wise multiplication operation according to the spatial information of the prediction end point and the spatial feature to obtain the attention score of the prediction end point;

[0032] determining the attention weight corresponding to the prediction start point and the prediction end point respectively according to the attention score of the prediction start point and the prediction end point.

[0033] In an embodiment, the determining the fusion temporal feature according to the temporal feature corresponding to the prediction start point and the prediction end point respectively and the attention weight comprises:

[0034] performing weighted summation according to the temporal feature corresponding to the prediction start point and the prediction end point respectively and the attention weight to obtain the fusion temporal feature.

[0035] In an embodiment, the determining the enhanced temporal feature according to the temporal feature of the prediction end point and the fusion temporal feature comprises:

[0036] performing first convolution operation on the temporal feature of the prediction end point, and determining a fusion gate according to the result of the first convolution operation, wherein the fusion gate is a Sigmoid activation function, and the numerical value is distributed between 0 and 1, 1 represents that the gate is open, and 0 represents that the gate is closed;

[0037] performing matrix element-wise multiplication operation according to the fusion gate and the temporal feature of the prediction end point;

[0038] performing matrix element-wise multiplication operation according to the difference between 1 and the fusion gate and the fusion temporal feature;

[0039] determining the enhanced temporal feature according to the sum of the results of the two matrix element-wise multiplication operations.

[0040] In an embodiment, the determining the interpolation reference frame according to the spatial information of the predicted end point and the enhanced temporal feature comprises:

[0041] performing a second convolution operation on the spatial information of the predicted end point, and determining an update gate according to a result of the second convolution operation, wherein the update gate is a Sigmoid activation function, and a numerical distribution of the update gate is between 0 and 1, 1 represents that the gate is open, and 0 represents that the gate is closed;

[0042] performing a third convolution operation on the spatial information of the predicted end point, and performing a matrix element-wise multiplication operation according to a result of the third convolution operation and the update gate;

[0043] performing a fourth convolution operation on the enhanced temporal feature, and performing a matrix element-wise multiplication operation according to a difference between 1 and the fusion gate and the fourth convolution operation, wherein the first convolution operation, the second convolution operation, the third convolution operation and the fourth convolution operation correspond to different network parameters respectively;

[0044] determining the interpolation reference frame according to a sum of results of the two matrix element-wise multiplication operations.

[0045] In a second aspect, an embodiment of the present application further provides an adaptive temporal variational resolution coding and decoding device, wherein the device comprises:

[0046] a splitting module configured to obtain an input video and split the input video into a plurality of equal-length subsequences;

[0047] a decision module configured to obtain scene motion features corresponding to the subsequences respectively, and determine a plurality of target subsequences from the subsequences according to the scene motion features;

[0048] a downsampling module configured to perform a downsampling operation on the target subsequences in the time domain to obtain a plurality of frame rate reduction sequences corresponding to the target subsequences respectively;

[0049] a coding and decoding module configured to perform a coding and decoding operation on the frame rate reduction sequences of the target subsequences to obtain a plurality of decoded sequences corresponding to the target subsequences respectively;

[0050] an upsampling module configured to perform an upsampling operation on the decoded sequences in the time domain to obtain a plurality of coding and decoding sequences corresponding to the target subsequences respectively.

[0051] In a third aspect, an embodiment of the present application further provides a terminal, wherein the terminal comprises a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing the adaptive temporal variational resolution coding and decoding method described in any of the above aspects; and the processor is configured to execute the programs.

[0052] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, which stores a plurality of instructions, wherein the instructions are adapted to be loaded and executed by a processor to implement the steps of the adaptive temporal variable resolution coding method.

[0053] The present application has the following beneficial effects: the embodiments of the present application divide an input video into a plurality of equal-length subsequences, acquire scene motion features of the subsequences, determine a plurality of target subsequences according to the scene motion features, down-sample the target subsequences to obtain a plurality of frame rate reduced sequences, encode and decode the frame rate reduced sequences to obtain a plurality of decoded sequences, and up-sample the decoded sequences to obtain encoded and decoded sequences of the target subsequences. The embodiments of the present application filter out the target subsequences according to the scene motion features, down-sample the target subsequences, then encode and decode the target subsequences, and finally up-sample the frame rate reduced decoded sequences at the decoding end, thereby adaptively adjusting the frame rate of specific regions in the time domain and saving the encoding code rate. The embodiments of the present application solve the problem that the existing technology wastes the encoding code rate because the scene motion of some time domain regions in a video sequence is flat and the fixed frame rate is used for encoding and decoding. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0055] Figure 1 FIG. 1 is a basic flow diagram of the adaptive temporal variable resolution coding method provided by the embodiments of the present application.

[0056] Figure 2 FIG. 2 is a framework diagram of the adaptive temporal variable resolution coding method provided by the embodiments of the present application.

[0057] Figure 3 FIG. 3 is a flow diagram of the time domain up-sampling method provided by the embodiments of the present application.

[0058] Figure 4 FIG. 4 is a working principle diagram of the motion perception unit provided by the embodiments of the present application.

[0059] Figure 5 FIG. 5 is a module diagram of the adaptive temporal variable resolution coding device provided by the embodiments of the present application.

[0060] Figure 6 FIG. 6 is a principle block diagram of the terminal provided by the embodiments of the present application. DETAILED DESCRIPTION

[0061] The application discloses an adaptive time domain variable resolution coding and decoding method and device, a terminal and a storage medium. In order to make the purpose, technical scheme and effect of the application more clear and explicit, the application is further described in detail below with reference to the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and not to limit the application.

[0062] Those skilled in the art can understand that the singular forms "a", "an" and "the" used herein include plural forms unless specifically stated otherwise. It should be further understood that the use of the term "include" in the specification of the application means that the stated features, integers, steps, operations, elements, and / or components are present, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be intermediate elements. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any single unit and all combinations of the associated listed items.

[0063] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as that generally understood by those skilled in the art to which the application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have meanings consistent with those in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as such.

[0064] In view of the above defects of the prior art, the present application provides a self-adaptive time domain variable resolution coding and decoding method, which comprises the following steps: acquiring an input video and dividing the input video into a plurality of equal-length subsequences; acquiring scene motion features corresponding to each of the subsequences, determining a plurality of target subsequences from each of the subsequences according to the scene motion features; performing downsampling operation on each of the target subsequences in the time domain to obtain a plurality of frame rate reduction sequences corresponding to each of the target subsequences; performing coding and decoding operation on each of the frame rate reduction sequences of the target subsequences to obtain a plurality of decoded sequences corresponding to each of the target subsequences; and performing upsampling operation on each of the decoded sequences in the time domain to obtain a plurality of coded sequences corresponding to each of the target subsequences. The present application screens target subsequences through the scene motion features of each subsequence, performs coding and decoding operation on the target subsequences after time domain downsampling, and then performs upsampling on the frame rate reduction decoded sequences at the decoding end, so that the frame rate of specific regions in the time domain is adaptively adjusted, and the coding rate is saved. The problem that a large amount of coding rate is wasted due to the flat scene motion of some time domain regions in the video sequence in the prior art is solved.

[0065] As shown in Figure 1 , the method comprises:

[0066] Step S100, acquiring an input video and dividing the input video into a plurality of equal-length subsequences;

[0067] Step S200, acquiring scene motion features corresponding to each of the subsequences, and determining a plurality of target subsequences from each of the subsequences according to the scene motion features;

[0068] Step S300, performing downsampling operation on each of the target subsequences in the time domain to obtain a plurality of frame rate reduction sequences corresponding to each of the target subsequences;

[0069] Step S400, performing coding and decoding operation on each of the frame rate reduction sequences of the target subsequences to obtain a plurality of decoded sequences corresponding to each of the target subsequences;

[0070] Step S500, performing upsampling operation on each of the decoded sequences in the time domain to obtain a plurality of coded sequences corresponding to each of the target subsequences.

[0071] Specifically, the present embodiment divides the input video sequence into a plurality of equal-length subsequences. As shown in Figure 2As shown, for each sub-sequence, the encoding end first detects the scene motion feature of the sub-sequence, and decides whether the sub-sequence is directly encoded or first down-sampled and then encoded based on the scene motion feature. The sub-sequence that needs to be down-sampled in the embodiment is defined as a target sub-sequence. After the target sub-sequence is down-sampled, the frame rate thereof is reduced to obtain a reduced frame rate sequence. Then, the reduced frame rate sequence is encoded to obtain an encoded sequence. The encoded sequence is decoded to obtain a decoded sequence. Then, the temporal up-sampling is activated at the decoding end to restore the frame rate of the decoded sequence. For the sub-sequence that does not need to be down-sampled, the traditional encoding and decoding mode is used for processing. The present application distinguishes the part with slow scene motion in the video through the scene motion feature, and then uses the mode of first reducing the frame rate and then encoding and decoding to save the encoded bitstream.

[0072] In an implementation manner, the length of each sub-sequence is equal to the length of a time domain basic unit, and the length of the time domain basic unit is 2s.

[0073] Specifically, the present embodiment defines 2s as a time domain basic unit, divides the input video into multiple 2s sub-sequences, and each sub-sequence can be regarded as a basic unit of encoding and decoding.

[0074] In an implementation manner, the down-sampling operation is frame rate two times down-sampling.

[0075] Specifically, the down-sampling operation in the present embodiment is frame rate two times down-sampling in the time domain of the target sub-sequence, so that the frame rate thereof is halved.

[0076] In an implementation manner, the step S200 specifically includes:

[0077] The step S201 includes: performing the down-sampling operation on each sub-sequence to obtain a reduced frame rate sequence corresponding to the sub-sequence.

[0078] The step S202 includes: performing frame rate restoration on the reduced frame rate sequence of the sub-sequence through a frame duplication operation to obtain a restored sequence of the sub-sequence.

[0079] The step S203 includes: obtaining the similarity between the sub-sequence and the restored sequence of the sub-sequence and a preset similarity threshold value, and judging that the scene motion feature is slow motion when the similarity value is higher than the similarity threshold value.

[0080] The step S204 includes: taking the sub-sequence with the slow motion scene motion feature as the target sub-sequence.

[0081] Specifically, for each subsequence, after performing a downsampling operation, frame rate recovery is achieved through frame duplication, where the sampled-out frames are directly copied from the previous frame. The recovered sequence is then compared with the original subsequence to obtain their similarity, for example, using the VMAF (Video Multimethod Assessment Fusion) metric. The calculated similarity is then compared with a preset similarity threshold, for example, 65. If the similarity is higher than the threshold, it indicates that the scene motion in the subsequence is smooth, and downsampling before encoding / decoding can save on the encoded bitstream; if the similarity is lower than or equal to the threshold, the scene motion in the subsequence is normal, and downsampling is unnecessary.

[0082] In one implementation, the upsampling operation involves generating interpolated frames between two adjacent video frames in each decoded sequence, wherein the method for generating the interpolated frames includes:

[0083] Step S601: Obtain the spatiotemporal domain information corresponding to the first video frame and the second video frame respectively;

[0084] Step S602: Perform forward and reverse prediction operations based on the two spatiotemporal domain information to obtain interpolation reference frames corresponding to the two prediction operations respectively. The forward prediction starts at the first video frame and ends at the second video frame; the reverse prediction starts at the second video frame and ends at the first video frame.

[0085] Step S603: Fuse the two interpolation reference frames, the first video frame, and the second video frame to obtain the interpolation frame corresponding to the first video frame and the second video frame.

[0086] Specifically, for downsampled target subsequences, after encoding and decoding, upsampling technology needs to be activated at the decoding end to restore the frame rate of the decoded sequence. This is equivalent to performing video frame interpolation at the decoding end to achieve temporal content restoration. Currently, mainstream video frame interpolation algorithms are typically based on optical flow. This requires first predicting the optical flow, then using the optical flow to deform the input frame to obtain the interpolation reference frame, and finally synthesizing the final interpolated frame using the interpolation reference frame and the input frame. The entire training process is relatively unstable and requires phased training, making the steps cumbersome. Therefore, this embodiment provides a new video frame interpolation method, such as... Figure 3 As shown, the interpolation reference frame is directly predicted using the spatiotemporal information of two video frames. Specifically, this embodiment requires two prediction operations: a forward prediction and a reverse prediction. The prediction start and end points of the two operations are different, so the prediction results can be mutually corrected, improving the reliability of the data. Finally, the two predicted interpolation reference frames are fused with the two original video frames to obtain the final interpolation frame.

[0087] For example, the first and second input video frames are I t-1 ,I t+1 The interpolation reference frame obtained from the forward prediction operation is The interpolation reference frame obtained by the reverse prediction operation is Using convolutional neural network tools, the final interpolated frame I can be obtained by fusing the components. t Convolutional neural network tools can include ResNet, Self-Attention units, etc.

[0088] In one implementation, the spatiotemporal information includes spatial information and temporal features, and the prediction operation is as follows:

[0089] Step S6021: Determine the attention weights corresponding to the prediction start point and the prediction end point respectively based on the spatial information corresponding to the prediction start point and the prediction end point respectively;

[0090] Step S6022: Determine the fused temporal features based on the temporal features and attention weights corresponding to the prediction start point and the prediction end point, respectively;

[0091] Step S6023: Determine the enhanced time domain features based on the time domain features of the predicted endpoint and the fused time domain features;

[0092] Step S6024: Determine the interpolation reference frame based on the spatial information of the predicted endpoint and the enhanced temporal features.

[0093] Specifically, such as Figure 4 As shown, this embodiment pre-constructs a motion sensing unit, which includes an attention module and a fusion module. The spatial and temporal features of the prediction start and end points are input into the attention module. The attention module first generates a weight distribution map based on the input spatial information to determine the attention weights for the prediction start and end points. Then, it generates fused temporal features using the input temporal features and the weight distribution map. Finally, it generates enhanced temporal features using the temporal features of the prediction end point and the fused temporal features. The fusion module then fuses the enhanced temporal features with the spatial information of the prediction end point to obtain an interpolated reference frame. It is understood that since the prediction ends of the forward and reverse prediction operations are different, the interpolated reference frames generated by the two prediction operations are also different.

[0094] In one implementation, step S6021 specifically includes:

[0095] Step S60211: Obtain the spatial characteristics of the predicted endpoint based on the spatial information of the predicted endpoint;

[0096] Step S60212, performing element-wise multiplication operation on the spatial information and the spatial feature of the predicted start point to obtain attention score of the predicted start point;

[0097] Step S60213, performing element-wise multiplication operation on the spatial information and the spatial feature of the predicted end point to obtain attention score of the predicted end point;

[0098] Step S60214, determining the attention weight corresponding to the predicted start point and the predicted end point respectively according to the attention scores of the predicted start point and the predicted end point.

[0099] Specifically, when the prediction operation is forward, the calculation process of the attention weight is as follows:

[0100]

[0101] q t-1 =SUM(S t-1 ⊙S'),q t+1 =SUM(S t+1 ⊙S')

[0102] S′=W s *S t+1

[0103] wherein (q t-1 ,q t+1 ) represents the attention score; (α t-1 ,α t+1 ) represents the normalized attention weight; W represents the weight of the convolution layer; * represents the convolution operation; S t-1 , S t+1 represent the spatial information of the first video frame and the second video frame respectively; S' represents the deep feature of S t+1 extracted by the neural network W s .

[0104] Similarly, when the prediction operation is reverse, the calculation process of the attention weight is as follows:

[0105]

[0106] q t-1 =SUM(S t-1 ⊙S'),q t+1 =SUM(S t+1 ⊙S')

[0107] S′=W s *S t-1

[0108] In an implementation manner, the step S6022 specifically includes:

[0109] The step S60221, weighted sum is performed on the time domain features and the attention weights corresponding to the predicted start point and the predicted end point respectively, to obtain the fused time domain features.

[0110] For example, the calculation process of the fused time domain features is as follows:

[0111] T att = α t-1 · T t-1 + α t+1 · T t+1

[0112] Wherein, T att represents the fused time domain features; T t-1 and T t+1 represent the time domain features (i.e. the deep features extracted in the time domain) of the first video frame and the second video frame respectively.

[0113] In an implementation manner, the step S6023 specifically includes:

[0114] The step S60231, a first convolution operation is performed on the time domain features of the predicted end point, and a fusion gate is determined according to the result of the first convolution operation, wherein the fusion gate is a Sigmoid activation function, and the numerical value is distributed between 0 and 1, 1 represents that the gate is open, and 0 represents that the gate is closed;

[0115] The step S60232, a matrix element-by-element multiplication operation is performed according to the fusion gate and the time domain features of the predicted end point;

[0116] The step S60233, a matrix element-by-element multiplication operation is performed according to the difference between 1 and the fusion gate and the fused time domain features;

[0117] The step S60234, the enhanced time domain features are determined according to the sum of the results of the two matrix element-by-element multiplication operations.

[0118] For example, when the prediction operation is forward, the calculation process of the enhanced time domain features is as follows:

[0119] U f = σ (W f * T t+1 )

[0120] T AMI = U f ⊙ T t+1 + (1-U f ) ⊙ T att

[0121] wherein, U f represents a fusion gate, the value is distributed between 0 and 1, the weight close to 1 represents that the gate is open, the weight close to 0 represents that the gate is closed, and the control information flow; σ represents a Sigmoid activation function; ⊙ represents a matrix element-by-element multiplication operation; W is a network parameter; * is a convolution operation; T AMI represents an enhanced time domain feature.

[0122] Similarly, when the prediction operation is reverse, the calculation process of the enhanced time domain feature is as follows:

[0123] U f = σ(W f *T t-1 )

[0124] T AMI = U f ⊙T t-1 + (1-U f )⊙T att

[0125] In an implementation mode, the step S6024 specifically comprises:

[0126] Step S60241, performing a second convolution operation on the spatial domain information of the predicted end point, and determining an update gate according to a result of the second convolution operation, wherein the update gate is a Sigmoid activation function, the value is distributed between 0 and 1, 1 represents that the gate is open, and 0 represents that the gate is closed;

[0127] Step S60242, performing a third convolution operation on the spatial domain information of the predicted end point, and performing a matrix element-by-element multiplication operation according to the update gate and a result of the third convolution operation;

[0128] Step S60243, performing a fourth convolution operation on the enhanced time domain feature, and performing a matrix element-by-element multiplication operation according to a difference between 1 and the fusion gate and the fourth convolution operation, wherein the first convolution operation, the second convolution operation, the third convolution operation and the fourth convolution operation correspond to different network parameters respectively;

[0129] Step S60244, determining the interpolation reference frame according to a sum of results of the two matrix element-by-element multiplication operations.

[0130] For example, when the prediction operation is forward, the generation process of the interpolation reference frame is as follows:

[0131] U s = σ(W su *S t+1 ),

[0132]

[0133] wherein, U s represents an update gate, the value is distributed between 0 and 1, and the weight close to 1 represents that the gate is opened, and the weight close to 0 represents that the gate is closed, and is used to control the information flow.

[0134] Similarly, when the prediction operation is reverse, the generation process of the interpolation reference frame is as follows:

[0135] U s =σ(W su *S t-1 ),

[0136]

[0137] In an implementation mode, the method further comprises:

[0138] each of the sub-sequences except the target sub-sequence is taken as a standard sub-sequence;

[0139] a coding operation is performed on each of the standard sub-sequences to obtain a coding sequence corresponding to each of the standard sub-sequences respectively;

[0140] the output video corresponding to the input video is determined according to the coding sequences of each of the standard sub-sequences and the coding sequences of each target sub-sequence.

[0141] Specifically, the scene motion of the sub-sequences except the target sub-sequence is normal, and the sub-sequences can be directly processed in a traditional coding mode without downsampling. After all the sub-sequences are coded, the output video is obtained by summarizing.

[0142] In an implementation mode, the method further comprises:

[0143] video reconstruction quality evaluation information is determined according to the input video and the output video.

[0144] Specifically, the coding effect of the input video can be determined through the video reconstruction quality evaluation information. In order to prove the technical effect of the present application, the method of the present application is integrated into the reference software of AVS3, and is tested on multiple 4K sequences. Compared with the reference software HPM12.2, the performance is as follows, wherein the negative value represents performance gain.

[0145]

[0146] The present application has the following advantages:

[0147] 1. Compared with other coding frameworks, the present application first deploys the time domain variational resolution idea to the coding platform, and saves the code rate.

[0148] 2. Compared with other video frame interpolation methods, the application creatively reconstructs the interpolation reference frame by using the space-time domain information of the video frame, thereby omitting the cumbersome steps of optical flow prediction and staged training, and the model is more practical.

[0149] Based on the above embodiment, the application further provides an adaptive time domain variational resolution coding and decoding device, as shown in the principle block diagram in Figure 5 The device comprises:

[0150] A splitting module 01 is configured to acquire an input video and split the input video into a plurality of equal-length subsequences.

[0151] A decision module 02 is configured to acquire scene motion features corresponding to each of the subsequences, respectively, and determine a plurality of target subsequences from each of the subsequences according to the scene motion features.

[0152] A downsampling module 03 is configured to perform a downsampling operation on each of the target subsequences in the time domain to obtain a frame rate reduction sequence corresponding to each of the target subsequences, respectively.

[0153] A coding and decoding module 04 is configured to perform a coding and decoding operation on the frame rate reduction sequence of each of the target subsequences to obtain a decoded sequence corresponding to each of the target subsequences, respectively.

[0154] An upsampling module 05 is configured to perform an upsampling operation on each of the decoded sequences in the time domain to obtain a coding and decoding sequence corresponding to each of the target subsequences, respectively.

[0155] Based on the above embodiment, the application further provides a terminal, and a principle block diagram thereof can be as shown in Figure 6 The terminal comprises a processor, a memory, a network interface, and a display screen connected through a system bus. The processor of the terminal is configured to provide computing and control capabilities. The memory of the terminal comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the terminal is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement the adaptive time domain variational resolution coding and decoding method. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0156] Those skilled in the art can understand that, Figure 6 The principle block diagram shown in the above embodiment is only a block diagram of part of the structure related to the application scheme, and does not constitute a limitation on the terminal to which the application scheme is applied. Specifically, the terminal can comprise more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0157] In one implementation, the memory of the terminal stores more than one program and is configured to execute the more than one program by more than one processor includes instructions for performing the adaptive time domain variable resolution coding method.

[0158] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments of each method. Any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0159] In summary, the present application discloses an adaptive time domain variable resolution coding method, device, terminal and storage medium. The method comprises: acquiring an input video, dividing the input video into a plurality of equal-length subsequences; acquiring scene motion features corresponding to each subsequence respectively, determining a plurality of target subsequences from each subsequence according to the scene motion features; performing downsampling operation on each target subsequence in the time domain to obtain a plurality of frame rate reduction sequences corresponding to each target subsequence respectively; performing coding operation on each target subsequence frame rate reduction sequence to obtain a plurality of decoding sequences corresponding to each target subsequence respectively; and performing upsampling operation on each decoding sequence in the time domain to obtain a plurality of coding sequences corresponding to each target subsequence respectively. The present application selects target subsequences through the scene motion features of each subsequence, performs coding operation on the target subsequences after time domain downsampling, and then performs upsampling on the frame rate reduction decoding sequences at the decoding end, thereby realizing adaptive frame rate adjustment of specific regions in the time domain and saving the coding rate. The present application solves the problem that in the prior art, because the scene motion of some time domain regions in a video sequence is flat, fixed frame rate coding is used for the video sequence, which wastes a lot of coding rate.

[0160] It is to be understood that the application is not limited to the examples described above, which can be modified or adapted in several ways by those skilled in the art without departing from the scope of the present application, as defined by the appended claims.

Claims

1. An adaptive time-domain variable resolution coding method, characterized by, The method comprises: acquiring an input video, and dividing the input video into a plurality of equal-length subsequences; acquiring scene motion features corresponding to each of the subsequences, and determining a plurality of target subsequences from each of the subsequences according to the scene motion features; performing downsampling operation on each of the target subsequences in the time domain to obtain a plurality of sequences with reduced frame rates corresponding to each of the target subsequences respectively; performing encoding and decoding operation on the sequences with reduced frame rates of each of the target subsequences to obtain a plurality of decoded sequences corresponding to each of the target subsequences respectively; The up-sampling operation is generating an interpolation frame between every two adjacent video frames of each decoding sequence, and the generating method of the interpolation frame comprises: obtaining spatial-temporal domain information corresponding to a first video frame and a second video frame; performing forward and reverse prediction operations according to the two spatial-temporal domain information to obtain interpolation reference frames corresponding to the two prediction operations respectively, wherein the forward prediction starts from the first video frame and ends at the second video frame, and the reverse prediction starts from the second video frame and ends at the first video frame; the spatial-temporal domain information comprises spatial domain information and time domain features, and the prediction operation comprises: obtaining spatial domain features of the prediction end point according to the spatial domain information of the prediction end point; performing matrix element-by-element multiplication operation on the spatial domain information and the spatial domain features of the prediction start point to obtain attention scores of the prediction start point; performing matrix element-by-element multiplication operation on the spatial domain information and the spatial domain features of the prediction end point to obtain attention scores of the prediction end point; determining attention weights corresponding to the prediction start point and the prediction end point according to the attention scores of the prediction start point and the prediction end point; performing weighted summation on the time domain features and the attention weights corresponding to the prediction start point and the prediction end point to obtain fused time domain features; performing first convolution operation on the time domain features of the prediction end point, determining a fusion gate according to the result of the first convolution operation, wherein the fusion gate is a Sigmoid activation function, and the numerical value is distributed between 0 and 1, 1 represents that the gate is open, and 0 represents that the gate is closed; performing matrix element-by-element multiplication operation on the fusion gate and the time domain features of the prediction end point; performing matrix element-by-element multiplication operation on the difference between 1 and the fusion gate and the fused time domain features; determining enhanced time domain features according to the sum of the results of the two matrix element-by-element multiplication operations; performing second convolution operation on the spatial domain information of the prediction end point, determining an update gate according to the result of the second convolution operation, wherein the update gate is a Sigmoid activation function, and the numerical value is distributed between 0 and 1, 1 represents that the gate is open, and 0 represents that the gate is closed; performing third convolution operation on the spatial domain information of the prediction end point, performing matrix element-by-element multiplication operation on the update gate and the result of the third convolution operation; performing fourth convolution operation on the enhanced time domain features, and performing matrix element-by-element multiplication operation on the difference between 1 and the fusion gate and the fourth convolution operation, wherein the first convolution operation, the second convolution operation, the third convolution operation and the fourth convolution operation correspond to different network parameters respectively; determining the interpolation reference frame according to the sum of the results of the two matrix element-by-element multiplication operations; and fusing the two interpolation reference frames, the first video frame and the second video frame to obtain the interpolation frame corresponding to the first video frame and the second video frame.

2. The adaptive time-domain variable resolution coding method of claim 1, wherein, The length of each sub-sequence is equal to the length of a time domain basic unit, and the length of the time domain basic unit is 2s.

3. The adaptive time-domain variable resolution coding method of claim 1, wherein, The downsampling operation is frame rate doubling downsampling.

4. The adaptive time-domain variable resolution coding method of claim 1, wherein, The scene motion feature corresponding to each of the sub-sequences is obtained, and a plurality of target sub-sequences are determined from the sub-sequences according to the scene motion features, including: The downsampling operation is performed on each of the sub-sequences to obtain a frame rate reduction sequence corresponding to the sub-sequence; Frame rate recovery is performed on the frame rate reduction sequence of the sub-sequence by frame duplication to obtain a recovery sequence of the sub-sequence; The similarity between the sub-sequence and the recovery sequence of the sub-sequence is obtained, and a preset similarity threshold is obtained, and when the similarity value is higher than the similarity threshold, it is judged that the scene motion feature is motion smooth; The sub-sequence with the motion smooth scene motion feature is taken as the target sub-sequence.

5. An apparatus for adaptive time-domain variable resolution coding, the apparatus comprising: The device comprises: The splitting module is configured to obtain an input video and split the input video into a plurality of equal-length sub-sequences; The decision module is configured to obtain a scene motion feature corresponding to each of the sub-sequences, and determine a plurality of target sub-sequences from the sub-sequences according to the scene motion features; The downsampling module is configured to perform a downsampling operation on each of the target sub-sequences in the time domain to obtain a frame rate reduction sequence corresponding to each of the target sub-sequences; The coding and decoding module is configured to perform a coding and decoding operation on the frame rate reduction sequence of each of the target sub-sequences to obtain a decoding sequence corresponding to each of the target sub-sequences; The upsampling module is configured to perform upsampling on each of the decoding sequences in the time domain to obtain a corresponding coding sequence of each of the target sub-sequences; the upsampling operation is to generate an interpolation frame between two adjacent video frames of each of the decoding sequences, and the generation method of the interpolation frame comprises: obtaining spatial-temporal domain information corresponding to a first video frame and a second video frame; performing forward and reverse prediction operations according to the two spatial-temporal domain information to obtain two interpolation reference frames corresponding to the two prediction operations respectively, wherein the forward prediction starts from the first video frame and ends at the second video frame, and the reverse prediction starts from the second video frame and ends at the first video frame; the spatial-temporal domain information comprises spatial domain information and time domain features, and the prediction operation comprises: obtaining spatial domain features of the prediction end point according to the spatial domain information of the prediction end point; performing matrix element-by-element multiplication operation on the spatial domain information and the spatial domain features of the prediction start point to obtain attention scores of the prediction start point; performing matrix element-by-element multiplication operation on the spatial domain information and the spatial domain features of the prediction end point to obtain attention scores of the prediction end point; determining attention weights corresponding to the prediction start point and the prediction end point according to the attention scores of the prediction start point and the prediction end point; performing weighted summation on the time domain features and the attention weights corresponding to the prediction start point and the prediction end point to obtain fused time domain features; performing first convolution operation on the time domain features of the prediction end point, determining a fusion gate according to a result of the first convolution operation, the fusion gate being a Sigmoid activation function with a value distribution between 0 and 1, 1 representing an open gate and 0 representing a closed gate; performing matrix element-by-element multiplication operation on the fusion gate and the time domain features of the prediction end point; performing matrix element-by-element multiplication operation on the fused time domain features and a difference between 1 and the fusion gate; determining enhanced time domain features according to a sum of results of the two matrix element-by-element multiplication operations; performing second convolution operation on the spatial domain information of the prediction end point, determining an update gate according to a result of the second convolution operation, the update gate being a Sigmoid activation function with a value distribution between 0 and 1, 1 representing an open gate and 0 representing a closed gate; performing third convolution operation on the spatial domain information of the prediction end point, performing matrix element-by-element multiplication operation on a result of the third convolution operation and the update gate; performing fourth convolution operation on the enhanced time domain features, performing matrix element-by-element multiplication operation on a result of the fourth convolution operation and the difference between 1 and the fusion gate, the first convolution operation, the second convolution operation, the third convolution operation and the fourth convolution operation corresponding to different network parameters respectively; determining the interpolation reference frame according to a sum of results of the two matrix element-by-element multiplication operations; and fusing the two interpolation reference frames, the first video frame and the second video frame to obtain the interpolation frame corresponding to the first video frame and the second video frame.

6. A terminal, characterized by comprising: The terminal comprises a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing the adaptive time domain variable resolution coding method as claimed in any one of claims 1-4; the processors are used to execute the programs.

7. A computer-readable storage medium storing a plurality of instructions thereon, characterized in that, The instructions are suitable for being loaded and executed by the processors to implement the steps of the adaptive time domain variable resolution coding method as claimed in any one of claims 1-4.

Images