Data processing method and device, equipment and storage medium

Abstract

The invention provides a data processing method and device, equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: acquiring a frame level parameter, determining whether a time domain sampling magnification change flag bit is set or not according to the frame level parameter, and under the condition that the time domain sampling magnification change flag bit is not set, performing frame compensation based on a frame compensation strategy corresponding to a previous frame of a current frame, thereby improving the video coding and decoding efficiency.

Description

Data processing methods, apparatus, equipment and storage media

[0001] Cross-references to related applications

[0002] This application claims ownership of the application filed on October 30, 2024, entitled "Data Processing Method, Apparatus, Equipment and Storage". The priority of Chinese patent application 202411535551.8 concerning “storage medium” is hereby acknowledged, the entire contents of which are incorporated herein by reference. middle. Technical Field

[0003] This disclosure relates to the field of computer technology, and in particular to a data processing method, apparatus, device and storage medium. Background Technology

[0004] With the development of the times, the speed of information transmission has accelerated. Currently, video information accounts for a large proportion of the transmitted information.

[0005] Frame extraction coding is a commonly used technique in signal encoding technology during information transmission. Current frame extraction coding techniques typically extract frames from video at a fixed extraction ratio, and then encode the extracted video. However, with technological advancements, the existing frame extraction and interpolation methods are no longer sufficient to meet user needs. How to efficiently encode and decode video is a problem that urgently needs to be solved in this field. Summary of the Invention

[0006] This disclosure provides a data processing method, apparatus, device, and storage medium that improves the efficiency of video encoding and decoding to at least a certain extent.

[0007] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part by practice of this disclosure.

[0008] According to one aspect of this disclosure, a data processing method is provided, applied at a decoding end, comprising:

[0009] Get frame-level parameters;

[0010] Determine whether to set the temporal sampling rate change flag based on frame-level parameters;

[0011] If the temporal sampling rate change flag is not set, frame interpolation is performed based on the interpolation strategy corresponding to the previous frame of the current frame.

[0012] In one embodiment of this disclosure, the temporal sampling rate change flag position includes at least one of sampling rate change, frame interpolation method change, and sampling interval frame number change.

[0013] In one embodiment of this disclosure, the sampling interval frame number includes the number of sampled frames and the number of predicted frames.

[0014] In one embodiment of this disclosure, the method further includes:

[0015] When the time-domain sampling rate change flag is set, frame interpolation is performed based on the changed sampling rate, the changed interpolation method, and / or the changed number of sampling frames.

[0016] In one embodiment of this disclosure, frame interpolation based on the changed interpolation method includes:

[0017] Methods for obtaining historical frame interpolation;

[0018] When the historical frame interpolation method is forward and backward interpolation, forward and backward interpolation and backward interpolation are completed based on the current frame.

[0019] In one embodiment of this disclosure, when the temporal sampling rate change flag is not set, frame interpolation is performed based on the interpolation strategy corresponding to the previous frame of the current frame, including:

[0020] Input the current frame into the reference frame sequence;

[0021] Determine whether the number of frames in the reference frame sequence reaches a preset threshold, wherein the preset threshold is determined by the number of sampled frames;

[0022] When the number of frames in the reference frame sequence reaches a preset threshold, the prediction frame is determined based on the reference frames in the reference frame sequence, the number of prediction frames, and the prediction module.

[0023] In one embodiment of this disclosure, when the temporal sampling rate change flag is not set, frame interpolation is performed based on the interpolation strategy corresponding to the previous frame of the current frame, including:

[0024] Obtain the current frame and the reference frame sequence;

[0025] Determine whether the number of frames within the reference frame sequence reaches a preset threshold, wherein the preset threshold is determined by the number of sampled frames;

[0026] When the number of frames in the reference frame sequence reaches a preset threshold, the prediction frame is determined based on the current frame, the reference frames in the reference frame sequence, the number of prediction frames, and the prediction module.

[0027] In one embodiment of this disclosure, the method further includes:

[0028] After determining the prediction frame, the current frame is input into the reference frame sequence.

[0029] In one embodiment of this disclosure, the method further includes:

[0030] Get the current frame;

[0031] Obtain frame-level parameters, including:

[0032] If the current frame is obtained, the corresponding frame temporal information is obtained, which includes frame-level parameters.

[0033] In one embodiment of this disclosure, the method further includes:

[0034] Determine the interpolation method and the number of remaining frames for the current frame, which includes the last sampled frame;

[0035] When the frame interpolation method is backward interpolation, the backward interpolation yields the predicted frame, and the number of predicted frames remaining is retained.

[0036] When the frame interpolation method is forward and backward interpolation, the current frame is copied to obtain a copied frame, and the number of copied frames remaining is retained.

[0037] In one embodiment of this disclosure, the current frame includes a sampled frame and a predicted frame.

[0038] In one embodiment of this disclosure, the method further includes:

[0039] The number of predictions is determined based on at least one of the following: the number of sampled frames in the interpolated frame segment, the total number of frames in the interpolated frame segment, the number of predicted frames, and the prediction capability of the prediction model. The prediction capability of the prediction model includes the number of predicted frames obtained by the prediction model each time it makes a prediction.

[0040] Input the sampled frame into the reference frame sequence;

[0041] Determine whether the number of sampled frames within the reference frame sequence has reached a preset threshold;

[0042] When the number of sampled frames in the reference frame sequence reaches a preset threshold, the predicted frame is obtained;

[0043] Repeat the above steps until the preset number of times is reached.

[0044] In one embodiment of this disclosure, the method further includes:

[0045] Input the sampled frame and the predicted frame into the reference frame sequence;

[0046] Determine whether the number of sampled frames within the reference frame sequence has reached a preset threshold;

[0047] When the number of sampled frames and predicted frames in the reference frame sequence reaches a preset threshold, a predicted frame is obtained.

[0048] Repeat the above steps until the number of sampled frames and predicted frames reaches the total number of frames in the interpolation segment.

[0049] In one embodiment of this disclosure, when the temporal sampling rate change flag is not set, frame interpolation based on the interpolation strategy corresponding to the previous frame of the current frame includes:

[0050] Obtain global parameters when the time-domain sampling rate change flag is not set;

[0051] Frame interpolation is performed based on at least one of the following global parameters: sampling rate, interpolation method, and sampling interval frame number.

[0052] In one embodiment of this disclosure, the method further includes:

[0053] When the temporal sampling rate change flag is set, the global parameters are updated based on the changed sampling rate, the changed frame interpolation method, and / or the changed sampling interval frame number.

[0054] According to another aspect of this disclosure, a data processing method is provided, applied at an encoding end, comprising:

[0055] The video is encoded to obtain bitstream data, so that the decoding end can implement any of the data processing methods described in the above embodiments.

[0056] According to another aspect of this disclosure, a data processing apparatus is provided for use at a decoding end, comprising:

[0057] The first acquisition module is used to acquire frame-level parameters;

[0058] The first determining module is used to determine whether the temporal sampling rate change flag is set based on the frame-level parameters;

[0059] The first frame interpolation module is used to perform frame interpolation based on the frame interpolation strategy corresponding to the previous frame of the current frame when the time domain sampling rate change flag is not set.

[0060] In one embodiment of this disclosure, the temporal sampling rate change flag position includes at least one of sampling rate change, frame interpolation method change, and sampling interval frame number change.

[0061] In one embodiment of this disclosure, the apparatus further includes:

[0062] The second frame interpolation module is used to perform frame interpolation based on the changed sampling rate, the changed frame interpolation method, and / or the changed number of sampling interval frames when the time-domain sampling rate change flag is in position.

[0063] In one embodiment of this disclosure, the second frame interpolation module includes:

[0064] The first acquisition unit is used to acquire historical frame interpolation methods;

[0065] The first interpolation unit is used to perform forward and backward interpolation and backward interpolation based on the current frame when the historical interpolation method is forward and backward interpolation.

[0066] In one embodiment of this disclosure, the first frame interpolation module includes:

[0067] The first input unit is used to input the current frame into the reference frame sequence;

[0068] The first determining unit is used to determine whether the number of frames in the reference frame sequence reaches a preset threshold, wherein the preset threshold is determined by the number of frames in the sampling interval;

[0069] The second determining unit is used to determine the prediction frame based on the reference frames in the reference frame sequence, the number of sampling interval frames, and the prediction module when the number of frames in the reference frame sequence reaches a preset threshold.

[0070] In one embodiment of this disclosure, the first frame interpolation module includes:

[0071] The second acquisition unit is used to acquire the current frame and the reference frame sequence;

[0072] The third determining unit is used to determine whether the number of frames in the reference frame sequence reaches a preset threshold, wherein the preset threshold is determined by the number of frames in the sampling interval;

[0073] The fourth determining unit is used to determine the prediction frame based on the current frame, the reference frames in the reference frame sequence, the number of sampling interval frames, and the prediction module when the number of frames in the reference frame sequence reaches a preset threshold.

[0074] In one embodiment of this disclosure, the apparatus further includes:

[0075] The second input module is used to input the current frame into the reference frame sequence after the predicted frame is determined.

[0076] In one embodiment of this disclosure, the apparatus further includes:

[0077] The second acquisition module is used to acquire the current frame;

[0078] The first acquisition module includes:

[0079] The third acquisition unit, when the current frame is acquired, is used to acquire the frame temporal information corresponding to the current frame, which includes frame-level parameters.

[0080] In one embodiment of this disclosure, the apparatus further includes:

[0081] The second determining module is used to determine the frame interpolation method and the number of remaining frames corresponding to the current frame, where the current frame includes the last sampled frame.

[0082] The first retention module is used to obtain the predicted frame by backward interpolation when the interpolation method is backward interpolation, and to retain the predicted frame of the remaining frame number.

[0083] The first copying module is used to copy the current frame when the frame interpolation method is forward and backward frame interpolation, to obtain a copied frame, and to retain the number of copied frames remaining.

[0084] In one embodiment of this disclosure, the current frame includes a sampled frame and a predicted frame.

[0085] In one embodiment of this disclosure, the apparatus further includes:

[0086] The third determining module is used to determine the number of predictions based on at least one of the following: the number of sampled frames in the interpolated frame segment, the total number of frames in the interpolated frame segment, the number of sampling interval frames, and the prediction capability of the prediction model. The prediction capability of the prediction model includes the number of prediction frames obtained by the prediction model each time it makes a prediction.

[0087] The third input module is used to input the sampled frame into the reference frame sequence;

[0088] Determine whether the number of sampled frames within the reference frame sequence has reached a preset threshold;

[0089] The fourth determining module is used to obtain the predicted frame when the number of sampled frames in the reference frame sequence reaches a preset threshold.

[0090] The first repeating module is used to repeat the above steps until the preset number of times is reached.

[0091] In one embodiment of this disclosure, the apparatus further includes:

[0092] The fourth input module is used to input the sampled frame and the predicted frame into the reference frame sequence;

[0093] The fifth determining module is used to determine whether the number of sampled frames in the reference frame sequence has reached a preset threshold.

[0094] The sixth determining module is used to obtain the predicted frame when the number of sampled frames and predicted frames in the reference frame sequence reaches a preset threshold.

[0095] The second repeating module is used to repeat the above steps until the number of sampled frames and predicted frames reaches the total number of frames in the interpolation segment.

[0096] In one embodiment of this disclosure, the first frame interpolation module includes:

[0097] The third acquisition unit is used to acquire global parameters when the time-domain sampling rate change flag is not set;

[0098] The fourth frame interpolation unit is used to perform frame interpolation based on at least one of the following: the sampling rate, the frame interpolation method, and the number of sampling interval frames corresponding to the global parameters.

[0099] In one embodiment of this disclosure, the apparatus further includes:

[0100] The update module is used to update global parameters based on the changed sampling rate, the changed frame interpolation method, and / or the changed sampling interval frame number when the time-domain sampling rate change flag is set.

[0101] According to another aspect of this disclosure, a data processing apparatus is provided, applied at an encoding end, comprising:

[0102] The encoding module is used to encode the video to obtain bitstream data, so that the decoding end can implement any of the data processing methods described above based on the bitstream data.

[0103] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above-described data processing method by executing the executable instructions.

[0104] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described data processing method.

[0105] The data processing method, apparatus, device, and storage medium provided in the embodiments of this disclosure acquire frame-level parameters, determine whether the temporal sampling rate change flag is set based on the frame-level parameters, and perform frame interpolation based on the interpolation strategy corresponding to the previous frame of the current frame when the temporal sampling rate change flag is not set, thereby improving the efficiency of video encoding and decoding.

[0106] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0107] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0108] Figure 1 shows a flowchart of a data processing method according to an embodiment of the present disclosure;

[0109] Figure 2 shows a flowchart of another data processing method in an embodiment of this disclosure;

[0110] Figure 3 shows a flowchart of another data processing method according to an embodiment of the present disclosure;

[0111] Figure 4 shows a frame interpolation schematic diagram according to an embodiment of the present disclosure;

[0112] Figure 5 shows a flowchart of another data processing method in an embodiment of this disclosure;

[0113] Figure 6 shows a flowchart of another data processing method according to an embodiment of the present disclosure;

[0114] Figure 7 shows a flowchart of another data processing method according to an embodiment of the present disclosure;

[0115] Figure 8 shows a flowchart of another data processing method according to an embodiment of the present disclosure;

[0116] Figure 9 shows a flowchart of another data processing method according to an embodiment of the present disclosure;

[0117] Figure 10 shows an architecture diagram of a data processing apparatus according to an embodiment of the present disclosure;

[0118] Figure 11 shows an architecture diagram of another data processing apparatus according to an embodiment of the present disclosure;

[0119] Figure 12 shows a structural block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0120] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0121] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0122] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0123] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0124] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0125] It should be noted that no parameters are limited in this disclosure.

[0126] This disclosure applies to image input and feature tensor input.

[0127] To address the aforementioned problems, this disclosure provides a data processing method, apparatus, device, and storage medium.

[0128] Figure 1 shows a flowchart of a data processing method according to an embodiment of the present disclosure.

[0129] As shown in Figure 1, the method may include:

[0130] S110, obtain frame-level parameters.

[0131] In some embodiments, frame-level parameters can correspond to frames. Frame-level parameters can be determined after a frame is determined. It should be noted that frame-level parameters correspond to frames, but frames may not correspond to frame-level parameters.

[0132] In some embodiments, the method may further include:

[0133] Get the current frame.

[0134] If the current frame is obtained, the corresponding frame temporal information is obtained, which includes frame-level parameters.

[0135] S120, determine whether to set the time-domain sampling rate change flag bit based on frame-level parameters.

[0136] In some embodiments, the temporal sampling rate change flag may include prdTemChangedFlag being 1. In some embodiments, the temporal sampling rate change flag may include prd_temporal_resampling_ratio_changed_flag = 1.

[0137] In some embodiments, the temporal sampling rate change flag position includes at least one of sampling rate change, frame interpolation method change, and sampling interval frame number change. In some embodiments, the sampling interval frame number includes the number of sampled frames and the number of predicted frames.

[0138] In some embodiments, the sampling rate includes the sampling rate in the case of forward and backward prediction.

[0139] For example, the frame interpolation method can be the current temporal restoration mode; the sampling ratio can be the current temporal interpolation ratio; and the number of frames in the sampling interval can be the current temporal extrapolation predict frames number or the current temporal extrapolation resample frames number.

[0140] In some embodiments, the time-domain sampling rate change flag is set when at least one of the following occurs: a change in sampling rate, a change in frame interpolation method, and a change in the number of frames in the sampling interval.

[0141] In some embodiments, when the temporal sampling rate change flag is set, the frame interpolation mode, TemporalRestorationMode[i], is updated. If the updated frame interpolation mode, TemporalRestorationMode[i] = 0, the sampling rate is also updated. That is, the sampling rate is updated for both forward and backward sampling. If the updated frame interpolation mode, TemporalRestorationMode[i] = 1, the number of sampling interval frames, namely the number of sampled frames, TemporalExtrapolationResampleNum[i], and the number of predicted frames, TemporalExtralationPredictNum[i], are also updated.

[0142] In some embodiments, the sampling ratio TemporalInterpolationRatio can be determined by the following formula:

[0143] TemporalInterpolationRatio=2^(srd_temporal_interpolation_ratio_idx+1)

[0144] The value of srd_temporal_interpolation_ratio_idx ranges from 0, 1, and 2.

[0145] In some embodiments, the number of sampling frames, TemporalExtralationResampleNum, can be determined by the following formula:

[0146] srd_temporal_extrapolation_resample_num_idx+2 = number of sampling frames

[0147] In some embodiments, when srd_temporal_restoration_mode = 0, the bitstream does not contain the back-padding frame parameter srd_temporal_extrapolation_resample_num_idx.

[0148] In some embodiments, the number of predicted frames, TemporalExtrapolationPredictNum, can be determined by the following formula:

[0149] TemporalExtrapolationPredictNum=srd_temporal_extrapolation_predict_num_idx+1

[0150] In some embodiments, when srd_temporal_restoration_mode = 0, i.e., the interpolation method is forward and backward interpolation, the bitstream does not contain srd_temporal_extrapolation_predict_num_idx.

[0151] The value range of srd_temporal_extrapolation_predict_num_idx is 0, 1, and 2.

[0152] In some embodiments, when prd_temporal_resampling_ratio_changed = 1, TemporalInterpolationRatio[i] is updated by the following formula:

[0153] TemporalInterpolationRatio[i]=2^(prd_temporal_interpolation_ratio_idx);

[0154] In some embodiments, when prd_temporal_resampling_ratio_changed = 0, TemporalInterpolationRatio[i] is updated by the following formula:

[0155] TemporalInterpolationRatio[i]=TemporalInterpolationRatio[i-1].

[0156] In some embodiments, when prd_temporal_resampling_ratio_changed = 0 or prd_temporal_restoration_mode = 1, prd_temporal_interpolation_ratio_idx will not be in the bitstream. The value range of prd_temporal_interpolation_ratio_idx is 0, 1, 2, and 3. When prd_temporal_interpolation_ratio_idx = 0, it indicates that temporal forward and backward frame interpolation is disabled.

[0157] In some embodiments, the number of sampling frames can be TemporalExtrapolationResampleNum.

[0158] In some embodiments, when prd_temporal_resampling_ratio_changed = 1:

[0159] TemporalExtrapolationResampleNum[i]=prd_temporal_extrapolation_resample_num_idx+2.

[0160] Wherein, prd_temporal_extrapolation_resample_num_idx is a preset value, which can be 0 or 1. This disclosure does not specify a particular size for the preset value.

[0161] otherwise:

[0162] TemporalExtrapolationResampleNum[i]=TemporalExtrapolationResampleNum[i-1]

[0163] In some embodiments, when prd_temporal_resampling_ratio_changed = 0 or prd_temporal_restoration_mode = 0, prd_temporal_extrapolation_resample_num_idx is not present in the bitstream. The value range of prd_temporal_extrapolation_resample_num_idx is 0 and 1.

[0164] In some embodiments, the number of predicted frames can be TemporalExtrapolationPredictNum.

[0165] In some embodiments, when prd_temporal_resampling_ratio_changed = 1:

[0166] TemporalExtrapolationPredictNum[i]=prd_temporal_extrapolation_predict_num_idx

[0167] otherwise:

[0168] TemporalExtrapolationPredictNum[i]=TemporalExtrapolationPredictNum[i-1]

[0169] That is, the predicted frame number can be the historical predicted frame number, and the historical predicted frame number can be the previous predicted frame number.

[0170] In some embodiments, prd_temporal_extrapolation_predict_num_idx will not be in the bitstream when prd_temporal_resampling_ratio_changed = 0 or prd_temporal_restoration_mode = 0.

[0171] The value range of prd_temporal_extrapolation_predict_num_idx is 0, 1, 2, and 3.

[0172] prd_temporal_extrapolation_predict_num_idx = 0 indicates that backward interpolation in the current temporal domain is disabled.

[0173] For example:

[0174] Any change to prd_temporal_resampling_ratio_idx, rd_temporal_extrapolation_flag, or prd_temporal_extra_resampling_length_idx is considered a change in the temporal sampling ratio flag.

[0175] prd_temporal_resampling_ratio_idx indicates a change in the sampling ratio.

[0176] prd_temporal_extrapolation_flag indicates a change in the frame interpolation method or sampling method.

[0177] prd_temporal_extra_resampling_length_idx can cause changes in the number of frames inserted after sampling or the number of frames sampled at the sampling interval.

[0178] S130: If the temporal sampling rate change flag is not set, perform frame interpolation based on the interpolation strategy corresponding to the previous frame of the current frame. In some embodiments, if the temporal sampling rate change flag is not set, the sampling rate, interpolation method, and sampling interval frame number remain unchanged.

[0179] In some embodiments, the fact that the temporal sampling rate change flag is not set may indicate that the frame interpolation method has not changed.

[0180] In some embodiments, keeping the frame interpolation method unchanged may include the current temporal reconstruction mode TemporalRestorationMode[i] = 0.

[0181] For example, if the current temporal reconstruction mode TemporalRestorationMode[i] = 0, and the current sampling ratio TemporalInterpolationRatio[i] is not equal to 1, then the current i-th image, the next (i+1) image, and the current sampling ratio TemporalInterpolationRatio[i] are used for temporal forward and backward frame interpolation.

[0182] In some embodiments, global parameters are obtained when the time-domain sampling rate change flag is not set.

[0183] Frame interpolation is performed based on at least one of the following global parameters: sampling rate, interpolation method, and sampling interval frame number.

[0184] In some embodiments, global parameters can be sequence-level parameters.

[0185] For example, sequence-level parameters may include RestorationRatio.

[0186] In some embodiments, when the temporal sampling rate change flag is set, the global parameters are updated based on the changed sampling rate, the changed frame interpolation method, and / or the changed sampling interval frame number.

[0187] For example, when prdTemChangedFlag is 1, the sequence-level parameter RestorationRatio is updated to the RestorationRatio corresponding to the current frame = curRestorationRatio = prd_temporal_resampling_ratio_idx.

[0188] The data processing method provided in the embodiments of this disclosure obtains frame-level parameters, determines whether the temporal sampling rate change flag is set based on the frame-level parameters, and performs frame interpolation based on the interpolation strategy corresponding to the previous frame of the current frame when the temporal sampling rate change flag is not set, thereby improving the efficiency of video encoding and decoding.

[0189] Figure 2 shows a flowchart of another data processing method in an embodiment of this disclosure.

[0190] As shown in Figure 2, the method may include:

[0191] S210, obtain frame-level parameters;

[0192] S220, determine whether to set the time-domain sampling rate change flag bit based on frame-level parameters.

[0193] S230, when the time-domain sampling rate change flag is set, performs frame interpolation based on the changed sampling rate, the changed interpolation method, and / or the changed sampling interval frame number.

[0194] In some embodiments, S230 may include: performing frame interpolation based on the changed interpolation method and the changed sampling rate when the time-domain sampling rate change flag is set.

[0195] In some embodiments, S230 may include: when the temporal sampling rate change flag is set, performing frame interpolation based on the changed interpolation method and the changed sampling interval frame number. In some embodiments, this may be based on RestorationRatio = curRestorationRatio = prd_temporal_resampling_ratio_idx corresponding to the current frame.

[0196] Modify global parameters.

[0197] In some embodiments, global parameters can also be updated based on information contained in the bitstream data.

[0198] In some embodiments, modifying global parameters includes modifying the sampling rate to obtain a modified sampling rate.

[0199] Frame interpolation is performed based on the modified sampling rate and interpolation method.

[0200] The data processing method provided in the embodiments of this disclosure obtains frame-level parameters, determines whether the temporal sampling rate change flag is set based on the frame-level parameters, and performs frame interpolation based on the interpolation strategy corresponding to the previous frame of the current frame when the temporal sampling rate change flag is not set, thereby improving the efficiency of video encoding and decoding.

[0201] Figure 3 shows a flowchart of another data processing method in an embodiment of this disclosure.

[0202] As shown in Figure 3, the method may include:

[0203] S310, obtain historical frame interpolation method.

[0204] In some embodiments, the historical frame interpolation method can be the frame interpolation method of the previous frame of the current frame.

[0205] S320, when the historical frame interpolation method is forward and backward frame interpolation, completes forward and backward frame interpolation and backward frame interpolation based on the current frame.

[0206] In some embodiments, if the previous frame interpolation mode TemporalRestorationMode[i-1] is 0 and the current frame interpolation mode emporalRestorationMode[i] is 1, then forward and backward frame interpolation is completed based on the current frame, and then backward frame interpolation is completed.

[0207] The previous frame interpolation method can be a historical frame interpolation method.

[0208] To provide a detailed explanation of the data processing method in the embodiments of this disclosure, Figure 4 shows a frame interpolation schematic diagram in an embodiment of this disclosure. As shown in Figure 4, the frame interpolation method used before the current frame is forward and backward frame interpolation. After the current frame is used as a reference frame to complete the prediction of the predicted frames before the current frame, it is then used as a reference frame to complete the prediction of the predicted frames after the current frame.

[0209] The data processing method provided in the embodiments of this disclosure obtains frame-level parameters, determines whether the temporal sampling rate change flag is set based on the frame-level parameters, and performs frame interpolation based on the interpolation strategy corresponding to the previous frame of the current frame when the temporal sampling rate change flag is not set, thereby improving the efficiency of video encoding and decoding.

[0210] Figure 5 shows a flowchart of another data processing method in an embodiment of this disclosure.

[0211] As shown in Figure 5, the method may include:

[0212] S510, input the current frame into the reference frame sequence;

[0213] S520, determine whether the number of frames in the reference frame sequence has reached a preset threshold, wherein the preset threshold is determined by the number of frames in the sampling interval.

[0214] In some embodiments, the sampling interval frame number can be the prediction frame number TemporalExtrapolationPredictNum[i]. S530, when the number of frames in the reference frame sequence reaches a preset threshold, the prediction frame is determined based on the reference frames in the reference frame sequence, the sampling interval frame number, and the prediction module.

[0215] In some embodiments, the reference sequence can be cleared when the interpolation ends or the interpolation method changes.

[0216] In some embodiments, after determining the predicted frame, the insertion position of the predicted frame in the frame-complementing segment can be determined according to the frame-complementing method and global parameters.

[0217] In some embodiments, if the interpolation method is backward interpolation, the predicted frame can be inserted after the current frame.

[0218] In some embodiments, if the interpolation method is forward and backward interpolation, the predicted frame can be inserted between the current frame and the frames preceding the current frame. For example, the specific insertion position of the predicted frame can be determined based on global parameters, and this disclosure does not make any specific limitations.

[0219] In some embodiments, if the frame interpolation method is forward and backward frame interpolation, the predicted frame can be inserted between the current frame and the frames preceding the current frame. For example, the specific insertion position of the predicted frame can be determined based on the number of sampling interval frames, and this disclosure does not make any specific limitations.

[0220] For example, when acquiring a new frame of data, if the acquisition is successful, the acquired frame data is pushed into the frameBuffer; otherwise, the remaining frame is processed, the temporal information of the current frame is acquired, and when prdTemChangedFlag is 1, the sequence-level parameter RestorationRatio is updated to the RestorationRatio corresponding to the current frame = curRestorationRatio = prd_temporal_resampling_ratio_idx. It is then checked whether the frameBuffer is full; if so, the frameBuffer and RestorationRatio are input to the extrapolation function to obtain RestorationRatio prediction frames. The sampled frame (frameBuffer) and prediction frames are saved sequentially. If the frameBuffer is empty, a new frame of data is acquired again.

[0221] The data processing method provided in the embodiments of this disclosure obtains frame-level parameters, determines whether the temporal sampling rate change flag is set based on the frame-level parameters, and performs frame interpolation based on the interpolation strategy corresponding to the previous frame of the current frame when the temporal sampling rate change flag is not set, thereby improving the efficiency of video encoding and decoding.

[0222] Figure 6 shows a flowchart of another data processing method according to an embodiment of this disclosure.

[0223] As shown in Figure 6, the method may include:

[0224] S610, Obtain the current frame and reference frame sequence;

[0225] In some embodiments, the reference frame may include a sampled frame and a predicted frame, wherein the predicted frame is a frame obtained through prediction.

[0226] For example, a predicted frame can be obtained by making a prediction based on a video prediction model and a reference frame.

[0227] For example, a prediction frame can be obtained by using the extrapolation function and a reference frame.

[0228] In some embodiments, the reference frame sequence may be a frameBuffer.

[0229] S620, determine whether the number of frames in the reference frame sequence reaches a preset threshold, wherein the preset threshold is determined by the number of frames in the sampling interval.

[0230] In some embodiments, after determining the number of sampling interval frames, the number of reference frames required to obtain the prediction frame can be determined, and the number of required reference frames is the preset threshold.

[0231] For example, the preset threshold can be M1.

[0232] S630, when the number of frames in the reference frame sequence reaches a preset threshold, the prediction frame is determined based on the current frame, the reference frames in the reference frame sequence, the number of sampling interval frames, and the prediction module.

[0233] In some embodiments, the current frame, reference frames in the reference sequence, and the number of sampling interval frames can be input into the prediction module to obtain the prediction frame.

[0234] For example, the sampling interval can be ResamplingLen, and the prediction interval can be RestorationLen.

[0235] In some embodiments, after determining the predicted frame, the current frame can be input into the reference frame sequence, and the number of frames in the reference frame sequence can be incremented by 1.

[0236] In some embodiments, if the number of frames in the reference frame sequence exceeds a second preset threshold after the current frame is input into the reference frame sequence, then the reference frame with the longest duration in the reference frame sequence is deleted. The second preset threshold may be the same as a preset threshold.

[0237] In some embodiments, after the predicted frame is determined, the insertion position of the predicted frame in the interpolated frame segment can be determined according to the interpolation method and global parameters.

[0238] In some embodiments, if the interpolation method is backward interpolation, the predicted frame can be inserted after the current frame.

[0239] In some embodiments, if the interpolation method is forward and backward interpolation, the predicted frame can be inserted between the current frame and the frames preceding the current frame. For example, the specific insertion position of the predicted frame can be determined based on global parameters, and this disclosure does not make any specific limitations.

[0240] In some embodiments, if the frame interpolation method is forward and backward frame interpolation, the predicted frame can be inserted between the current frame and the frames preceding the current frame. For example, the specific insertion position of the predicted frame can be determined based on the number of sampling interval frames, and this disclosure does not make any specific limitations.

[0241] The data processing method provided in the embodiments of this disclosure obtains frame-level parameters, determines whether the temporal sampling rate change flag is set based on the frame-level parameters, and performs frame interpolation based on the interpolation strategy corresponding to the previous frame of the current frame when the temporal sampling rate change flag is not set, thereby improving the efficiency of video encoding and decoding.

[0242] Figure 7 shows a flowchart of another data processing method according to an embodiment of this disclosure.

[0243] As shown in Figure 7, the method may include:

[0244] S710, determine the frame interpolation method and the number of remaining frames corresponding to the current frame. The current frame includes the last sampled frame.

[0245] In this embodiment of the disclosure, the current frame can be the last sampled frame in the interpolation segment.

[0246] In some embodiments, erd_num_temporal_remain can be the remaining frames, i.e., the number of images after the last sampled frame.

[0247] In some embodiments, the last sampled frame can be determined based on the number of sampled frames in the interpolation frame segment.

[0248] S720, when the frame interpolation method is backward frame interpolation, the backward frame interpolation yields a predicted frame, and the predicted frame retains the remaining frame count. In some embodiments, the prediction can be performed based on the current frame and the frames preceding the current frame as reference frames to obtain the predicted frame. In some embodiments, the remaining frame count can be determined based on the bitstream data; the method for determining the remaining frame count in this disclosure embodiment is not specifically limited.

[0249] For example, if there are 3 predicted frames and 2 remaining frames, then the 2 predicted frames can be retained as remaining frames.

[0250] S730, when the frame interpolation method is forward and backward frame interpolation, copies the current frame to obtain a copied frame, and retains the number of copied frames remaining.

[0251] In some embodiments, the current frame can be a sampled frame. Since the current frame is the last sampled frame, forward and backward prediction cannot be completed. Therefore, the current frame can be copied according to the number of remaining frames to obtain the final remaining frames.

[0252] For example, if the encoding end finishes sampling during a sampling interval, the actual number of frames within the last sampling interval needs to be transmitted in the bitstream. `erd_temporal_restoration_data()` is the header data following the last payload data in the V3C framework.

[0253] erd_num_temporal_remain:

[0254] This indicates the number of images after the last temporal sampling frame.

[0255] When srd_temporal_restoration_mode is 0, erd_num_temporal_remain ranges from [0, 2srd_temporal_resampling_ratio_idx + 1).

[0256] When srd_temporal_restoration_mode is 1, erd_num_temporal_remain ranges from [0, RestorationRatio);

[0257] The video prediction model is capable of predicting 2 consecutive frames out of 4 consecutive frames, where the TemporalRatio is 6 / 4. The encoder finishes sampling precisely during the sampling interval, which is 1 frame. `erd_temporal_restoration_data()` data needs to be sent after the final payload data, with `erd_num_temporal_remain` set to 1. Upon receiving the last decoded sampled subsequence, the decoder calls the `extrapolation` function to generate 2 predicted frames. When `erd_temporal_restoration_data()` data is received and `erd_num_temporal_remain` is found to be 1, only the first predicted frame is retained.

[0258] Specifically, when `srd_temporal_restoration_mode` is 1, the encoding end finishes sampling precisely when frame extraction ends. Therefore, `erd_temporal_restoration_data()` data needs to be sent after the final payload data, with `erd_num_temporal_remain` set to 0. When the decoding end receives the last decoded sampled subsequence, it calls the `extrapolation` function to generate two prediction frames. If `erd_temporal_restoration_data()` data is received and parsed to show `erd_num_temporal_remain` as 0, no prediction frames are displayed.

[0259] For example, `prd_temporal_resampling_ratio_changed_flag`: a flag indicating a change in the temporal sampling ratio.

[0260] 1: Changes have occurred

[0261] 0: No change

[0262] Any change in prd_temporal_resampling_ratio_idx, rd_temporal_extrapolation_flag, or prd_temporal_extra_resampling_length_idx is considered a change in the temporal sampling ratio.

[0263] prd_temporal_resampling_ratio_idx indicates a change in the sampling ratio.

[0264] prd_temporal_extrapolation_flag indicates that the sampling method has changed.

[0265] prd_temporal_extra_resampling_length_idx will cause changes in the number of frames after the frame skipping.

[0266] prd_temporal_extra_resampling_length_idx: Index of the frame-level temporal extrapolation sampling frame length

[0267] If prd_temporal_extra_resampling_length is represented by M, then

[0268] prd_temporal_extra_resampling_length_idx=M-2

[0269] When the mode is extrapolation, prd_temporal_resampling_ratio_idx is represented by i.

[0270] prd_temporal_resampling_ratio=(M+i) / M

[0271] When the mode is interpolation, prd_temporal_resampling_ratio_idx is represented by i.

[0272] prd_temporal_resampling_ratio=2i.

[0273] The data processing method provided in the embodiments of this disclosure obtains frame-level parameters, determines whether the temporal sampling rate change flag is set based on the frame-level parameters, and performs frame interpolation based on the interpolation strategy corresponding to the previous frame of the current frame when the temporal sampling rate change flag is not set, thereby improving the efficiency of video encoding and decoding.

[0274] Figure 8 shows a flowchart of another data processing method according to an embodiment of this disclosure.

[0275] As shown in Figure 8, the method may include:

[0276] S810, determine the number of predictions based on at least one of the number of sampled frames in the interpolated frame segment, the total number of frames in the interpolated frame segment, the number of sampling interval frames, and the prediction capability of the prediction model. The prediction capability of the prediction model includes the number of prediction frames obtained by the prediction model each time it makes a prediction.

[0277] S820, input the sampled frame into the reference frame sequence;

[0278] S830, determine whether the number of sampled frames in the reference frame sequence has reached a preset threshold;

[0279] S840: When the number of sampled frames in the reference frame sequence reaches a preset threshold, the predicted frame is obtained.

[0280] S850, repeat the above prediction steps until the preset number of times is reached.

[0281] In this embodiment, all reference frames in the reference frame sequence are sampling frames.

[0282] In some embodiments, the reference frame sequence is updated; the predicted frame is obtained based on the updated reference frame sequence.

[0283] Repeat the step of obtaining the predicted frame based on the updated reference frame sequence until the preset number of times is reached.

[0284] In some embodiments, the reference frame sequence can be updated based on the predicted frame.

[0285] In some embodiments, the method for determining the preset threshold can be based on the method for determining the preset threshold in the above embodiments, and is not specifically limited in this disclosure.

[0286] In some embodiments, after the number of repetitions reaches a preset number, all sampled frames and unsampled frames in the supplementary frame segment can be obtained.

[0287] For example, based on the sampling interval frame number RestorationRatio and the number of frames n predicted by the video prediction model, the video prediction module extrapolation(refList, RestorationRatio) is used, where refList is the image of the last m frames in the sampled and predicted frames. When RestorationRatio is 0, the module directly returns an empty list.

[0288] The reference frame must contain the sample frame.

[0289] Initially i = 0, execute the video prediction model len([0, floor(m / / n))) times:

[0290] The predicted frames returned by extrapolation are added to the prediction sequence queue preList;

[0291] Determine the length of preList and the size of RestorationRatio. If the length of preList is greater than or equal to RestorationRatio, return the last RestorationRatio frame of the predicted sequence.

[0292] When i == floor(m / / n), the extrapolation function returns (N - preList.size()) * preList[-1].

[0293] The data processing method provided in the embodiments of this disclosure obtains frame-level parameters, determines whether the temporal sampling rate change flag is set based on the frame-level parameters, and performs frame interpolation based on the interpolation strategy corresponding to the previous frame of the current frame when the temporal sampling rate change flag is not set, thereby improving the efficiency of video encoding and decoding.

[0294] Figure 9 shows a flowchart of another data processing method according to an embodiment of this disclosure.

[0295] As shown in Figure 9, the method may include:

[0296] S910, input the sampled frame and the predicted frame into the reference frame sequence;

[0297] In some embodiments, the reference frame may include a sampled frame and / or a predicted frame.

[0298] S920, determine whether the number of sampled frames in the reference frame sequence has reached a preset threshold.

[0299] In some embodiments, the method for determining the preset threshold has been described in detail in the above embodiments and will not be repeated here.

[0300] S930, when the number of sampled frames and predicted frames in the reference frame sequence reaches a preset threshold, a predicted frame is obtained;

[0301] S940, repeat the above steps until the number of sampled frames and predicted frames reaches the total number of frames in the interpolation segment.

[0302] In some embodiments, the method further includes:

[0303] Update the reference frame sequence;

[0304] The predicted frame is obtained based on the updated reference frame sequence;

[0305] Repeat the step of obtaining predicted frames based on the updated reference frame sequence until the number of sampled frames and predicted frames reaches the total number of frames in the interpolation segment.

[0306] In some embodiments, updating the reference frame sequence includes updating the reference frame sequence based on subsequent sampled frames and / or generated predicted frames.

[0307] In some embodiments, updating the reference frame sequence includes updating the reference sequence based solely on the generated prediction frames.

[0308] For example, if the preset threshold is 2 and the number of sampled frames is 2, then 1 prediction frame will be obtained based on 2 sampled frames. Then, based on the 1 prediction frame and 1 sampled frame, the next prediction frame will be obtained. Then, based on 2 prediction frames, the next prediction frame will be obtained. Then, the prediction will continue to roll to obtain prediction frames of length N. The total number of frames in the supplementary frame segment is N+2.

[0309] The data processing method provided in the embodiments of this disclosure obtains frame-level parameters, determines whether the temporal sampling rate change flag is set based on the frame-level parameters, and performs frame interpolation based on the interpolation strategy corresponding to the previous frame of the current frame when the temporal sampling rate change flag is not set, thereby improving the efficiency of video encoding and decoding.

[0310] To provide a detailed description of this application, this disclosure provides an exemplary embodiment of a data processing method.

[0311] Determine whether to perform temporal reconstruction. Setting srd_temporal_restoration_flag=1 indicates that temporal reconstruction is enabled, while setting srd_temporal_restoration_flag=0 indicates that the temporal reconstruction module is disabled.

[0312] Given a temporal reconstruction, the frame interpolation method is determined. Here, srd_temporal_restoration_mode=0 indicates that forward and backward frame interpolation is used, and srd_temporal_restoration_mode=1 indicates that backward frame interpolation is used.

[0313] When using forward and backward frame interpolation, i.e., srd_temporal_restoration_mode=0, the number of forward and backward frame interpolations is determined, where the number of frame interpolations can be determined by the frame interpolation ratio;

[0314] The frame interpolation ratio is Temporal Interpolation Ratio.

[0315] The frame interpolation ratio can be determined by the following formula:

[0316] TemporalInterpolationRatio=2^(srd_temporal_interpolation_ratio_idx+1)

[0317] in,

[0318] When using forward and backward interpolation, the value of srd_temporal_interpolation_ratio_idx ranges from 0, 1, and 2.

[0319] When using backward interpolation, the number of sampled frames can be initialized as follows: TemporalExtrapolationResampleNum is initialized as follows:

[0320] TemporalExtrapolationResampleNum=

[0321] srd_temporal_extrapolation_resample_num_idx+2.

[0322] When using backward framing, the prediction frame can be determined, where the prediction frame can be TemporalExtrapolationPredictNum.

[0323] TemporalExtrapolationPredictNum = srd_temporal_extrapolation_predict_num_idx + 1. Based on the same inventive concept, this disclosure also provides another data processing method, as shown in the following embodiment. Since the principle of solving the problem in this method embodiment is similar to that of the above method embodiment, the implementation of this method embodiment can refer to the implementation of the above method embodiment, and repeated details will not be described again.

[0324] In some embodiments, this disclosure also discloses a data processing method applied to the encoding end.

[0325] The data processing method may include: encoding the video to obtain bitstream data, so that the decoding end can implement any of the data processing methods in the above embodiments based on the bitstream data.

[0326] Figure 10 shows a structural diagram of a data processing device according to an embodiment of this disclosure. It should be noted that this device is applied to the decoding end.

[0327] As shown in Figure 10, the device 1000 may include:

[0328] The first acquisition module 1010 is used to acquire frame-level parameters;

[0329] The first determining module 1020 is used to determine whether the temporal sampling rate change flag bit is set according to the frame-level parameters;

[0330] The first frame interpolation module 1030 is used to perform frame interpolation based on the frame interpolation strategy corresponding to the previous frame of the current frame when the time domain sampling rate change flag is not set.

[0331] In one embodiment of this disclosure, the temporal sampling rate change flag position includes at least one of sampling rate change, frame interpolation method change, and sampling interval frame number change.

[0332] In one embodiment of this disclosure, the apparatus further includes:

[0333] The second frame interpolation module is used to perform frame interpolation based on the changed sampling rate, the changed frame interpolation method, and / or the changed number of sampling interval frames when the time-domain sampling rate change flag is in position.

[0334] In one embodiment of this disclosure, the second frame interpolation module includes:

[0335] The first acquisition unit is used to acquire historical frame interpolation methods;

[0336] The first interpolation unit is used to perform forward and backward interpolation and backward interpolation based on the current frame when the historical interpolation method is forward and backward interpolation.

[0337] In one embodiment of this disclosure, the first frame interpolation module includes:

[0338] The first input unit is used to input the current frame into the reference frame sequence;

[0339] The first determining unit is used to determine whether the number of frames in the reference frame sequence reaches a preset threshold, wherein the preset threshold is determined by the number of frames in the sampling interval;

[0340] The second determining unit is used to determine the prediction frame based on the reference frames in the reference frame sequence, the number of sampling interval frames, and the prediction module when the number of frames in the reference frame sequence reaches a preset threshold.

[0341] In one embodiment of this disclosure, the first frame interpolation module includes:

[0342] The second acquisition unit is used to acquire the current frame and the reference frame sequence;

[0343] The third determining unit is used to determine whether the number of frames in the reference frame sequence reaches a preset threshold, wherein the preset threshold is determined by the number of frames in the sampling interval;

[0344] The fourth determining unit is used to determine the prediction frame based on the current frame, the reference frames in the reference frame sequence, the number of sampling interval frames, and the prediction module when the number of frames in the reference frame sequence reaches a preset threshold.

[0345] In one embodiment of this disclosure, the apparatus further includes:

[0346] The second input module is used to input the current frame into the reference frame sequence after the predicted frame is determined.

[0347] In one embodiment of this disclosure, the apparatus further includes:

[0348] The second acquisition module is used to acquire the current frame;

[0349] The first acquisition module includes:

[0350] The third acquisition unit, when the current frame is acquired, is used to acquire the frame temporal information corresponding to the current frame, which includes frame-level parameters.

[0351] In one embodiment of this disclosure, the apparatus further includes:

[0352] The second determining module is used to determine the frame interpolation method and the number of remaining frames corresponding to the current frame, where the current frame includes the last sampled frame.

[0353] The first retention module is used to obtain the predicted frame by backward interpolation when the interpolation method is backward interpolation, and to retain the predicted frame of the remaining frame number.

[0354] The first copying module is used to copy the current frame when the frame interpolation method is forward and backward frame interpolation, to obtain a copied frame, and to retain the number of copied frames remaining.

[0355] In one embodiment of this disclosure, the current frame includes a sampled frame and a predicted frame.

[0356] In one embodiment of this disclosure, the apparatus further includes:

[0357] The third determining module is used to determine the number of predictions based on at least one of the following: the number of sampled frames in the interpolated frame segment, the total number of frames in the interpolated frame segment, the number of sampling interval frames, and the prediction capability of the prediction model. The prediction capability of the prediction model includes the number of prediction frames obtained by the prediction model each time it makes a prediction.

[0358] The third input module is used to input the sampled frame into the reference frame sequence;

[0359] Determine whether the number of sampled frames within the reference frame sequence has reached a preset threshold;

[0360] The fourth determining module is used to obtain the predicted frame when the number of sampled frames in the reference frame sequence reaches a preset threshold.

[0361] The first repeating module is used to repeat the above steps until the preset number of times is reached.

[0362] In one embodiment of this disclosure, the apparatus further includes:

[0363] The fourth input module is used to input the sampled frame and the predicted frame into the reference frame sequence;

[0364] The fifth determining module is used to determine whether the number of sampled frames in the reference frame sequence has reached a preset threshold.

[0365] The sixth determining module is used to obtain the predicted frame when the number of sampled frames and predicted frames in the reference frame sequence reaches a preset threshold.

[0366] The second repeating module is used to repeat the above steps until the number of sampled frames and predicted frames reaches the total number of frames in the interpolation segment.

[0367] In one embodiment of this disclosure, the first frame interpolation module includes:

[0368] The third acquisition unit is used to acquire global parameters when the time-domain sampling rate change flag is not set;

[0369] The fourth frame interpolation unit is used to perform frame interpolation based on at least one of the following: the sampling rate, the frame interpolation method, and the number of sampling interval frames corresponding to the global parameters.

[0370] In one embodiment of this disclosure, the apparatus further includes:

[0371] The update module is used to update global parameters based on the changed sampling rate, the changed frame interpolation method, and / or the changed sampling interval frame number when the time-domain sampling rate change flag is set.

[0372] The data processing apparatus provided in the embodiments of this disclosure acquires frame-level parameters, determines whether the temporal sampling rate change flag is set based on the frame-level parameters, and performs frame interpolation based on the interpolation strategy corresponding to the previous frame of the current frame when the temporal sampling rate change flag is not set, thereby improving the efficiency of video encoding and decoding.

[0373] Figure 11 shows a structural diagram of a data processing device according to an embodiment of this disclosure. It should be noted that this device is applied to the decoding end.

[0374] As shown in Figure 11, the device 1100 may include:

[0375] The encoding module 1110 is used to encode the video to obtain bitstream data, so that the decoding end can implement any of the data processing methods described above based on the bitstream data.

[0376] The data processing apparatus provided in the embodiments of this disclosure acquires frame-level parameters, determines whether the temporal sampling rate change flag is set based on the frame-level parameters, and performs frame interpolation based on the interpolation strategy corresponding to the previous frame of the current frame when the temporal sampling rate change flag is not set, thereby improving the efficiency of video encoding and decoding.

[0377] The data processing apparatus provided in this disclosure can be used to execute the positioning methods provided in the above-described method embodiments. The implementation principle and technical effect are similar, and for the sake of simplicity, they will not be described in detail here.

[0378] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0379] The electronic device 1200 according to this embodiment of the present disclosure will now be described with reference to FIG12. The electronic device 1200 shown in FIG12 is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present disclosure.

[0380] As shown in Figure 12, the electronic device 1200 is presented in the form of a general-purpose computing device. The components of the electronic device 1200 may include, but are not limited to: at least one processing unit 1212, at least one storage unit 1220, and a bus 1230 connecting different system components (including storage unit 1220 and processing unit 1212).

[0381] The storage unit stores program code, which can be executed by the processing unit 1212 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 1212 can perform the following steps of the above-described method embodiments:

[0382] Get frame-level parameters;

[0383] Determine whether to set the temporal sampling rate change flag based on frame-level parameters;

[0384] If the temporal sampling rate change flag is not set, frame interpolation is performed based on the interpolation strategy corresponding to the previous frame of the current frame.

[0385] Storage unit 1220 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 12201 and / or cache memory 12202, and may further include a read-only memory (ROM) 12203.

[0386] Storage unit 1220 may also include a program / utility 12204 having a set (at least one) of program modules 12205, such program modules 12205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0387] Bus 1230 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0388] Electronic device 1200 can also communicate with one or more external devices 1240 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1200, and / or any device that enables electronic device 1200 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1250. Furthermore, electronic device 1200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1260. As shown, network adapter 1260 communicates with other modules of electronic device 1200 via bus 1230. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0389] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0390] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the methods described above is stored thereon. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code, which, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0391] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0392] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein readable program code is carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0393] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0394] In practical implementation, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0395] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0396] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0397] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0398] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A data processing method, characterized in that, Applied to the decoding end, it includes: acquiring frame-level parameters; determining whether the temporal sampling rate change flag is set according to the frame-level parameters; and performing frame interpolation based on the interpolation strategy corresponding to the previous frame when the temporal sampling rate change flag is not set.

2. The method according to claim 1, characterized in that, The time-domain sampling rate change flag position includes at least one of the following: sampling rate change, frame interpolation method change, and sampling interval frame number change.

3. The method according to claim 2, characterized in that, The sampling interval frame number includes the number of sampled frames and the number of predicted frames, and the sampling multiplier includes the sampling multiplier when performing forward and backward frame interpolation.

4. The method according to claim 3, characterized in that, The method further includes: when the time-domain sampling rate change flag is in position, performing frame interpolation based on the changed sampling rate, the changed interpolation method, and / or the changed number of sampling frames.

5. The method according to claim 4, characterized in that, The frame interpolation based on the changed interpolation method includes: obtaining the historical interpolation method; and, if the historical interpolation method is forward and backward interpolation, performing forward and backward interpolation and backward interpolation based on the current frame.

6. The method according to claim 1, characterized in that, When the temporal sampling rate change flag is not set, the frame interpolation based on the interpolation strategy corresponding to the previous frame of the current frame includes: inputting the current frame into a reference frame sequence; determining whether the number of frames in the reference frame sequence reaches a preset threshold, wherein the preset threshold is determined by the number of sampled frames; and when the number of frames in the reference frame sequence reaches the preset threshold, determining the prediction frame based on the reference frames in the reference frame sequence, the number of prediction frames, and the prediction module.

7. The method according to claim 1, characterized in that, When the temporal sampling rate change flag is not set, the frame interpolation is performed based on the interpolation strategy corresponding to the previous frame of the current frame, including: acquiring the current frame and the reference frame sequence; determining whether the number of frames in the reference frame sequence reaches a preset threshold, wherein the preset threshold is determined by the number of sampled frames; and when the number of frames in the reference frame sequence reaches the preset threshold, determining the prediction frame based on the current frame, the reference frames in the reference frame sequence, the number of prediction frames, and the prediction module.

8. The method according to claim 7, characterized in that, The method further includes: after determining the predicted frame, inputting the current frame into a reference frame sequence.

9. The method according to claim 1, characterized in that, The method further includes: acquiring the current frame; the acquisition of frame-level parameters includes: when the current frame is acquired, acquiring the frame temporal information corresponding to the current frame, the frame temporal information including frame-level parameters.

10. The method according to claim 9, characterized in that, The method further includes: determining the frame interpolation method corresponding to the current frame and the number of remaining frames, wherein the current frame includes the last sampled frame; when the frame interpolation method is backward frame interpolation, obtaining a predicted frame through backward frame interpolation, and retaining the number of predicted frames in the remaining frame count; when the frame interpolation method is forward and backward frame interpolation, copying based on the current frame to obtain a copied frame, and retaining the number of copied frames in the remaining frame count.

11. The method according to claim 9, characterized in that, The current frame includes the sampled frame and the predicted frame.

12. The method according to claim 11, characterized in that, The method further includes: determining the number of prediction attempts based on at least one of the number of sampled frames in the interpolated frame segment, the total number of frames in the interpolated frame segment, the number of predicted frames, and the prediction capability of the prediction model, wherein the prediction capability of the prediction model includes the number of predicted frames obtained by the prediction model each time; inputting the sampled frames into a reference frame sequence; determining whether the number of sampled frames in the reference frame sequence reaches a preset threshold; if the number of sampled frames in the reference frame sequence reaches the preset threshold, obtaining a predicted frame; updating the reference frame sequence; obtaining a predicted frame based on the updated reference frame sequence; and repeating the step of obtaining a predicted frame based on the updated reference frame sequence until the preset number of prediction attempts is reached.

13. The method according to claim 11, characterized in that, The method further includes: inputting the sampled frame and the predicted frame into a reference frame sequence; determining whether the number of sampled frames in the reference frame sequence reaches a preset threshold; if the number of sampled frames and the predicted frame in the reference frame sequence reaches the preset threshold, obtaining a predicted frame; updating the reference frame sequence; obtaining a predicted frame based on the updated reference frame sequence; repeating the step of obtaining a predicted frame based on the updated reference frame sequence until the number of sampled frames and the predicted frame reaches the total number of frames in the interpolation segment.

14. The method according to claim 1, characterized in that, When the temporal sampling rate change flag is not set, performing frame interpolation based on the interpolation strategy corresponding to the previous frame of the current frame includes: when the temporal sampling rate change flag is not set, obtaining global parameters; and performing frame interpolation based on at least one of the sampling rate, interpolation method, and sampling interval frame number corresponding to the global parameters.

15. The method according to claim 1, characterized in that, The method further includes: when the time-domain sampling rate change flag is in position, updating global parameters based on the changed sampling rate, the changed frame interpolation method, and / or the changed sampling interval frame number.

16. A data processing method, characterized in that, Applied to the encoding end, it includes: encoding video to obtain bitstream data, so that the decoding end can implement any one of the data processing methods as described in claims 1-15 based on the bitstream data.

17. A data processing apparatus, characterized in that, Applied to the decoding end, it includes: a first acquisition module for acquiring frame-level parameters; a first determination module for determining whether the temporal sampling rate change flag is set according to the frame-level parameters; and a first frame interpolation module for performing frame interpolation based on the frame interpolation strategy corresponding to the previous frame of the current frame when the temporal sampling rate change flag is not set.

18. A data processing apparatus, characterized in that, Applied to the encoding end, it includes: an encoding module for encoding video to obtain bitstream data, so that the decoding end can implement any one of the data processing methods as described in claims 1-16 based on the bitstream data.

19. An electronic device, characterized in that, include: processor; And a memory for storing executable instructions of the processor; wherein the processor is configured to perform the data processing method of any one of claims 1 to 16 by executing the executable instructions.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data processing method according to any one of claims 1 to 16.

Images