Virtual reference frame generation method, apparatus and device, and readable storage medium

Abstract

The invention provides a virtual reference frame generation method, device and equipment and a readable storage medium, and the method comprises the steps: querying a first frame and a second frame in a reference frame list, and enabling the time domain distance between the first frame and the second frame to be equal to the time domain distance between the second frame and a current coding frame, the first frame and the second frame are located before the current coding frame; when the first frame and the second frame are queried, generating a first prediction reference frame according to the first frame and the second frame; under the condition that the first frame and the second frame are not queried, generating a second prediction reference frame according to a third frame and a fourth frame; wherein the third frame and the fourth frame are frames in the reference frame list, and the time domain distance between the third frame and the current coding frame is the smallest; and taking the first prediction reference frame or the second prediction reference frame as a virtual reference frame. Therefore, the scheme can improve the coding performance.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of video coding, and in particular, to a method, apparatus, device, and readable storage medium for generating a virtual reference frame. Background Art

[0002] In the video encoding and decoding process, especially the inter-frame prediction stage of video encoding, virtual reference frames generated based on deep learning can provide higher quality reference for the current frame to be encoded compared to traditional video reference frames.

[0003] However, for the virtual reference frame generation method based on deep learning, how to ensure higher quality virtual reference frames under low delay P frame configuration to improve encoding performance has become an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present application provide a virtual reference frame generation method, apparatus, device and readable storage medium to generate a higher quality virtual reference frame, thereby achieving the purpose of improving encoding performance.

[0005] In order to solve the above problems, this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for generating a virtual reference frame, comprising:

[0007] Searching for a first frame and a second frame in a reference frame list, wherein a temporal distance between the first frame and the second frame is equal to a temporal distance between the second frame and a current coded frame, and the first frame and the second frame are located before the current coded frame;

[0008] In the case where the first frame and the second frame are queried, generating a first prediction reference frame according to the first frame and the second frame;

[0009] If the first frame and the second frame are not found, generating a second prediction reference frame according to a third frame and a fourth frame; wherein the third frame and the fourth frame are frames in the reference frame list having the smallest time domain distance from the current coded frame;

[0010] The first prediction reference frame or the second prediction reference frame is used as a virtual reference frame.

[0011] Optionally, generating a first prediction reference frame according to the first frame and the second frame includes:

[0012] Inputting the first frame and the second frame into a first generation network;

[0013] The output of the first generation network is used as the first prediction reference frame.

[0014] Optionally, the first generation network includes a plurality of encoding units, a plurality of decoding units, and a feature fusion unit;

[0015] The plurality of encoding units are connected in sequence, the plurality of decoding units are connected in sequence, and the plurality of encoding units and the plurality of decoding units are connected in one-to-one correspondence;

[0016] The feature fusion unit is connected to the multiple encoding units and the multiple decoding units respectively, and the feature fusion unit is used to perform feature fusion on the encoding information output by the multiple encoding units;

[0017] The multiple encoding units support different resolutions, and each encoding unit is used to perform downsampling and feature extraction according to the resolution supported by each encoding unit.

[0018] Optionally, the feature fusion unit is further used to perform upsampling and / or downsampling processing on the acquired encoding information based on a resolution supported by the target decoding unit before performing feature fusion.

[0019] Optionally, the method further comprises:

[0020] Obtaining a key frame corresponding to the current encoding frame;

[0021] The key frame is used as a virtual reference frame.

[0022] In a second aspect, an embodiment of the present application provides a virtual reference frame generation device, including:

[0023] A first processing module, configured to search a first frame and a second frame in a reference frame list, wherein a temporal distance between the first frame and the second frame is equal to a temporal distance between the second frame and a current coded frame, and the first frame and the second frame are located before the current coded frame;

[0024] A second processing module, configured to generate a first prediction reference frame according to the first frame and the second frame when the first frame and the second frame are queried;

[0025] A third processing module is configured to generate a second prediction reference frame according to a third frame and a fourth frame when the first frame and the second frame are not found; wherein the third frame and the fourth frame are frames in the reference frame list having the smallest time domain distance from the current coded frame;

[0026] The fourth processing module is configured to use the first prediction reference frame or the second prediction reference frame as a virtual reference frame.

[0027] Optionally, the second processing module is further used for:

[0028] Inputting the first frame and the second frame into a first generation network;

[0029] The output of the first generation network is used as the first prediction reference frame.

[0030] Optionally, the first generation network includes a plurality of encoding units, a plurality of decoding units, and a feature fusion unit;

[0031] The plurality of encoding units are connected in sequence, the plurality of decoding units are connected in sequence, and the plurality of encoding units and the plurality of decoding units are connected in one-to-one correspondence;

[0032] The feature fusion unit is connected to the multiple encoding units and the multiple decoding units respectively, and the feature fusion unit is used to perform feature fusion on the encoding information output by the multiple encoding units;

[0033] The multiple encoding units support different resolutions, and each encoding unit is used to perform downsampling and feature extraction according to the resolution supported by each encoding unit.

[0034] Optionally, the feature fusion unit is further used to perform upsampling and / or downsampling processing on the acquired encoding information based on a resolution supported by the target decoding unit before performing feature fusion.

[0035] Optionally, the device further comprises:

[0036] An acquisition module, used for acquiring a key frame corresponding to the current encoding frame;

[0037] The fifth processing module is used to use the key frame as a virtual reference frame.

[0038] In a third aspect, an embodiment of the present application provides a virtual reference frame generation device, comprising: a memory, a processor, and a program stored in the memory and executable on the processor; the processor is used to read the program in the memory to implement the steps in the virtual reference frame generation method as described above.

[0039] In a fourth aspect, an embodiment of the present application provides a readable storage medium for storing a program, which, when executed by a processor, implements the steps in the virtual reference frame generation method as described above.

[0040] In a fifth aspect, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the virtual reference frame generation method as described in the first aspect.

[0041] The method of the embodiment of the present application searches for the first frame and the second frame in the reference frame list, preferably generates a first predicted reference frame based on the first frame and the second frame as a virtual reference frame; and if the first frame and the second frame are not found, generates a second predicted reference frame based on the third frame and the fourth frame as a virtual reference frame. Since the time domain distance between the first frame and the second frame is equal to the time domain distance between the second frame and the current coded frame, in the inter-frame prediction link of video coding, the first predicted reference frame can provide a higher quality reference for the current coded frame than the second predicted reference frame, thereby improving the overall coding performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 A schematic diagram showing a flow chart of a method for generating a virtual reference frame according to an embodiment of the present application;

[0044] Figure 2 A schematic diagram showing the structure of a first generation network in an embodiment of the present application;

[0045] Figure 3 A schematic diagram showing the structure of an encoding unit in an embodiment of the present application;

[0046] Figure 4 A schematic diagram showing the structure of a decoding unit according to an embodiment of the present application;

[0047] Figure 5 A schematic diagram showing the structure of a feature fusion unit in an embodiment of the present application;

[0048] Figure 6 A schematic diagram showing the structure of a virtual reference frame generating device according to an embodiment of the present application;

[0049] Figure 7 A schematic diagram showing the structure of a virtual reference frame generating device according to an embodiment of the present application;

[0050] Figure 8 A schematic diagram of a reference frame list according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0051] In order to make the technical problems, technical solutions and advantages to be solved by the application clearer, the following will be described in detail in conjunction with the accompanying drawings and specific embodiments. In the following description, specific details such as specific configurations and components are provided only to help fully understand the embodiments of the application. Therefore, it should be clear to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the application. In addition, for clarity and brevity, the description of known functions and structures has been omitted.

[0052] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present application. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0053] In various embodiments of the present application, it should be understood that the order of execution of the following processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, the terms "system" and "network" are often used interchangeably in this article.

[0054] For ease of understanding, some contents involved in the embodiments of the present application are described below:

[0055] In the Low delay P configuration, there are more reference frames available in the Reference Frame List (RFL). If the two frames with the closest temporal distance to the current frame are selected for the generation of the virtual reference frame, the motion amplitude in the actual coded picture is ignored, resulting in a large texture difference between the generated virtual reference frame and the current frame to be coded.

[0056] The reference frames in the RFL are identified by the Picture Order Count (POC).

[0057] like Figure 1 As shown, an embodiment of the present application provides a method for generating a virtual reference frame, comprising:

[0058] Step 11, searching a first frame and a second frame in a reference frame list, wherein a time domain distance between the first frame and the second frame is equal to a time domain distance between the second frame and a current coded frame, and the first frame and the second frame are located before the current coded frame.

[0059] Here, the current coded frame can be understood as the current frame to be coded. The reference frame list is a configured list containing multiple reference frames, such as the reference frame list under the Low delay P configuration. Moreover, the temporal distance can be calculated by the POC of each frame. Therefore, for the first frame and the second frame before the current coded frame, since the temporal distance between the first frame and the second frame is equal to the temporal distance between the second frame and the current coded frame, the first frame, the second frame and the current coded frame form three frames with equally spaced △POC.

[0060] Step 12: when the first frame and the second frame are found, generate a first prediction reference frame according to the first frame and the second frame.

[0061] That is, after executing step 11, if a first frame and a second frame that meet the requirements are found in the reference frame list, these two frames will be used to generate a first prediction reference frame.

[0062] Step 13, if the first frame and the second frame are not found, generate a second prediction reference frame according to the third frame and the fourth frame; wherein the third frame and the fourth frame are the frames in the reference frame list with the smallest time domain distance from the current encoded frame.

[0063] That is, after executing step 11, if the first frame and the second frame that meet the requirements cannot be found in the reference frame list, the two frames with the smallest temporal distance to the current coded frame will be used to generate the second prediction reference frame. At this time, for the other frames in the reference frame list except the third frame and the fourth frame, the temporal distance between them and the current coded frame is less than or equal to d0, d0 is the minimum value between d1 and d2, d1 is the temporal distance between the third frame and the current coded frame, and d2 is the temporal distance between the fourth frame and the current coded frame.

[0064] Step 14: Use the first prediction reference frame or the second prediction reference frame as a virtual reference frame.

[0065] Thus, according to the above steps 11-14, the method of the embodiment of the present application will generate a first predicted reference frame as a virtual reference frame by querying the first frame and the second frame in the reference frame list, preferably based on the first frame and the second frame; and if the first frame and the second frame are not queried, then generate a second predicted reference frame as a virtual reference frame based on the third frame and the fourth frame. Among them, since the time domain distance between the first frame and the second frame is equal to the time domain distance between the second frame and the current coded frame, in the inter-frame prediction link of video coding, the first predicted reference frame can provide a higher quality reference for the current coded frame than the second predicted reference frame, thereby improving the overall coding performance.

[0066] It should be noted that the change in picture content between adjacent frames is small. Therefore, the motion amplitude and trend between the first frame and the second frame, and the motion amplitude and trend between the second frame and the current encoding frame are closer. Using the first predicted reference frame generated by the first frame and the second frame as a virtual reference frame can greatly increase the encoding performance gain.

[0067] It should be known that, in this embodiment, after the first predicted reference frame is used as a virtual reference frame, or the second predicted reference frame is used as a virtual reference frame, the virtual reference frame will be added to the reference frame list to obtain an updated reference array table. As an optional implementation, the virtual reference frame is added to the end of the original reference frame list.

[0068] Example 1, the current coding frame (POC: 8), the original reference frame list (RPL0) includes 4 frames, recorded as RPL0: [7 5 60]. Through query, it is determined that the POC of the first frame and the second frame are 6 and 7 respectively, and the first prediction reference frame mark POC is further generated as -1, then the updated RPL0 is: [7 5 6 0 -1].

[0069] Example 2, the current coding frame (POC: 24), the original reference frame list (RPL0) includes 4 frames, recorded as RPL0: [23 168 0]. Through query, it is determined that the POC of the first frame and the second frame are 8 and 16 respectively, and the first prediction reference frame mark POC is further generated as -1, then the updated RPL0 is: [23 16 8 0 -1].

[0070] Example 3, the current coding frame (POC: 25), the original reference frame list (RPL0) includes 4 frames, recorded as RPL0: [24 168 0]. When the first and second frames that meet the conditions are not found, and the third and fourth frames are determined to have POCs of 16 and 24 respectively, the second prediction reference frame mark POC is further generated to be -1, then the updated RPL0 is: [24 16 8 0 -1].

[0071] In addition, in this embodiment, if multiple groups of first frames and second frames are found, a group of first frames and second frames with the smallest time domain distance is selected to generate the first prediction reference frame.

[0072] Optionally, in this embodiment, generating a first prediction reference frame according to the first frame and the second frame includes:

[0073] Inputting the first frame and the second frame into a first generation network;

[0074] The output of the first generation network is used as the first prediction reference frame.

[0075] The first generation network is pre-trained to take the first frame and the second frame as input and the first prediction reference frame as output. Thus, the first frame and the second frame obtained by query are input into the first generation network to obtain the first prediction reference frame.

[0076] Optionally, in this embodiment, before inputting the first frame and the second frame into the first generation network, the method further includes:

[0077] The preset network is trained based on the first training set to obtain the first generation network; wherein the first training set includes multiple groups of frames, and the time domain distance between two adjacent frames in each group of frames is the same.

[0078] As an implementation mode, each group of frames in the first training set includes four frames, wherein frame 1, frame 2, and frame 3 are adjacent frames, the temporal distance between frame 1 and frame 2 is equal to the temporal distance between frame 2 and frame 3, and frame 4 is a prediction reference frame of frame 3. The preset network is trained using the first training set, so that when the first frame and the second frame are input to the first generation network obtained by the training, the first prediction reference frame outputted is closer to the picture content of the current coding frame, thereby providing a higher quality reference for the current coding frame in the inter-frame prediction link of video coding, thereby achieving the purpose of improving the overall coding performance.

[0079] In addition, during the training process, adjacent frames can be divided into multiple training sets according to different time domain distances to train multiple first generation networks. When the time domain distances between the first frame, the second frame and the current coded frame are different, the first generation network applicable to the time domain distance is used to generate the first prediction reference frame; or, multiple sets of parameters of a first generation network are obtained by training. When the time domain distances between the first frame, the second frame and the current coded frame are different, the first generation network calls a set of parameters applicable to the time domain distance to generate the first prediction reference frame.

[0080] Optionally, in this embodiment, generating the second prediction reference frame according to the third frame and the fourth frame includes:

[0081] Inputting the third frame and the fourth frame into a second generation network;

[0082] The output of the second generation network is the second predicted reference frame.

[0083] The second generation network is pre-trained, and takes the third frame and the fourth frame as input and the second prediction reference frame as output. Thus, in the case where the first frame and the second frame are not found, the third frame and the fourth frame are input into the second generation network to obtain the second prediction reference frame.

[0084] Optionally, in this embodiment, before inputting the third frame and the fourth frame into the second generation network, the method further includes:

[0085] The preset network is trained based on the second training set to obtain the second generation network; wherein the second training set includes multiple groups of frames, and the time domain distance between the reference frame and the target frame in each group of frames is less than or equal to the specific time domain distance.

[0086] As an implementation method, each group of frames in the second training set includes four frames, wherein the time domain distance between frame 1', frame 2' and frame 3' is less than or equal to a specific time domain distance, the specific time domain distance is pre-configured or defined, and frame 4' is a prediction reference frame of frame 3'. The preset network is trained using the second training set, so that when the trained second generation network inputs the third frame and the fourth frame, the output second prediction reference frame provides a high-quality reference for the current coded frame in the inter-frame prediction link of video coding.

[0087] Optionally, in this embodiment, the first generation network includes a plurality of encoding units, a plurality of decoding units, and a feature fusion unit;

[0088] The plurality of encoding units are connected in sequence, the plurality of decoding units are connected in sequence, and the plurality of encoding units and the plurality of decoding units are connected in one-to-one correspondence;

[0089] The feature fusion unit is connected to the multiple encoding units and the multiple decoding units respectively, and the feature fusion unit is used to perform feature fusion on the encoding information output by the multiple encoding units;

[0090] The multiple encoding units support different resolutions, and each encoding unit is used to downsample and extract features according to the resolution supported by each encoding unit. In this way, the first generation network can adopt a multi-scale encoder-decoder network (Multi-Scale Encoder-Decoder Network, MSEDN), and the large-scale Encoder and Decoder are mainly beneficial for static content prediction, and the small-scale Encoder and Decoder are mainly beneficial for fine motion prediction. Among them, the resolution supported by the multiple encoding units decreases in the connection direction, and the multiple decoding units support the same resolution as the corresponding connected encoding units. In this embodiment, the structure of the second generation network and the preset network is the same as the first generation network.

[0091] For example, Figure 2As shown, the first generation network includes an encoder unit (Encoder) 1, an encoder unit 2, an encoder unit 3, an encoder unit 4, a decoder unit (Decoder) 1, a decoder unit 2, a decoder unit 3, a decoder unit 4 and a feature fusion unit. The resolutions supported by encoder units 1 to 4 decrease in sequence, that is, the resolution of the feature map F1 output by encoder unit 1 is greater than the resolution of the feature map F2 output by encoder unit 2, the resolution of F2 is greater than the resolution of the feature map F3 output by encoder unit 3, and the resolution of F3 is greater than the resolution of the feature map F4 output by encoder unit 4. Similarly, the resolutions supported by decoders 1 to 4 also decrease in sequence, and the corresponding encoder and decoder units support the same resolution. The structure of the encoder unit is as follows: Figure 3 As shown, the structure of the decoding unit is as follows Figure 4 shown.

[0092] Specifically, the network structure is input into the first frame (I0) and the second frame (I1). On the one hand, I0 and I1 are input into Encoder1 for downsampling feature extraction, and on the other hand, they are output with Decoder1 to obtain the optical flow obtained by feature splitting. The reconstructed image closest to the current encoded frame is obtained through warping operations, and then fused with the total feature D1 output by the Decoder to obtain I2.

[0093] Among them, on the Encoder side, I0 and I1 are first input into Encoder1 respectively (here it means that there is only one Encoder1, and I0 and I1 are input into reasoning separately), and two features f are obtained respectively. 1_0 and f 1_1 On the one hand, these two features are warped through the yellow line to the right and the bidirectional optical flow obtained by the output of the previous decoder Decoder2 to obtain a reconstruction image that is closer to the current encoded frame, which is then fused with the output D2 of Decoder2 and input to Decoder1; on the other hand, f 1_0 and f 1_1 The merged F1 is input into the next level Encoder 2.

[0094] In addition, at the decoder end, for example, Decoder4 receives the input of the F4 feature and the fused feature (blue line) obtained by cross-scale feature fusion. After passing through Decoder4, the reconstructed image D4 is obtained. After the same channel splitting operation as the encoder end, D4 is split into two optical flows. and Warp the features of the two frames of Encoder3 respectively.

[0095] Optionally, in this embodiment, the feature fusion unit is further used to perform upsampling and / or downsampling processing on the acquired encoding information based on a resolution supported by the target decoding unit before performing feature fusion.

[0096] The target decoding unit refers to the task object of the feature fusion unit. In this way, the feature fusion unit (Cross-scale Feature Fusion) will upsample and / or downsample the acquired encoding information (such as the above-mentioned feature map) based on the resolution supported by the target decoding unit before performing feature fusion. For example, Figure 5 The feature fusion unit shown, if the target decoding unit is encoding unit 3, will downsample F1 by 4 times, downsample F2 by 2 times, upsample F3 by 2 times, and then perform feature fusion (feature fusion through 1x1 convolution), so that the features output to encoding unit 3 support the same resolution as encoding unit 3.

[0097] In this embodiment, the feature fusion unit takes into account the information loss in the feature extraction and downsampling process, so the lost information is retained through learning, and this information is added in the prediction process (i.e., the process of gradually upsampling and restoring the original image scale). In addition, because it is cross-scale, it can also dynamically learn information that is beneficial to prediction at different scales, which can further improve the accuracy of the prediction results.

[0098] It should also be noted that, in this embodiment, the first generation network may also be referred to as a POC model, and the second generation network may also be referred to as a reference frame generation model (Closest model).

[0099] Optionally, in this embodiment, the method further includes:

[0100] Obtaining a key frame corresponding to the current encoding frame;

[0101] The key frame is used as a virtual reference frame.

[0102] Here, the key frame is the first frame in the video frame group (Group of Pictures, GoP). In this way, after obtaining the key frame corresponding to the current coding frame, the key frame can be added to the reference frame list as a virtual reference frame to improve the coding gain under the Low delay P coding configuration.

[0103] As an implementation method, Figure 8 As shown, the key frame and the predicted reference frame (the first predicted reference frame or the second predicted reference frame) can be added to the end of the reference frame list as virtual reference frames. Specifically, the key frame is located after the first predicted reference frame or the second predicted reference frame.

[0104] As an implementation method, only key frames may be added to the end of the reference frame list, for example, when the picture is mostly static content; or, only predicted reference frames (first predicted reference frames or second predicted reference frames) may be added to the end of the reference frame list, for example, when the picture is mostly violent motion content.

[0105] In summary, the method of the embodiment of the present application uses different networks to obtain virtual reference frames with different motion amplitudes and generation qualities, so that the virtual reference frame is more similar to the current coded frame in texture to provide a high-quality reference. Moreover, adding an additional key frame can provide a high-quality background and subject reference.

[0106] like Figure 6 As shown, the embodiment of the present application further provides a virtual reference frame generating device 60, comprising:

[0107] A first processing module 61 is configured to query a first frame and a second frame in a reference frame list, wherein a temporal distance between the first frame and the second frame is equal to a temporal distance between the second frame and a current coded frame;

[0108] A second processing module 62 is configured to generate a first prediction reference frame according to the first frame and the second frame when the first frame and the second frame are found, and the first frame and the second frame are located before the current coding frame;

[0109] A third processing module 63 is configured to generate a second prediction reference frame according to a third frame and a fourth frame when the first frame and the second frame are not found; wherein the third frame and the fourth frame are frames in the reference frame list with the smallest time domain distance from the current coded frame;

[0110] The fourth processing module 64 is configured to use the first prediction reference frame or the second prediction reference frame as a virtual reference frame.

[0111] Optionally, the second processing module is further used for:

[0112] Inputting the first frame and the second frame into a first generation network;

[0113] The output of the first generation network is used as the first prediction reference frame.

[0114] Optionally, the device further comprises:

[0115] The first training module is used to train a preset network based on a first training set to obtain the first generated network; wherein the first training set includes multiple groups of frames, and the time domain distance between two adjacent frames in each group of frames is the same.

[0116] Optionally, the third processing module is further used for:

[0117] Inputting the third frame and the fourth frame into a second generation network;

[0118] The output of the second generation network is the second predicted reference frame.

[0119] Optionally, the device further comprises:

[0120] The second training module is used to train the preset network based on the second training set to obtain the second generation network; wherein the second training set includes multiple groups of frames, and the time domain distance between the reference frame and the target frame in each group of frames is less than or equal to the specific time domain distance.

[0121] Optionally, the first generation network includes a plurality of encoding units, a plurality of decoding units, and a feature fusion unit;

[0122] The plurality of encoding units are connected in sequence, the plurality of decoding units are connected in sequence, and the plurality of encoding units and the plurality of decoding units are connected in one-to-one correspondence;

[0123] The feature fusion unit is connected to the multiple encoding units and the multiple decoding units respectively, and the feature fusion unit is used to perform feature fusion on the encoding information output by the multiple encoding units;

[0124] The multiple encoding units support different resolutions, and each encoding unit is used to perform downsampling and feature extraction according to the resolution supported by each encoding unit.

[0125] Optionally, the feature fusion unit is further used to perform upsampling and / or downsampling processing on the acquired encoding information based on a resolution supported by the target decoding unit before performing feature fusion.

[0126] Optionally, the device further comprises:

[0127] An acquisition module, used for acquiring a key frame corresponding to the current encoding frame;

[0128] The fifth processing module is used to use the key frame as a virtual reference frame.

[0129] The implementation principle and technical effect of the device in the embodiment of the present application are similar, and this embodiment will not be repeated here.

[0130] like Figure 7 As shown, an embodiment of the present application also provides a virtual reference frame generation device, including: a processor 701, a memory 702, and a computer program stored in the memory 702 and executable on the processor 701, and the processor 701 implements the steps of the above-mentioned virtual reference frame generation method when executing the computer program.

[0131] The virtual reference frame generation device provided in the embodiment of the present application can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated in this embodiment.

[0132] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by instructing relevant hardware through a computer program, wherein the computer program includes instructions for executing part or all of the steps of the above method; and the computer program may be stored in a readable storage medium, and the storage medium may be any form of storage medium.

[0133] In addition, the specific embodiment of the present application further provides a readable storage medium on which a program is stored, and when the program is executed by a processor, the steps in the above-mentioned virtual reference frame generation method are implemented. And the same technical effect can be achieved, so in order to avoid repetition, it will not be repeated here.

[0134] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 1 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0135] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0136] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0137] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the sending and receiving methods described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0138] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary personnel in this technical field, several improvements and modifications can be made without departing from the principles described in the present application. These improvements and modifications are also within the scope of protection of the present application.

Claims

1. A method for generating a virtual reference frame, characterized in that: include: Searching for a first frame and a second frame in a reference frame list, wherein a temporal distance between the first frame and the second frame is equal to a temporal distance between the second frame and a current coded frame, and the first frame and the second frame are located before the current coded frame; In the case where the first frame and the second frame are queried, generating a first prediction reference frame according to the first frame and the second frame; If the first frame and the second frame are not found, generating a second prediction reference frame according to a third frame and a fourth frame; wherein the third frame and the fourth frame are frames in the reference frame list having the smallest time domain distance from the current coded frame; The first prediction reference frame or the second prediction reference frame is used as a virtual reference frame.

2. The method according to claim 1, characterized in that The step of generating a first prediction reference frame according to the first frame and the second frame comprises: Inputting the first frame and the second frame into a first generation network; The output of the first generation network is used as the first prediction reference frame.

3. The method according to claim 2, characterized in that The first generation network includes a plurality of encoding units, a plurality of decoding units, and a feature fusion unit; The plurality of encoding units are connected in sequence, the plurality of decoding units are connected in sequence, and the plurality of encoding units and the plurality of decoding units are connected in one-to-one correspondence; The feature fusion unit is connected to the multiple encoding units and the multiple decoding units respectively, and the feature fusion unit is used to perform feature fusion on the encoding information output by the multiple encoding units; The multiple encoding units support different resolutions, and each encoding unit is used to perform downsampling and feature extraction according to the resolution supported by each encoding unit.

4. The method according to claim 3, characterized in that The feature fusion unit is further used to perform upsampling and / or downsampling processing on the acquired coding information based on the resolution supported by the target decoding unit before performing feature fusion.

5. The method according to claim 1, characterized in that Also includes: Obtaining a key frame corresponding to the current encoding frame; The key frame is used as a virtual reference frame.

6. A virtual reference frame generating device, characterized in that: include: A first processing module, configured to search a first frame and a second frame in a reference frame list, wherein a temporal distance between the first frame and the second frame is equal to a temporal distance between the second frame and a current coded frame, and the first frame and the second frame are located before the current coded frame; A second processing module, configured to generate a first prediction reference frame according to the first frame and the second frame when the first frame and the second frame are queried; A third processing module is configured to generate a second prediction reference frame according to a third frame and a fourth frame when the first frame and the second frame are not found; wherein the third frame and the fourth frame are frames in the reference frame list having the smallest time domain distance from the current coded frame; The fourth processing module is configured to use the first prediction reference frame or the second prediction reference frame as a virtual reference frame.

7. The device according to claim 6, characterized in that The second processing module is also used for: Inputting the first frame and the second frame into a first generation network; The output of the first generation network is used as the first prediction reference frame.

8. The device according to claim 6, characterized in that Also includes: An acquisition module, used for acquiring a key frame corresponding to the current encoding frame; The fifth processing module is used to use the key frame as a virtual reference frame.

9. A virtual reference frame generating device, comprising: A memory, a processor, and a program stored in the memory and executable on the processor; characterized in that: The processor is used to read the program in the memory to implement the steps in the virtual reference frame generation method according to any one of claims 1 to 5.

10. A readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps in the virtual reference frame generation method according to any one of claims 1 to 5 are implemented.

Images