Virtual reference frame generation method, apparatus and device, and readable storage medium

Abstract

Provided are a virtual reference frame generation method, apparatus and device, and a readable storage medium. The virtual reference frame generation method comprises: querying a first frame and a second frame from a reference frame list, the time domain distance between the first frame and the second frame being equal to the time domain distance between the second frame and a current frame to be encoded, and the first frame and the second frame being located before said current frame (11); when the first frame and the second frame are found, generating a first prediction reference frame on the basis of the first frame and the second frame (12); when the first frame and the second frame are not found, generating a second prediction reference frame on the basis of a third frame and a fourth frame, wherein the third frame and the fourth frame are frames in the reference frame list having the smallest time domain distance from said current frame (13); and using the first prediction reference frame or the second prediction reference frame as a virtual reference frame (14).

Description

Virtual reference frame generation method, device and equipment and readable storage medium

[0001] Cross-reference to related applications

[0002] The present disclosure claims priority to Chinese Patent Application No. 202411301229.9, filed on September 18, 2024 in China, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] Embodiments of the present disclosure relate to the field of video coding, in particular to a virtual reference frame generation method, device, equipment and readable storage medium. BACKGROUND

[0004] In the video coding process, especially in the inter prediction link of video coding, the virtual reference frame generated based on deep learning can provide higher quality reference for the current frame to be coded compared to the traditional video reference frame.

[0005] However, for the virtual reference frame generation method based on deep learning, how to ensure that a higher quality virtual reference frame is obtained under a low delay P frame configuration to improve the coding performance has become a problem to be solved. SUMMARY

[0006] Embodiments of the present disclosure provide a virtual reference frame generation method, device, equipment and readable storage medium to generate a higher quality virtual reference frame and achieve the purpose of improving the coding performance.

[0007] To solve the above problems, the present disclosure is implemented as follows:

[0008] In a first aspect, embodiments of the present disclosure provide a virtual reference frame generation method, comprising:

[0009] querying a first frame and a second frame in a reference frame list, wherein the time domain distance between the first frame and the second frame is equal to the time domain distance between the second frame and a current coding frame, and the first frame and the second frame are located before the current coding frame;

[0010] generating a first prediction reference frame according to the first frame and the second frame in a case where the first frame and the second frame are queried;

[0011] generating a second prediction reference frame according to a third frame and a fourth frame in a case where the first frame and the second frame are not queried, wherein the third frame and the fourth frame are frames in the reference frame list with the smallest time domain distance from the current coding frame;

[0012] using the first prediction reference frame or the second prediction reference frame as a virtual reference frame.

[0013] Optionally, the generating the first prediction reference frame according to the first frame and the second frame comprises:

[0014] inputting the first frame and the second frame into a first generation network;

[0015] outputting the first generation network as the first prediction reference frame.

[0016] Optionally, the first generation network comprises a plurality of encoding units, a plurality of decoding units, and a feature fusion unit;

[0017] The plurality of encoding units are connected in sequence, the plurality of decoding units are connected in sequence, and the plurality of encoding units and the plurality of decoding units are connected in one-to-one correspondence.

[0018] The feature fusion unit is connected with the plurality of encoding units and the plurality of decoding units respectively, and the feature fusion unit is configured to perform feature fusion on the encoding information output by the plurality of encoding units.

[0019] The plurality of encoding units support different resolutions, and each encoding unit is configured to perform down-sampling and feature extraction according to the respective supported resolution.

[0020] Optionally, the feature fusion unit is further configured to, before performing feature fusion, perform up-sampling and / or down-sampling processing on the obtained encoding information based on the resolution supported by the target decoding unit.

[0021] Optionally, the method further comprises:

[0022] obtaining a key frame corresponding to the current encoding frame;

[0023] using the key frame as a virtual reference frame.

[0024] In a second aspect, embodiments of the present disclosure provide a virtual reference frame generation apparatus, comprising:

[0025] a first processing module configured to query a first frame and a second frame in a reference frame list, wherein a time domain distance between the first frame and the second frame is equal to a time domain distance between the second frame and a current encoding frame, and the first frame and the second frame are located before the current encoding frame;

[0026] a second processing module configured to, in a case where the first frame and the second frame are queried, generate a first prediction reference frame according to the first frame and the second frame;

[0027] The third processing module is configured to generate a second prediction reference frame according to a third frame and a fourth frame when the first frame and the second frame are not queried; wherein the third frame and the fourth frame are frames in the reference frame list that have the smallest time domain distance from the current coding frame.

[0028] The fourth processing module is configured to take the first prediction reference frame or the second prediction reference frame as a virtual reference frame.

[0029] Optionally, the second processing module is further configured to:

[0030] input the first frame and the second frame into a first generation network;

[0031] take an output of the first generation network as the first prediction reference frame.

[0032] Optionally, the first generation network comprises a plurality of encoding units, a plurality of decoding units, and a feature fusion unit.

[0033] The plurality of encoding units are connected in sequence, the plurality of decoding units are connected in sequence, and the plurality of encoding units and the plurality of decoding units are connected in one-to-one correspondence.

[0034] The feature fusion unit is connected with the plurality of encoding units and the plurality of decoding units respectively, and the feature fusion unit is configured to perform feature fusion on encoding information output by the plurality of encoding units.

[0035] The plurality of encoding units support different resolutions, and each encoding unit is configured to perform down-sampling and feature extraction according to a resolution supported by the encoding unit.

[0036] Optionally, the feature fusion unit is further configured to, before performing feature fusion, perform up-sampling and / or down-sampling processing on the obtained encoding information based on a resolution supported by a target decoding unit.

[0037] Optionally, the apparatus further comprises:

[0038] The obtaining module is configured to obtain a key frame corresponding to the current coding frame.

[0039] The fifth processing module is configured to take the key frame as a virtual reference frame.

[0040] In a third aspect, an embodiment of the present disclosure provides a virtual reference frame generation device, comprising a memory, a processor, and a program stored in the memory and capable of running on the processor; the processor is configured to read the program in the memory to implement the steps in the virtual reference frame generation method described above.

[0041] In a fourth aspect, the present disclosure provides a readable storage medium for storing a program, which, when executed by a processor, implements the steps of the virtual reference frame generation method as described above.

[0042] In a fifth aspect, a computer program product is provided, which includes computer instructions, which, when executed by a processor, implement the steps of the virtual reference frame generation method as described in the first aspect.

[0043] The method of the present disclosure, by querying the first frame and the second frame in the reference frame list, preferably generates a first predicted reference frame as a virtual reference frame according to the first frame and the second frame; and if the first frame and the second frame are not queried, generates a second predicted reference frame as a virtual reference frame according to the third frame and the fourth frame. Since the time domain distance between the first frame and the second frame is equal to the time domain distance between the second frame and the current encoding frame, the first predicted reference frame can provide a higher quality reference for the current encoding frame than the second predicted reference frame in the inter prediction link of video encoding, thereby improving the overall encoding performance. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the present disclosure, the drawings needed in the description of the embodiments of the present disclosure will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor.

[0045] FIG. 1 shows a flowchart of the virtual reference frame generation method of the present disclosure;

[0046] FIG. 2 shows a structural diagram of a first generation network in the present disclosure;

[0047] FIG. 3 shows a structural diagram of an encoding unit in the present disclosure;

[0048] FIG. 4 shows a structural diagram of a decoding unit in the present disclosure;

[0049] FIG. 5 shows a structural diagram of a feature fusion unit in the present disclosure;

[0050] FIG. 6 shows a structural diagram of a virtual reference frame generation device in the present disclosure;

[0051] FIG. 7 shows a structural diagram of a virtual reference frame generation device in the present disclosure;

[0052] FIG. 8 shows a reference frame list diagram in the present disclosure. DETAILED DESCRIPTION

[0053] To make the technical problems to be solved by the present disclosure, technical solutions and advantages clearer, specific embodiments will be described in detail below with reference to the drawings. In the following description, specific details such as specific configurations and components are provided only to help a comprehensive understanding of embodiments of the present disclosure. Therefore, it should be clear to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, descriptions of known functions and structures are omitted for clarity and brevity.

[0054] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present disclosure. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner.

[0055] In various embodiments of the present disclosure, it should be understood that the size of the serial number of the following processes does not mean the order of execution, and the execution order of the processes should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure. In addition, the terms "system" and "network" are often used interchangeably herein.

[0056] For the convenience of understanding, some of the contents related to the embodiments of the present disclosure are described as follows:

[0057] Under the low delay P configuration, there are more reference frames available in the reference frame list (RFL). If the two frames closest in time domain to the current frame are selected for the generation of virtual reference frames, the motion amplitude in the actual coded picture is ignored, resulting in a large texture difference between the generated virtual reference frame and the current frame to be coded.

[0058] The reference frames in the RFL are identified using the picture order count (POC).

[0059] As shown in FIG. 1, the embodiments of the present disclosure provide a virtual reference frame generation method, which comprises:

[0060] Step 11, querying a first frame and a second frame in a reference frame list, the time domain distance between the first frame and the second frame is equal to the time domain distance between the second frame and the current coded frame, and the first frame and the second frame are located before the current coded frame.

[0061] Here, the current encoding frame can be understood as a current frame to be encoded. The reference frame list is a list of multiple reference frames configured, such as the reference frame list under the Low delay P configuration. Moreover, the time domain distance can be calculated by the POC of each frame, so for the first frame and the second frame located before the current encoding frame, since the time domain distance between the first frame and the second frame is equal to the time domain distance between the second frame and the current encoding frame, the first frame, the second frame and the current encoding frame form three frames with equal interval of △POC.

[0062] Step 12, in the case of querying the first frame and the second frame, generating a first prediction reference frame according to the first frame and the second frame.

[0063] That is, after performing step 11, if the first frame and the second frame meeting the requirements are queried in the reference frame list, the first prediction reference frame will be generated using the two frames.

[0064] Step 13, in the case of not querying the first frame and the second frame, generating a second prediction reference frame according to a third frame and a fourth frame; wherein the third frame and the fourth frame are the frames with the smallest time domain distance to the current encoding frame in the reference frame list.

[0065] That is, after performing step 11, if the first frame and the second frame meeting the requirements cannot be queried in the reference frame list, the second prediction reference frame will be generated using the two frames with the smallest time domain distance to the current encoding frame. At this time, for the other frames in the reference frame list except the third frame and the fourth frame, the time domain distance to the current encoding frame is less than or equal to d0, d0 is the minimum value of d1 and d2, d1 is the time domain distance between the third frame and the current encoding frame, and d2 is the time domain distance between the fourth frame and the current encoding frame.

[0066] Step 14, taking the first prediction reference frame or the second prediction reference frame as a virtual reference frame.

[0067] In this way, according to the above steps 11-14, the method of the embodiment of the disclosure will query the first frame and the second frame in the reference frame list, preferably generate a first prediction reference frame according to the first frame and the second frame to serve as a virtual reference frame; and if the first frame and the second frame are not queried, generate a second prediction reference frame according to the third frame and the fourth frame to serve as a virtual reference frame. Since the time domain distance between the first frame and the second frame is equal to the time domain distance between the second frame and the current encoding frame, in the inter prediction link of video encoding, the first prediction reference frame can provide a higher quality reference for the current encoding frame compared to the second prediction reference frame, thereby improving the overall encoding performance.

[0068] It should be noted that the picture content between adjacent frames changes less, and therefore the motion amplitude and trend between the first frame and the second frame and the motion amplitude and trend between the second frame and the current encoding frame are more similar. Using the first prediction reference frame generated from the first frame and the second frame as the virtual reference frame can greatly increase the encoding performance gain.

[0069] It should be understood that after the first prediction reference frame is used as the virtual reference frame or the second prediction reference frame is used as the virtual reference frame in this embodiment, the virtual reference frame is added to the reference frame list to obtain an updated reference frame list. As an optional implementation, the virtual reference frame is added to the end of the original reference frame list.

[0070] Example 1, the current encoding frame (POC: 8), the original reference frame list (Request Parameter List 0, RPL0) includes 4 frames, denoted as RPL0: [7 5 6 0]. It is determined through the query that the POCs of the first frame and the second frame are 6 and 7 respectively, and a first prediction reference frame with a mark POC of -1 is further generated, and then the updated RPL0 is: [7 5 6 0 -1].

[0071] Example 2, the current encoding frame (POC: 24), the original reference frame list (RPL0) includes 4 frames, denoted as RPL0: [23 16 8 0]. It is determined through the query that the POCs of the first frame and the second frame are 8 and 16 respectively, and a first prediction reference frame with a mark POC of -1 is further generated, and then the updated RPL0 is: [23 16 8 0 -1].

[0072] Example 3, the current encoding frame (POC: 25), the original reference frame list (RPL0) includes 4 frames, denoted as RPL0: [24 16 8 0]. When no first frame and second frame satisfying the condition are found, it is determined that the POCs of the third frame and the fourth frame are 16 and 24 respectively, a second prediction reference frame with a mark POC of -1 is further generated, and then the updated RPL0 is: [24 16 8 0 -1].

[0073] In addition, in this embodiment, if multiple groups of first frames and second frames are found through the query, the first frame and the second frame with the smallest time domain distance are selected to generate the first prediction reference frame.

[0074] Optionally, in this embodiment, the generating the first prediction reference frame according to the first frame and the second frame comprises:

[0075] inputting the first frame and the second frame into a first generation network;

[0076] using the output of the first generation network as the first prediction reference frame.

[0077] The first generation network is pre-trained, and takes the first frame and the second frame as input and outputs the first prediction reference frame. Thus, for the queried first frame and second frame, the first prediction reference frame is obtained by inputting the first frame and the second frame into the first generation network.

[0078] Optionally, before the first frame and the second frame are input into the first generation network, the method further comprises:

[0079] The first generation network is obtained by training the preset network based on a first training set. The first training set includes multiple groups of frames, and the time domain distance between adjacent frames in each group of frames is the same.

[0080] As an implementation manner, each group of frames in the first training set includes four frames, wherein frame 1, frame 2 and frame 3 are adjacent frames, the time domain distance between frame 1 and frame 2 is equal to the time domain distance between frame 2 and frame 3, and frame 4 is the prediction reference frame of frame 3. The first generation network is obtained by training the preset network using the first training set, so that the first prediction reference frame output by the first generation network under the condition of inputting the first frame and the second frame is more similar to the content of the current encoding frame, thereby providing a higher quality reference for the current encoding frame in the inter prediction link of video encoding, and achieving the purpose of improving the overall encoding performance.

[0081] In addition, in the training process, the adjacent frames can be divided into multiple training sets according to different time domain distances to obtain multiple first generation networks. When the time domain distances of the first frame, the second frame and the current encoding frame are different, the first prediction reference frame is generated by using the first generation network suitable for the time domain distance. Alternatively, multiple groups of parameters of a first generation network are obtained, and when the time domain distances of the first frame, the second frame and the current encoding frame are different, the first generation network calls a group of parameters suitable for the time domain distance to generate the first prediction reference frame.

[0082] Optionally, in this embodiment, the second prediction reference frame is generated according to the third frame and the fourth frame, comprising:

[0083] The third frame and the fourth frame are input into a second generation network.

[0084] The second prediction reference frame is output by the second generation network.

[0085] The second generation network is pre-trained, and takes the third frame and the fourth frame as input and outputs the second prediction reference frame. Thus, for the case that the first frame and the second frame are not queried, the second prediction reference frame is obtained by inputting the third frame and the fourth frame into the second generation network.

[0086] Optionally, in this embodiment, before the third frame and the fourth frame are input into the second generation network, the method further comprises:

[0087] The preset network is trained based on a second training set to obtain the second generation network; wherein the second training set comprises a plurality of groups of frames, and the time domain distance between the reference frame and the target frame in each group of frames is less than or equal to a specific time domain distance.

[0088] As an implementation, each group of frames in the second training set comprises four frames, wherein the time domain distance between frame 1', frame 2' and frame 3' is less than or equal to a specific time domain distance, which is pre-configured or defined, and frame 4' is the predicted reference frame of frame 3'. The preset network is trained using the second training set, so that the second generation network obtained by training outputs a second predicted reference frame in the case of inputting a third frame and a fourth frame, which provides a high-quality reference for the current coding frame in the inter prediction link of video coding.

[0089] Optionally, in this embodiment, the first generation network comprises a plurality of encoding units, a plurality of decoding units, and a feature fusion unit;

[0090] The plurality of encoding units are connected in sequence, the plurality of decoding units are connected in sequence, and the plurality of encoding units and the plurality of decoding units are connected in one-to-one correspondence;

[0091] The feature fusion unit is connected with the plurality of encoding units and the plurality of decoding units respectively, and the feature fusion unit is configured to perform feature fusion on the encoding information output by the plurality of encoding units;

[0092] The plurality of encoding units support different resolutions, and each encoding unit is configured to perform downsampling and feature extraction according to the resolution it supports. In this way, the first generation network can adopt a multi-scale encoder-decoder network (MSEDN), and large-scale encoders and decoders are mainly beneficial for static content prediction, and small-scale encoders and decoders are mainly beneficial for fine motion prediction. Among them, the resolutions supported by the plurality of encoding units decrease in the connection direction, and the plurality of decoding units support the same resolution as the corresponding connected encoding units. In this embodiment, the structure of the second generation network and the preset network is the same as that of the first generation network.

[0093] For example, as shown in FIG. 2, the first generation network includes an encoding unit (Encoder) 1, an encoding unit 2, an encoding unit 3, an encoding unit 4, a decoding unit (Decoder) 1, a decoding unit 2, a decoding unit 3, a decoding unit 4, and a feature fusion unit. Among them, the resolutions supported by the encoding units 1-4 decrease in turn, that is, the resolution of the feature map F1 output by the encoding unit 1 is greater than the resolution of the feature map F2 output by the encoding unit 2, the resolution of F2 is greater than the resolution of the feature map F3 output by the encoding unit 3, the resolution of F3 is greater than the resolution of the feature map F4 output by the encoding unit 4, and similarly, the resolutions supported by the decoding units 1-4 also decrease in turn, and the resolutions supported by the corresponding encoding units and decoding units are the same. Among them, the structure of the encoding unit is as shown in FIG. 3, and the structure of the decoding unit is as shown in FIG. 4.

[0094] Specifically, the first frame (I0) and the second frame (I1) are input into the network structure, I0 and I1 are input into the Encoder 1 for down-sampling feature extraction on one hand, and the optical flow obtained by splitting the features output by the Decoder 1 on the other hand to obtain the reconstructed image closest to the current encoding frame through the warp operation, and the total features D1 output by the Decoder are fused to obtain I2.

[0095] Among them, at the Encoder end, I0 and I1 are input into the Encoder 1 (here, it means that the Encoder 1 is only one, and I0 and I1 are separately input for inference) to obtain two features f 1_0 and f 1_1 On one hand, the two features are warped by the bidirectional optical flow output by the upper-level decoder Decoder 2 through the yellow line to obtain the reconstructed image closer to the current encoding frame, which is then fused with the output D2 of the Decoder 2 and input into the Decoder 1; on the other hand, f 1_0 and f 1_1 are combined to obtain F1, which is continuously input into the next-level Encoder 2.

[0096] In addition, at the Decoder end, for example, the Decoder 4 receives the input of the F4 feature and the fused feature (blue line) obtained by cross-scale feature fusion, and after passing through the Decoder 4, the reconstructed image D4 is obtained. After the same channel splitting operation at the Encoder end, two optical flows and are split out and warped with the features of the two frames of the Encoder 3, respectively.

[0097] Optionally, in this embodiment, the feature fusion unit is further configured to, before performing the feature fusion, perform up-sampling and / or down-sampling on the obtained encoding information based on a resolution supported by the target decoding unit.

[0098] In this embodiment, the target decoding unit refers to the task object of the feature fusion unit. In this way, the feature fusion unit (Cross-scale Feature Fusion) performs up-sampling and / or down-sampling on the obtained encoding information (such as the feature map) based on a resolution supported by the target decoding unit before performing the feature fusion. For example, as shown in the feature fusion unit in FIG. 5, if the target decoding unit is the encoding unit 3, the F1 is down-sampled by 4 times, the F2 is down-sampled by 2 times, and the F3 is up-sampled by 2 times, and then the feature fusion is performed (the feature fusion is performed by 1x1 convolution), so that the feature output to the encoding unit 3 has the same resolution as the encoding unit 3 supports.

[0099] In this embodiment, the feature fusion unit considers that there is information loss in the feature extraction and down-sampling process, so it retains the lost part of information by learning, and adds this part of information in the prediction process (i.e., the process of gradually up-sampling to restore the original image scale). In addition, since it is cross-scale, it can also dynamically learn the information beneficial to prediction under different sizes, which can further improve the accuracy of the prediction result.

[0100] It should be further noted that, in this embodiment, the first generation network can also be referred to as a POC model, and the second generation network can also be referred to as a reference frame generation model (Closest model).

[0101] Optionally, in this embodiment, the method further comprises:

[0102] obtaining a key frame corresponding to the current encoding frame;

[0103] adding the key frame as a virtual reference frame.

[0104] Here, the key frame is the first frame in a Group of Pictures (GoP). In this way, after obtaining the key frame corresponding to the current encoding frame, the key frame can be added to the reference frame list as a virtual reference frame, thereby improving the encoding gain under the Low delay P encoding configuration.

[0105] As an implementation, as shown in FIG. 8, the key frame and the prediction reference frame (the first prediction reference frame or the second prediction reference frame) can be added to the end of the reference frame list as virtual reference frames. Specifically, the key frame is located after the first prediction reference frame or the second prediction reference frame.

[0106] As an implementation, only the key frame can be added to the end of the reference frame list, for example, the picture is mainly static content; or only the prediction reference frame (the first prediction reference frame or the second prediction reference frame) is added to the end of the reference frame list, for example, the picture is mainly content with violent motion.

[0107] In summary, the method of the embodiments of the present disclosure uses different networks to obtain virtual reference frames with different motion amplitudes and generation quality, so that the virtual reference frames are more similar to the current encoding frame in texture, to provide high-quality reference. Moreover, the additional key frame can provide high-quality background and subject reference.

[0108] As shown in FIG. 6, the embodiments of the present disclosure further provide a virtual reference frame generation device 60, comprising:

[0109] A first processing module 61 is configured to query a first frame and a second frame in a reference frame list, wherein a time domain distance between the first frame and the second frame is equal to a time domain distance between the second frame and a current encoding frame;

[0110] A second processing module 62 is configured to, in a case where the first frame and the second frame are queried, generate a first prediction reference frame according to the first frame and the second frame, and the first frame and the second frame are located before the current encoding frame;

[0111] A third processing module 63 is configured to, in a case where the first frame and the second frame are not queried, generate a second prediction reference frame according to a third frame and a fourth frame; wherein the third frame and the fourth frame are frames with the smallest time domain distance to the current encoding frame in the reference frame list;

[0112] A fourth processing module 64 is configured to take the first prediction reference frame or the second prediction reference frame as a virtual reference frame.

[0113] Optionally, the second processing module is further configured to:

[0114] input the first frame and the second frame into a first generation network;

[0115] take an output of the first generation network as the first prediction reference frame.

[0116] Optionally, the device further comprises:

[0117] A first training module is configured to train a preset network based on a first training set to obtain the first generation network; wherein the first training set comprises multiple groups of frames, and adjacent two frames in each group of frames have the same time domain distance.

[0118] Optionally, the third processing module is further configured to:

[0119] inputting the third frame and the fourth frame into a second generation network;

[0120] outputting, by the second generation network, the second predicted reference frame.

[0121] Optionally, the apparatus further comprises:

[0122] a second training module configured to train a preset network based on a second training set to obtain the second generation network, wherein the second training set comprises a plurality of groups of frames, and a time domain distance between a reference frame and a target frame in each group of frames is less than or equal to a specific time domain distance.

[0123] Optionally, the first generation network comprises a plurality of encoding units, a plurality of decoding units, and a feature fusion unit.

[0124] The plurality of encoding units are connected in sequence, the plurality of decoding units are connected in sequence, and the plurality of encoding units and the plurality of decoding units are connected in one-to-one correspondence.

[0125] The feature fusion unit is connected with the plurality of encoding units and the plurality of decoding units respectively, and the feature fusion unit is configured to perform feature fusion on encoding information output by the plurality of encoding units.

[0126] The plurality of encoding units support different resolutions, and each encoding unit is configured to perform down-sampling and feature extraction according to a respective supported resolution.

[0127] Optionally, the feature fusion unit is further configured to, before performing feature fusion, perform up-sampling and / or down-sampling processing on the obtained encoding information based on a resolution supported by a target decoding unit.

[0128] Optionally, the apparatus further comprises:

[0129] an acquisition module configured to acquire a key frame corresponding to the current encoding frame;

[0130] a fifth processing module configured to take the key frame as a virtual reference frame.

[0131] The implementation principle and technical effects of the apparatus of the present disclosure are similar, and will not be described here.

[0132] As shown in FIG. 7, the present disclosure also provides a virtual reference frame generation device, which comprises a processor 701, a memory 702, and a computer program stored in the memory 702 and executable on the processor 701, wherein the processor 701 implements the steps of the virtual reference frame generation method described above when executing the computer program.

[0133] The virtual reference frame generation device provided by the embodiments of the present disclosure can execute the method embodiments described above, and the implementation principles and technical effects are similar, and thus will not be repeated here.

[0134] Those skilled in the art can understand that all or part of the steps of the above embodiments can be completed by hardware, or by a computer program instructing the relevant hardware to complete part or all of the steps of the above method, and the computer program can be stored in a readable storage medium, which can be any form of storage medium.

[0135] In addition, the specific embodiments of the present disclosure also provide a readable storage medium having a program stored thereon, and the program is executed by a processor to implement the steps of the virtual reference frame generation method described above. The same technical effects can be achieved, and thus will not be repeated here.

[0136] The embodiments of the present disclosure also provide a computer program product including computer instructions, which are executed by a processor to implement each process of the method embodiment shown in FIG. 1, and the same technical effects can be achieved, and thus will not be repeated here.

[0137] In several embodiments provided by the present disclosure, it should be understood that the disclosed method and device can be implemented by other ways. For example, the above- described device embodiments are only schematic, and for example, the division of the units is only a logical function division, and there can be another division way in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0138] In addition, each functional unit in the various embodiments of the present disclosure can be integrated in one processing unit, or each unit can be physically separated, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware, or in the form of hardware plus software function unit.

[0139] The integrated unit in the form of software function unit can be stored in a computer readable storage medium. The software function unit is stored in a storage medium, and includes a plurality of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform part of the steps of the transceiving method according to the embodiments of the present disclosure. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various storage medium capable of storing program codes.

[0140] The above is the preferred embodiment of the present disclosure, and it should be pointed out that those skilled in the art can make several improvements and refinements without departing from the principles of the present disclosure, and these improvements and refinements are also within the protection scope of the present disclosure.

Claims

1. A method for generating a virtual reference frame, comprising: querying a first frame and a second frame in a reference frame list, wherein a time domain distance between the first frame and the second frame is equal to a time domain distance between the second frame and a current encoding frame, and the first frame and the second frame are located before the current encoding frame; generating a first prediction reference frame according to the first frame and the second frame in a case that the first frame and the second frame are queried; generating a second prediction reference frame according to a third frame and a fourth frame in a case that the first frame and the second frame are not queried, wherein the third frame and the fourth frame are frames in the reference frame list having a minimum time domain distance with the current encoding frame; taking the first prediction reference frame or the second prediction reference frame as a virtual reference frame.

2. The method of claim 1, wherein, The generating the first prediction reference frame according to the first frame and the second frame comprises: inputting the first frame and the second frame into a first generation network; taking an output of the first generation network as the first prediction reference frame.

3. The method of claim 2, wherein, The first generation network comprises a plurality of encoding units, a plurality of decoding units, and a feature fusion unit; wherein the plurality of encoding units are connected in sequence, the plurality of decoding units are connected in sequence, and the plurality of encoding units and the plurality of decoding units are connected in one-to-one correspondence; the feature fusion unit is connected with the plurality of encoding units and the plurality of decoding units respectively, and is configured to perform feature fusion on encoding information output by the plurality of encoding units; the plurality of encoding units support different resolutions, and each encoding unit is configured to perform down-sampling and feature extraction according to a respective supported resolution.

4. The method of claim 3, wherein, The feature fusion unit is further configured to perform up-sampling and / or down-sampling processing on the obtained encoding information based on a resolution supported by a target decoding unit before performing the feature fusion.

5. The method of claim 1, wherein, The method further comprises: obtaining a key frame corresponding to the current encoding frame; taking the key frame as a virtual reference frame. 6.A device for generating a virtual reference frame, comprising: a first processing module configured to query a first frame and a second frame in a reference frame list, wherein a time domain distance between the first frame and the second frame is equal to a time domain distance between the second frame and a current encoding frame, and the first frame and the second frame are located before the current encoding frame; a second processing module configured to generate a first prediction reference frame according to the first frame and the second frame in a case that the first frame and the second frame are queried; a third processing module configured to generate a second prediction reference frame according to a third frame and a fourth frame in a case that the first frame and the second frame are not queried, wherein the third frame and the fourth frame are frames in the reference frame list having a minimum time domain distance with the current encoding frame; a fourth processing module configured to take the first prediction reference frame or the second prediction reference frame as a virtual reference frame.

7. The apparatus of claim 6, wherein, The second processing module is further configured to: input the first frame and the second frame into a first generation network; take an output of the first generation network as the first prediction reference frame.

8. The apparatus of claim 6, wherein, The device further comprises: an obtaining module configured to obtain a key frame corresponding to the current encoding frame; a fifth processing module configured to take the key frame as a virtual reference frame.

9. A virtual reference frame generation apparatus comprising: a memory, a processor, and a program stored in the memory and capable of running on the processor; the processor is configured to read the program in the memory to implement the steps in the virtual reference frame generation method according to any one of claims 1 to 5.

10. A readable storage medium for storing a program, wherein the program is executed by a processor to implement the steps in the virtual reference frame generation method according to any one of claims 1 to 5.

Images