Video super-resolution and model training method and device, electronic equipment and medium

Abstract

The embodiment of the invention discloses a video super-resolution and model training method and device, electronic equipment and a medium, a video super-resolution model comprises a generator network and a discriminator network, the video super-resolution model training method comprises at least one round of training process, and any round of training process comprises the following steps: obtaining a sample video frame image; inputting the sample video frame image into a generator network to generate a super-resolution image corresponding to the sample video frame image; determining a first down-sampling image and a second down-sampling image corresponding to the super-resolution image; inputting the super-resolution image, the first down-sampling image and the second down-sampling image into a discriminator network, and generating a first discrimination result based on the super-resolution image, the first down-sampling image and the second down-sampling image through the discriminator network; according to the first judgment result, the loss of the video super-resolution model is constructed, and the loss is used for adjusting model parameters of the video super-resolution model.

Description

Technical Field

[0001] The present disclosure relates to the field of video super-resolution technology, and more specifically, to a video super-resolution and model training method, device, electronic device, and medium. Background Art

[0002] In video super-resolution (VSR) technology, unsupervised video super-resolution algorithms often achieve better visual effects when processing low-resolution video frames. RealBasicVSR, an open-source video super-resolution framework, is one of the most widely used algorithms in academia. However, the existing RealBasicVSR framework still suffers from poor visual quality when processing video frames. Summary of the Invention

[0003] The embodiments of the present disclosure provide a video super-resolution and model training method, device, electronic device and medium.

[0004] According to a first aspect of the present disclosure, a video super-resolution model training method is provided, wherein the video super-resolution model includes a generator network and a discriminator network. The video super-resolution model training method includes at least one round of training process, and any round of the training process includes:

[0005] Get sample video frame images;

[0006] Inputting the sample video frame image into the generator network to generate a super-resolution image corresponding to the sample video frame image;

[0007] Determine a first down-sampled image and a second down-sampled image corresponding to the super-resolution image;

[0008] Inputting the super-resolution image, the first down-sampled image, and the second down-sampled image into the discriminator network, and generating a first discrimination result based on the super-resolution image, the first down-sampled image, and the second down-sampled image by the discriminator network;

[0009] According to the first discrimination result, a loss of the video super-resolution model is constructed, and the loss is used to adjust model parameters of the video super-resolution model.

[0010] Optionally, the method further includes:

[0011] Inputting the sample video frame image into the generator network to generate an optical flow corresponding to the super-resolution image;

[0012] Obtaining the GT optical flow corresponding to the sample video frame image;

[0013] The optical flow corresponding to the super-resolution image and the GT optical flow are input into the discriminator network to generate a first optical flow map of the optical flow corresponding to the super-resolution image and a second optical flow map of the GT optical flow, and the discriminator network generates a second discrimination result based on the first optical flow map and the second optical flow map.

[0014] Optionally, constructing the loss of the video super-resolution model according to the first discrimination result includes:

[0015] Constructing a loss of the video super-resolution model according to the first discrimination result and the second discrimination result.

[0016] Optionally, inputting the optical flow corresponding to the super-resolution image and the GT optical flow into the discriminator network to generate a first optical flow map of the optical flow corresponding to the super-resolution image and a second optical flow map of the GT optical flow includes:

[0017] The optical flow corresponding to the super-resolution image and the GT optical flow are input into the discriminator network, and the optical flow corresponding to the super-resolution image and the GT optical flow are processed respectively by the optical flow expert algorithm to generate a first optical flow map of the optical flow corresponding to the super-resolution image and a second optical flow map of the GT optical flow.

[0018] Optionally, the loss includes a first loss of the generator network and a second loss of the discriminator network, the model parameters include a first network parameter of the generator network and a second network parameter of the discriminator network, the first loss is used to adjust the first network parameter, and the second loss is used to adjust the second network parameter.

[0019] According to a second aspect of the present disclosure, a video super-resolution method is provided, the method comprising:

[0020] Obtain the video frame image to be processed;

[0021] Inputting the to-be-processed video frame image into a preset video super-resolution model to obtain a super-resolution image corresponding to the to-be-processed video frame image;

[0022] The video super-resolution model is obtained according to the first aspect above.

[0023] According to a third aspect of the present disclosure, a video super-resolution model training apparatus is provided, wherein the video super-resolution model includes a generator network and a discriminator network, and the video super-resolution model training method includes at least one round of training process, wherein any round of the training process includes:

[0024] A first acquisition module is used to acquire a sample video frame image;

[0025] a generation module, configured to input the sample video frame image into the generator network to generate a super-resolution image corresponding to the sample video frame image;

[0026] A determination module, configured to determine a first down-sampled image and a second down-sampled image corresponding to the super-resolution image;

[0027] a discriminant module, configured to input the super-resolution image, the first down-sampled image, and the second down-sampled image into the discriminator network, and generate a first discrimination result based on the super-resolution image, the first down-sampled image, and the second down-sampled image through the discriminator network;

[0028] A construction module is used to construct a loss of the video super-resolution model according to the first discrimination result, and the loss is used to adjust the model parameters of the video super-resolution model.

[0029] According to a fourth aspect of the present disclosure, a video super-resolution apparatus is provided, the apparatus comprising:

[0030] A second acquisition module is used to acquire the image to be processed;

[0031] A processing module, configured to input the image to be processed into a preset video super-resolution model to obtain a super-resolution image corresponding to the image to be processed;

[0032] The video super-resolution model is obtained according to the first aspect above.

[0033] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising a memory for storing executable computer instructions; and a processor for executing, under the control of the executable computer instructions, the video super-resolution model training method according to the first aspect above, or the video super-resolution method according to the second aspect above.

[0034] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, on which computer instructions are stored. When the computer instructions are executed by a processor, the video super-resolution model training method described in the first aspect above is executed, or the video super-resolution method described in the second aspect above is executed.

[0035] According to the video super-resolution model training method of the embodiment of the present disclosure, the video super-resolution model includes a generator network and a discriminator network. The generator network can generate a super-resolution image corresponding to a sample video frame image. The input of the discriminator network includes a super-resolution image, a first down-sampled image, and a second down-sampled image corresponding to the super-resolution image. Based on the super-resolution image, the first down-sampled image, and the second down-sampled image corresponding to the super-resolution image, a first discrimination result can be generated, and then the loss of the video super-resolution model is constructed based on the first discrimination result. In other words, through the embodiment of the present disclosure, it uses three images of different resolutions as input to the discriminator network, so that the three images of different resolutions are mixed through the discriminator network to obtain richer high-resolution images, thereby achieving better model training effects.

[0036] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0038] Figure 1 is a schematic diagram of the hardware configuration of an electronic device according to an embodiment of the present disclosure;

[0039] Figure 2 is a flow chart of a video super-resolution model training method according to an embodiment of the present disclosure;

[0040] Figure 3 is a schematic diagram of the structure of a discriminator network according to an embodiment of the present disclosure;

[0041] Figure 4 is a flowchart of a video super-resolution method according to an embodiment of the present disclosure;

[0042] Figure 5 is a principle block diagram of a video super-resolution model training device according to an embodiment of the present disclosure;

[0043] Figure 6 is a principle block diagram of a video super-resolution apparatus according to an embodiment of the present disclosure;

[0044] Figure 7 Schematic diagram of the hardware configuration of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0045] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0046] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0047] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0048] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0049] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0050] <Hardware Configuration>

[0051] Figure 1 is a block diagram of a hardware configuration of an electronic device 1000 according to an embodiment of the present disclosure.

[0052] In one embodiment, electronic device 1000 may be a server or a terminal device. The server may be a monolithic server or a distributed server across multiple computers or computer data centers. The terminal device may be a portable computer, a desktop computer, a wearable device, or any other device having a computing device such as a processor and a storage device such as a memory, and this embodiment does not limit this.

[0053] like Figure 1 As shown, the electronic device 1000 may include a processor 1100 , a memory 1200 , an interface device 1300 , a communication device 1400 , a display device 1500 , an input device 1600 , a speaker 1700 , a microphone 1800 , and the like.

[0054] The processor 1100 may be a mobile processor. The memory 1200 may include, for example, ROM (read-only memory), RAM (random access memory), and non-volatile memory such as a hard disk. The interface device 1300 may include, for example, a USB interface, a headphone jack, and the like. The communication device 1400 may be capable of wired or wireless communication. The communication device 1400 may include a short-range communication device, such as any device that performs short-range wireless communication based on short-range wireless communication protocols such as Hilink protocol, WiFi (IEEE 802.11 protocol), Mesh, Bluetooth, ZigBee, Thread, Z-Wave, NFC, UWB, LiFi, etc. The communication device 1400 may also include a long-range communication device, such as any device that performs WLAN, GPRS, 2G / 3G / 4G / 5G long-range communication. The display device 1500 may be, for example, an LCD display, a touch screen display, and the like. The input device 1600 may include, for example, a touch screen, a keyboard, and the like. The electronic device 1000 may output audio information through the speaker 1700 and may collect audio information through the microphone 1800.

[0055] Despite Figure 1 Although multiple devices are shown for the electronic device 1000 , the present disclosure may only involve some of the devices. For example, the electronic device 1000 only involves the memory 1200 and the processor 1100 .

[0056] In the embodiment of the present disclosure, the memory 1200 of the electronic device 1000 is used to store instructions, which are used to control the processor 1100 to execute the video super-resolution model training method or video super-resolution method provided by the embodiment of the present disclosure.

[0057] In the above description, a person skilled in the art can design instructions according to the scheme disclosed in this disclosure. How instructions control the operation of a processor is well known in the art and will not be described in detail here.

[0058] <Method Example 1>

[0059] In this embodiment, a video super-resolution model training method is provided, which can be implemented by an electronic device, such as Figure 1 The electronic device 1000 is shown.

[0060] It should be noted that before introducing the video super-resolution model training method of the embodiment of the present disclosure, a brief introduction to super-resolution (SR) and video super-resolution (VSR) is first given. Typically, super-resolution is used to recover or reconstruct high-resolution (HR) images from low-resolution (LR) images. Video super-resolution is used to convert a low-resolution video into a high-resolution version, thereby providing more details and clarity. It can be understood that video super-resolution (VSR) can be understood as super-resolution (SR) of the video frame images of the video.

[0061] according to Figure 2 As shown, the video super-resolution model training method of the embodiment of the present disclosure may include the following steps S2100 to S2500.

[0062] Step S2100: Obtain a sample video frame image.

[0063] The sample video frame images are video frame images in a sample video, and the sample video is a video that needs to be processed with video super-resolution. Usually, the sample video can be processed by dividing the frames into frames to obtain the sample video frame images.

[0064] As can be understood, during a round of video super-resolution model training, sample video frames are typically input based on a batch size. The selection of a batch size significantly impacts the training performance, convergence speed, and memory utilization of the video super-resolution model. Therefore, obtaining sample video frames in step S2100 may specifically include obtaining a batch size of sample video frames.

[0065] For example, taking the batch size Batch Size as M (M is an integer greater than 1) as an example, when performing any round of video super-resolution model training, the electronic device can obtain M sample video frame images and use these M sample video frame images as training samples for this round of video super-resolution model, so as to perform this round of video super-resolution model training based on the training samples.

[0066] After executing the above step S2100 to obtain the sample video frame image, proceed to:

[0067] Step S2200: Input the sample video frame image into the generator network to generate a super-resolution image corresponding to the sample video frame image.

[0068] The video super-resolution model includes a generator network that can be used to generate super-resolution images corresponding to sample video frames. The generator network here is roughly the same as the generator network in RealBasicVSR.

[0069] In one embodiment, the electronic device may input a batch size of sample video frame images into a generator network, and generate an XN-fold super-resolution image corresponding to each sample video frame image through the generator network, where N is the super-resolution multiple. When XN-fold, it means that the horizontal pixels of the generated super-resolution image are N times that of the corresponding sample video frame image, and the vertical pixels are N times that of the corresponding sample video frame image.

[0070] For example, assuming N is 4, the electronic device can input the M sample video frames into the generator network, which then generates an x4x super-resolution image corresponding to each sample video frame. In the x4x scenario, the x4x super-resolution image can be denoted as an HxW SR video frame image, where H is the height of the x4x super-resolution image and W is the width of the x4x super-resolution image.

[0071] In another embodiment, the video super-resolution model further includes a pre-cleaning network for cleaning sample video frames to reduce the impact of input degradation on video super-resolution model training. In this embodiment, the electronic device may first input a batch size of sample video frames into the pre-cleaning network. After the pre-cleaning network cleans the batch size of video frames, the cleaned batch size of sample video frames is then input into the generator network.

[0072] After executing step S2200 above to input the sample video frame image into the generator network and generate a super-resolution image corresponding to the sample video frame image, proceed to:

[0073] Step S2300: Determine a first down-sampled image and a second down-sampled image corresponding to the super-resolution image.

[0074] In this embodiment, the electronic device may downsample the super-resolution image using a first downsampling method to obtain a first downsampled image, and may downsample the super-resolution image using a second downsampling method to obtain a second downsampled image, wherein the sampling magnification of the first downsampling method is different from the sampling magnification of the second downsampling method.

[0075] Taking the sampling magnification of the first sampling method as 2 and the sampling magnification of the second sampling method as 4 as an example, the electronic device may downsample the x4 super-resolution image corresponding to each sample video frame image generated by the generator network at a sampling magnification of 2 to obtain a first downsampled image corresponding to each x4 super-resolution image, and downsample the generated x4 super-resolution image corresponding to each sample video frame image at a sampling magnification of 4 to obtain a second downsampled image corresponding to each x4 super-resolution image. Since the x4 super-resolution image is an HxW SR video frame image, the x4 super-resolution image after downsampling at a sampling magnification of 2 may be denoted as an H / 2xW / 2 low-resolution video frame image, and the x4 super-resolution image after downsampling at a sampling magnification of 4 may be denoted as an H / 4xW / 4 low-resolution video frame image.

[0076] Typically, for a x4-scale scenario, the HxW SR video frame image generated by the generator network, as well as its downsampled H / 2xW / 2 low-resolution video frame image and H / 4xW / 4 low-resolution video frame image, should all be true. Subsequently, in the discriminator network, to ensure that the loss of the video super-resolution model remains unchanged, the discriminator network can add and average the three multi-level structures. The three scales provide images of three different resolutions. Mixing the information at each level yields a richer high-resolution image.

[0077] After executing step S2300 above to determine the first down-sampled image and the second down-sampled image corresponding to the super-resolution image, proceed to:

[0078] Step S2400: Input the super-resolution image, the first down-sampled image, and the second down-sampled image into the discriminator network, and generate a first discrimination result based on the super-resolution image, the first down-sampled image, and the second down-sampled image through the discriminator network.

[0079] In this embodiment, the electronic device can input the super-resolution image corresponding to each sample video frame image, and the first down-sampled image and the second down-sampled image corresponding to the super-resolution image into the discriminator network, and generate a first discrimination result based on the super-resolution image, the first down-sampled image and the second down-sampled image through the discriminator network.

[0080] Specifically, the electronic device can input the super-resolution image corresponding to each sample video frame image, and the first down-sampled image and the second down-sampled image corresponding to the super-resolution image into the discriminator network. The discriminator network obtains the average value of the super-resolution image, the first down-sampled image and the second down-sampled image, and generates a first discrimination result based on the average value.

[0081] In other words, the three scales provide images of three different resolutions. By mixing information at all levels through the discriminator network, richer high-resolution images can be obtained, achieving better model training results.

[0082] After executing step S2400 above to input the super-resolution image, the first down-sampled image, and the second down-sampled image into the discriminator network, and generating a first discrimination result based on the super-resolution image, the first down-sampled image, and the second down-sampled image through the discriminator network, proceed to:

[0083] Step S2500: constructing a loss of the video super-resolution model according to the first discrimination result, and the loss is used to adjust model parameters of the video super-resolution model.

[0084] The loss includes a first loss of the generator network and a second loss of the discriminator network. The model parameters include first network parameters of the generator network and second network parameters of the discriminator network. The first loss is used to adjust the first network parameters, and the second loss is used to adjust the second network parameters. The first loss typically includes L1 loss and perceptual loss, and the second loss typically includes GAN loss.

[0085] In this embodiment, the electronic device can construct a first loss of the generator network and a second loss of the discriminator network based on the first discrimination result, and then adjust the first network parameters of the generator network based on the first loss, and adjust the second network parameters of the discriminator network based on the second loss.

[0086] According to the method of the embodiment of the present disclosure, the video super-resolution model includes a generator network and a discriminator network. The generator network can generate a super-resolution image corresponding to a sample video frame image. The input of the discriminator network includes a super-resolution image, a first down-sampled image corresponding to the super-resolution image, and a second down-sampled image. Based on the super-resolution image, the first down-sampled image and the second down-sampled image corresponding to the super-resolution image, a first discrimination result can be generated, and then the loss of the video super-resolution model is constructed based on the first discrimination result. In other words, through the embodiment of the present disclosure, it uses three images of different resolutions as input to the discriminator network, so that the three images of different resolutions are mixed through the discriminator network to obtain richer high-resolution images, thereby achieving better model training effects.

[0087] In one embodiment, the video super-resolution model training method of the embodiment of the present disclosure further includes the following steps S3100 to S3300:

[0088] Step S3100: Input the sample video frame image into the generator network to generate the optical flow corresponding to the super-resolution image.

[0089] In this embodiment, the generator network includes a convolutional network for calculating optical flow, through which the optical flow corresponding to the super-resolution image can be generated, which is referred to as SR optical flow.

[0090] Step S3200: Obtain the GT (Ground Truth) optical flow corresponding to the sample video frame image.

[0091] Among them, GT optical flow is the real optical flow corresponding to the super-resolution image.

[0092] Step S3300: Input the optical flow corresponding to the super-resolution image and the GT optical flow into the discriminator network to generate a first optical flow map of the optical flow corresponding to the super-resolution image and a second optical flow map of the GT optical flow, and generate a second discrimination result based on the first optical flow map and the second optical flow map through the discriminator network.

[0093] Optionally, this step S3300 inputs the optical flow corresponding to the super-resolution image and the GT optical flow into the discriminator network to generate a first optical flow map of the optical flow corresponding to the super-resolution image and a second optical flow map of the GT optical flow. It can further include: inputting the optical flow corresponding to the super-resolution image and the GT optical flow into the discriminator network, processing the optical flow corresponding to the super-resolution image and the GT optical flow respectively through the optical flow expert algorithm to generate a first optical flow map of the optical flow corresponding to the super-resolution image and a second optical flow map of the GT optical flow.

[0094] The optical flow expert algorithm may be a SPYNet algorithm.

[0095] Specifically, the discriminator network can process the SR optical flow and GT optical flow corresponding to the super-resolution image through the optical flow expert algorithm, generate a first optical flow map of the SR optical flow corresponding to the super-resolution image, and a second optical flow map of the GT optical flow, and generate a second discrimination result based on the first optical flow map and the second optical flow map.

[0096] Correspondingly, the above-mentioned step S2300 of constructing the loss of the video super-resolution model according to the first discrimination result may further include: constructing the loss of the video super-resolution model according to the first discrimination result and the second discrimination result.

[0097] Specifically, refer to Figure 3The electronic device can input the super-resolution image corresponding to each sample video frame image, as well as the first and second down-sampled images corresponding to the super-resolution image, into the discriminator network. The discriminator network can also input the SR optical flow and GT optical flow corresponding to each super-resolution image. The discriminator network can obtain the average of the super-resolution image, the first and second down-sampled images, and generate a first discrimination result based on the average. The optical flow expert algorithm in the discriminator network can process the SR optical flow and GT optical flow corresponding to the super-resolution image, generating a first optical flow map of the SR optical flow and a second optical flow map of the GT optical flow corresponding to the super-resolution image. The discriminator network can then generate a second discrimination result based on the first and second optical flow maps. The electronic device can further construct a first loss for the generator network and a second loss for the discriminator network based on the first and second discrimination results. The first network parameters of the generator network can be adjusted based on the first loss, and the second network parameters of the discriminator network can be adjusted based on the second loss, ultimately achieving better model training results.

[0098] According to this embodiment, the input of the generator network not only includes images of three different resolutions so that richer high-resolution images can be obtained by mixing information at all levels, but also the input of the generator network also includes the SR optical flow and GT optical flow corresponding to the super-resolution image, so as to increase the accuracy of the SR optical flow generated by the generator network, thereby achieving better model training effects.

[0099] <Method Example 2>

[0100] In this embodiment, a video super-resolution method is provided. The video super-resolution method can be implemented by an electronic device, which can be Figure 1 The electronic device 1000 is shown.

[0101] according to Figure 4 As shown, the video super-resolution method according to the embodiment of the present disclosure may include the following steps S4100 to S4200.

[0102] Step S4100: Obtain the video frame image to be processed.

[0103] The video frame image to be processed may be any video frame image that needs to be super-resolution processed.

[0104] Step S4200: Input the video frame image to be processed into a preset video super-resolution model to obtain a super-resolution image corresponding to the video frame image to be processed.

[0105] The video super-resolution model is obtained according to the first embodiment of the above method.

[0106] In this embodiment, the video frame image to be processed is input into a trained video super-resolution model, and a super-resolution image corresponding to the video frame image to be processed can be generated through the generator network in the video super-resolution model.

[0107] Experiments show that the self-developed face super-resolution algorithm can obtain facial effects that are consistent with the real world, and the problems of artifacts and blur are effectively solved.

[0108] According to the embodiment of the present disclosure, the video super-resolution model includes a generator network and a discriminator network. The generator network can generate a super-resolution image corresponding to a sample video frame image. The input of the discriminator network includes a super-resolution image, a first down-sampled image corresponding to the super-resolution image, and a second down-sampled image. Based on the super-resolution image, the first down-sampled image and the second down-sampled image corresponding to the super-resolution image, a first discrimination result can be generated, and then the loss of the video super-resolution model is constructed based on the first discrimination result. That is to say, according to the embodiment of the present disclosure, images of three different resolutions are used as inputs of the discriminator network, so that the three images of different resolutions are mixed through the discriminator network to obtain richer high-resolution images, thereby achieving better model training effects. In this way, the super-resolution video frame images obtained by the video super-resolution model have better visual effects.

[0109] <Device Example 1>

[0110] In this embodiment, a video super-resolution model training device 5000 is provided. Figure 5 As shown, the video super-resolution model training device 5000 may include a first acquisition module 5100, a generation module 5200, a determination module 5300, a discrimination module 5400 and a construction module 5500. The video super-resolution model includes a generator network and a discriminator network. The video super-resolution model training method includes at least one round of training process, and any round of the training process includes:

[0111] A first acquisition module 5100 is used to acquire a sample video frame image;

[0112] A generating module 5200 is configured to input the sample video frame image into the generator network to generate a super-resolution image corresponding to the sample video frame image;

[0113] A determination module 5300 is configured to determine a first down-sampled image and a second down-sampled image corresponding to the super-resolution image;

[0114] a discriminant module 5400, configured to input the super-resolution image, the first down-sampled image, and the second down-sampled image into the discriminator network, and generate a first discrimination result based on the super-resolution image, the first down-sampled image, and the second down-sampled image through the discriminator network;

[0115] The construction module 5500 is used to construct the loss of the video super-resolution model, and the loss is used to adjust the model parameters of the video super-resolution model.

[0116] In one embodiment, the generating module 5200 is further configured to input the sample video frame image into the generator network to generate an optical flow corresponding to the super-resolution image; and obtain the GT optical flow corresponding to the sample video frame image;

[0117] The discriminant module 5400 is further used to input the optical flow corresponding to the super-resolution image and the GT optical flow into the discriminator network, generate a first optical flow map of the optical flow corresponding to the super-resolution image, and a second optical flow map of the GT optical flow, and generate a second discrimination result based on the first optical flow map and the second optical flow map through the discriminator network.

[0118] In one embodiment, the discrimination module 5400 is specifically configured to construct a loss of the video super-resolution model according to the first discrimination result and the second discrimination result.

[0119] In one embodiment, the discriminant module 5400 is specifically used to input the optical flow corresponding to the super-resolution image and the GT optical flow into the discriminator network, and process the optical flow corresponding to the super-resolution image and the GT optical flow respectively through the optical flow expert algorithm to generate a first optical flow map of the optical flow corresponding to the super-resolution image and a second optical flow map of the GT optical flow.

[0120] In one embodiment, the loss includes a first loss of the generator network and a second loss of the discriminator network, the model parameters include a first network parameter of the generator network and a second network parameter of the discriminator network, the first loss is used to adjust the first network parameter, and the second loss is used to adjust the second network parameter.

[0121] According to this embodiment, the video super-resolution model includes a generator network and a discriminator network. The generator network can generate a super-resolution image corresponding to a sample video frame image. The input of the discriminator network includes a super-resolution image, a first down-sampled image corresponding to the super-resolution image, and a second down-sampled image. Based on the super-resolution image, the first down-sampled image, and the second down-sampled image corresponding to the super-resolution image, a first discrimination result can be generated, and then the loss of the video super-resolution model is constructed based on the first discrimination result. In other words, through the embodiment of the present disclosure, it uses three images of different resolutions as input to the discriminator network, so that the three images of different resolutions are mixed through the discriminator network to obtain richer high-resolution images, thereby achieving better model training effects.

[0122] <Device Example 2>

[0123] In this embodiment, a video super-resolution device 6000 is provided. Figure 6 As shown, the video super-resolution device 6000 may include a second acquisition module 6100 and a processing module 6200 .

[0124] A second acquisition module is used to acquire the image to be processed;

[0125] A processing module, configured to input the image to be processed into a preset video super-resolution model to obtain a super-resolution image corresponding to the image to be processed;

[0126] The video super-resolution model is obtained according to the first embodiment of the above method.

[0127] According to this embodiment, the video super-resolution model includes a generator network and a discriminator network. The generator network can generate a super-resolution image corresponding to a sample video frame image. The input of the discriminator network includes a super-resolution image, a first down-sampled image corresponding to the super-resolution image, and a second down-sampled image. Based on the super-resolution image, the first down-sampled image and the second down-sampled image corresponding to the super-resolution image, a first discrimination result can be generated, and then the loss of the video super-resolution model is constructed based on the first discrimination result. That is to say, through the embodiment of the present disclosure, it uses three images of different resolutions as the input of the discriminator network, so that the three images of different resolutions are mixed through the discriminator network to obtain richer high-resolution images, thereby achieving better model training effects. In this way, the super-resolution video frame images obtained by the video super-resolution model have better visual effects.

[0128] <Equipment Example>

[0129] Figure 7 FIG. 1 is a schematic diagram of the hardware structure of an electronic device according to an embodiment. Figure 7As shown, the electronic device 7000 includes a processor 7100 and a memory 7200 .

[0130] The memory 7200 can be used to store executable computer instructions.

[0131] The processor 7100 can be used to execute the video super-resolution model training method described in Example 1 of the method of the present disclosure or the video super-resolution method described in Example 2 of the method according to the control of the executable computer instructions.

[0132] The electronic device 7000 may be Figure 1 The electronic device 1000 shown may also be a device with other hardware structures, which is not limited here.

[0133] In another embodiment, the electronic device 7000 may include the video super-resolution model training device 5000 described in the first device embodiment or the video super-resolution device 6000 described in the second device embodiment.

[0134] In one embodiment, each module of the above video super-resolution model training device 5000 and video super-resolution device 6000 can be implemented by the processor 7100 running computer instructions stored in the memory 7200.

[0135] <Computer-readable storage medium>

[0136] The embodiments of the present disclosure further provide a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, execute the video super-resolution model training method provided in the above method embodiment 1 or the video super-resolution method provided in the above method embodiment 2.

[0137] The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0138] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0139] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0140] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0141] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0142] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0143] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0144] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0145] While various embodiments of the present disclosure have been described above, the foregoing description is illustrative and non-exhaustive, and is not intended to limit the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles, practical applications, or market-leading technological improvements of the various embodiments, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the present disclosure is defined by the appended claims.

Claims

1. A video super-resolution model training method, characterized in that: The video super-resolution model includes a generator network and a discriminator network. The video super-resolution model training method includes at least one round of training process. Any round of the training process includes: Get sample video frame images; Inputting the sample video frame image into the generator network to generate a super-resolution image corresponding to the sample video frame image; Determine a first down-sampled image and a second down-sampled image corresponding to the super-resolution image; Inputting the super-resolution image, the first down-sampled image, and the second down-sampled image into the discriminator network, and generating a first discrimination result based on the super-resolution image, the first down-sampled image, and the second down-sampled image through the discriminator network; According to the first discrimination result, a loss of the video super-resolution model is constructed, and the loss is used to adjust model parameters of the video super-resolution model.

2. The method according to claim 1, characterized in that The method further comprises: Inputting the sample video frame image into the generator network to generate an optical flow corresponding to the super-resolution image; Obtaining the GT optical flow corresponding to the sample video frame image; The optical flow corresponding to the super-resolution image and the GT optical flow are input into the discriminator network to generate a first optical flow map of the optical flow corresponding to the super-resolution image and a second optical flow map of the GT optical flow, and a second discrimination result is generated based on the first optical flow map and the second optical flow map through the discriminator network.

3. The method according to claim 2, characterized in that The step of constructing the loss of the video super-resolution model according to the first discrimination result includes: The loss of the video super-resolution model is constructed according to the first discrimination result and the second discrimination result.

4. The method according to claim 2, characterized in that: The step of inputting the optical flow corresponding to the super-resolution image and the GT optical flow into the discriminator network to generate a first optical flow map of the optical flow corresponding to the super-resolution image and a second optical flow map of the GT optical flow includes: The optical flow corresponding to the super-resolution image and the GT optical flow are input into the discriminator network, and the optical flow corresponding to the super-resolution image and the GT optical flow are processed respectively by the optical flow expert algorithm to generate a first optical flow map of the optical flow corresponding to the super-resolution image and a second optical flow map of the GT optical flow.

5. The method according to claim 2, characterized in that: The loss includes a first loss of the generator network and a second loss of the discriminator network, the model parameters include a first network parameter of the generator network and a second network parameter of the discriminator network, the first loss is used to adjust the first network parameter, and the second loss is used to adjust the second network parameter.

6. A video super-resolution method, characterized in that: The method comprises: Obtain the video frame image to be processed; Inputting the to-be-processed video frame image into a preset video super-resolution model to obtain a super-resolution image corresponding to the to-be-processed video frame image; The video super-resolution model is a video super-resolution model obtained according to any one of claims 1 to 5.

7. A video super-resolution model training device, characterized in that: The video super-resolution model includes a generator network and a discriminator network. The video super-resolution model training method includes at least one round of training process. Any round of the training process includes: A first acquisition module, used to acquire a sample video frame image; A generation module, used for inputting the sample video frame image into the generator network to generate a super-resolution image corresponding to the sample video frame image; A determination module, configured to determine a first down-sampled image and a second down-sampled image corresponding to the super-resolution image; a discriminant module, configured to input the super-resolution image, the first down-sampled image, and the second down-sampled image into the discriminator network, and generate a first discriminant result based on the super-resolution image, the first down-sampled image, and the second down-sampled image through the discriminator network; A construction module is used to construct the loss of the video super-resolution model according to the first discrimination result, and the loss is used to adjust the model parameters of the video super-resolution model.

8. A video super-resolution device, characterized in that: The device comprises: A second acquisition module, used for acquiring an image to be processed; A processing module, used for inputting the image to be processed into a preset video super-resolution model to obtain a super-resolution image corresponding to the image to be processed; The video super-resolution model is a video super-resolution model obtained according to any one of claims 1 to 5.

9. An electronic device, characterized in that: include: Memory for storing executable computer instructions; A processor, configured to execute the video super-resolution model training method according to any one of claims 1 to 5, or execute the video super-resolution method according to claim 6, under the control of the executable computer instructions.

10. A computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions, when executed by a processor, execute the video super-resolution model training method described in any one of claims 1 to 5, or execute the video super-resolution method described in claim 6.

Images