Image super-resolution processing method, device and storage medium

Abstract

The present disclosure relates to an image super-resolution processing method, device and storage medium. The method includes: determining an image to be processed; processing the image to be processed through a neural network model to obtain a reconstructed image; wherein the processing includes extracting features from edge information of the image to be processed through a reparameterization module of the neural network model, the edge information includes edge information of the image to be processed in an inclined direction, and the resolution of the reconstructed image is greater than that of the image to be processed. According to an embodiment of the present application, the image to be processed is processed through a neural network model to obtain a reconstructed image, so that the resolution of the reconstructed image is greater than that of the image to be processed, thereby realizing super-resolution processing of the image, wherein the edge information of the image to be processed in an inclined direction is extracted through the reparameterization module of the neural network model, so that the reconstruction quality of the image can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of deep learning, and in particular to an image super-resolution processing method, device and storage medium. Background Art

[0002] Image super-resolution technology in the field of deep learning is a technology that uses a neural network model to improve the resolution of an image. Super-resolution technology can upsample a low-resolution image to a high-resolution image. Super-resolution technology is widely used in image processing, video reconstruction, and visual enhancement. It can be used to improve image quality, enhance detail information, and improve image details. The image reconstruction quality in the current image super-resolution processing solution is poor. Therefore, a new image super-resolution processing method is urgently needed to improve the image reconstruction quality. Summary of the invention

[0003] In view of this, the present disclosure proposes an image super-resolution processing method, device and storage medium.

[0004] According to one aspect of the present disclosure, a method for image super-resolution processing is provided. The method comprises:

[0005] Determine the image to be processed;

[0006] The image to be processed is processed through a neural network model to obtain a reconstructed image;

[0007] The processing includes extracting features from edge information of the image to be processed through a reparameterization module of a neural network model, the edge information includes edge information of the image to be processed in an inclined direction, and the resolution of the reconstructed image is greater than that of the image to be processed.

[0008] In a possible implementation, the parameters of the re-parameterization module are obtained by fusing the weights of preset operators, where the preset operators include operators for extracting features of edge information of the image to be processed in an oblique direction.

[0009] In a possible implementation manner, the image to be processed includes a stitched image, and determining the image to be processed includes:

[0010] A spliced ​​image is determined based on a current frame image, a frame image before the current frame image, and a motion vector from the previous frame image to the current frame image.

[0011] In a possible implementation, determining a spliced ​​image based on a current frame image, a frame image before the current frame image, and a motion vector from the previous frame image to the current frame image includes:

[0012] Perform motion compensation on the previous frame image using the motion vector to determine the compensated image;

[0013] The compensated image and the current frame image are spliced ​​to determine a spliced ​​image.

[0014] In a possible implementation, the method further includes:

[0015] Sharpening is performed based on the reconstructed image to obtain a sharpened image, and the clarity of the sharpened image is greater than that of the reconstructed image.

[0016] In a possible implementation, the image to be processed includes a current frame image, and the image to be processed is processed by a neural network model to obtain a reconstructed image, including:

[0017] The spliced ​​image is processed through a neural network model to obtain a processed image;

[0018] Through the neural network model, the current frame image is upsampled to obtain an upsampled image;

[0019] Through the neural network model, a reconstructed image is obtained based on the processed image and the upsampled image.

[0020] In a possible implementation, the neural network model includes a reparameterized network and a first upsampling module. The spliced ​​image is processed by the neural network model to obtain a processed image, including:

[0021] The spliced ​​image is processed using a reparameterized network to obtain a processed image;

[0022] Through the neural network model, the current frame image is upsampled to obtain the upsampled image, including:

[0023] The first up-sampling module is used to perform up-sampling processing on the current frame image to obtain an up-sampled image.

[0024] In a possible implementation, the reparameterized network includes a reparameterized module, a first reparameterized residual module, a second reparameterized residual module, a third reparameterized residual module, a fourth reparameterized residual module, a first activation module, a second activation module, a convolution module, and a second upsampling module. The first reparameterized residual module, the second reparameterized residual module, the third reparameterized residual module, and the fourth reparameterized residual module are in a cascade relationship. The spliced ​​image is processed using the reparameterized network to obtain a processed image, including:

[0025] Processing the spliced ​​image using the reparameterization module and the first activation module to obtain a first intermediate image;

[0026] Using a first parameterized residual module to extract features from the first intermediate image, to obtain a second intermediate image;

[0027] Using a second parameterized residual module to extract features from the second intermediate image, to obtain a third intermediate image;

[0028] extracting features from the third intermediate image using a third parameterized residual module to obtain a fourth intermediate image;

[0029] Using a fourth parameterized residual module to extract features from the second intermediate image and the fourth intermediate image, to obtain a fifth intermediate image;

[0030] The first intermediate image and the fifth intermediate image are processed by using a convolution module, a second activation module, and a second upsampling module to obtain processed images.

[0031] In a possible implementation, the first parameterized residual module, the second parameterized residual module, the third parameterized residual module, and the fourth parameterized residual module are residual structures, respectively consisting of a re-parameterized module and an activation module.

[0032] In a possible implementation, the weight of the preset operator is obtained in a training phase of the neural network model, the neural network model is a trained neural network model, and the method further includes:

[0033] Iteratively train the initial neural network model using training data;

[0034] After every preset number of iterations, the performance of the neural network model after iterative training is evaluated using test data;

[0035] The iteratively trained neural network model with the best performance evaluation result is used as the trained neural network model.

[0036] In a possible implementation, the preset operator also includes one or more of the following: a Laplace operator, an operator for extracting features of edge information of the image to be processed in the horizontal direction, and an operator for extracting features of edge information of the image to be processed in the vertical direction.

[0037] According to another aspect of the present disclosure, an image super-resolution processing device is provided. The device comprises:

[0038] A first determining module, used to determine an image to be processed;

[0039] The second determination module is used to process the image to be processed through a neural network model to obtain a reconstructed image;

[0040] The processing includes extracting features from edge information of the image to be processed through a reparameterization module of a neural network model, the edge information includes edge information of the image to be processed in an inclined direction, and the resolution of the reconstructed image is greater than that of the image to be processed.

[0041] In a possible implementation, the parameters of the re-parameterization module are obtained by fusing the weights of preset operators, where the preset operators include operators for extracting features of edge information of the image to be processed in an oblique direction.

[0042] In a possible implementation, the image to be processed includes a spliced ​​image, and the first determining module is used to:

[0043] A spliced ​​image is determined based on a current frame image, a frame image before the current frame image, and a motion vector from the previous frame image to the current frame image.

[0044] In a possible implementation, determining a spliced ​​image based on a current frame image, a frame image before the current frame image, and a motion vector from the previous frame image to the current frame image includes:

[0045] Perform motion compensation on the previous frame image using the motion vector to determine the compensated image;

[0046] The compensated image and the current frame image are spliced ​​to determine a spliced ​​image.

[0047] In a possible implementation, the device further includes:

[0048] The sharpening module is used to perform sharpening based on the reconstructed image to obtain a sharpened image, and the clarity of the sharpened image is greater than that of the reconstructed image.

[0049] In a possible implementation manner, the image to be processed includes a current frame image, and the second determining module is used to:

[0050] The spliced ​​image is processed through a neural network model to obtain a processed image;

[0051] Through the neural network model, the current frame image is upsampled to obtain an upsampled image;

[0052] Through the neural network model, a reconstructed image is obtained based on the processed image and the upsampled image.

[0053] In a possible implementation, the neural network model includes a reparameterized network and a first upsampling module. The spliced ​​image is processed by the neural network model to obtain a processed image, including:

[0054] The spliced ​​image is processed using a reparameterized network to obtain a processed image;

[0055] Through the neural network model, the current frame image is upsampled to obtain the upsampled image, including:

[0056] The first up-sampling module is used to perform up-sampling processing on the current frame image to obtain an up-sampled image.

[0057] In a possible implementation, the reparameterized network includes a reparameterized module, a first reparameterized residual module, a second reparameterized residual module, a third reparameterized residual module, a fourth reparameterized residual module, a first activation module, a second activation module, a convolution module, and a second upsampling module. The first reparameterized residual module, the second reparameterized residual module, the third reparameterized residual module, and the fourth reparameterized residual module are in a cascade relationship. The spliced ​​image is processed using the reparameterized network to obtain a processed image, including:

[0058] Processing the spliced ​​image using the reparameterization module and the first activation module to obtain a first intermediate image;

[0059] Using a first parameterized residual module to extract features from the first intermediate image, to obtain a second intermediate image;

[0060] Using a second parameterized residual module to extract features from the second intermediate image, to obtain a third intermediate image;

[0061] extracting features from the third intermediate image using a third parameterized residual module to obtain a fourth intermediate image;

[0062] Using a fourth parameterized residual module to extract features from the second intermediate image and the fourth intermediate image, to obtain a fifth intermediate image;

[0063] The first intermediate image and the fifth intermediate image are processed by using a convolution module, a second activation module, and a second upsampling module to obtain processed images.

[0064] In a possible implementation, the first parameterized residual module, the second parameterized residual module, the third parameterized residual module, and the fourth parameterized residual module are residual structures, respectively consisting of a re-parameterized module and an activation module.

[0065] In a possible implementation, the weight of the preset operator is obtained in the training phase of the neural network model, the neural network model is a trained neural network model, and the device further includes:

[0066] An iterative training module, used to iteratively train the initial neural network model using training data;

[0067] A performance evaluation module is used to evaluate the performance of the neural network model after iterative training using test data every preset number of iterative training times;

[0068] The third determination module is used to use the iteratively trained neural network model with the best performance evaluation result as the trained neural network model.

[0069] In a possible implementation, the preset operator also includes one or more of the following: a Laplace operator, an operator for extracting features of edge information of the image to be processed in the horizontal direction, and an operator for extracting features of edge information of the image to be processed in the vertical direction.

[0070] According to another aspect of the present disclosure, there is provided an image super-resolution processing device, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0071] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0072] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0073] According to an embodiment of the present application, the image to be processed is processed by a neural network model to obtain a reconstructed image, so that the resolution of the reconstructed image is greater than that of the image to be processed, thereby realizing super-resolution processing of the image, wherein the edge information of the image to be processed in the inclined direction is extracted through the re-parameterization module of the neural network model, thereby improving the reconstruction quality of the image.

[0074] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0076] Figure 1 A schematic diagram showing an application scenario according to an embodiment of the present application.

[0077] Figure 2 A flowchart of an image super-resolution processing method according to an embodiment of the present application is shown.

[0078] Figure 3 A schematic diagram of the structure of a re-parameterized network according to an embodiment of the present application is shown.

[0079] Figure 4 A schematic diagram of the structure of a re-parameterized residual module according to an embodiment of the present application is shown.

[0080] Figure 5 A schematic diagram of the structure of a re-parameterization module according to an embodiment of the present application is shown.

[0081] Figure 6 A schematic diagram of a preset operator according to an embodiment of the present application is shown.

[0082] Figure 7 A flowchart of an image super-resolution processing method according to an embodiment of the present application is shown.

[0083] Figure 8 An overall flow chart of image super-resolution processing according to an embodiment of the present application is shown.

[0084] Fig. 9 A structural diagram of an image super-resolution processing device according to an embodiment of the present application is shown.

[0085] Fig.10 It is a block diagram of a device 1900 for image super-resolution processing according to an exemplary embodiment. DETAILED DESCRIPTION

[0086] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0087] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0088] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0089] Image super-resolution technology (super-resolution for short) in the field of deep learning is a technology that uses a neural network model to improve the resolution of an image. Super-resolution technology can upsample a low-resolution image to a high-resolution image. Super-resolution technology is widely used in image processing, video reconstruction, and visual enhancement. It can be used to improve image quality, enhance detail information, and improve image details. The image reconstruction quality in the current image super-resolution processing scheme is poor. Therefore, a new image super-resolution processing method is urgently needed to improve the image reconstruction quality.

[0090] In view of this, the present application provides an image super-resolution processing method. The method of the embodiment of the present application processes the image to be processed through a neural network model to obtain a reconstructed image, so that the resolution of the reconstructed image is greater than that of the image to be processed, and super-resolution processing of the image is achieved, wherein the edge information of the image to be processed in the inclined direction is extracted by the re-parameterization module of the neural network model, which can improve the reconstruction quality of the image.

[0091] Figure 1 A schematic diagram of an application scenario according to an embodiment of the present application is shown. For example, in a game scenario, a graphics processing unit (GPU) has to render each frame of the game screen. If the rendering resolution is higher, the rendering load of the GPU will be greater, which will cause the frame rate of the GPU rendering, that is, the number of frames per second (FPS) to drop, resulting in problems such as screen freeze and unsmoothness, which reduces the gaming experience. The method of the embodiment of the present application can be used in the above-mentioned game scenarios, such as Figure 1 As shown, a low-resolution game screen can be obtained by GPU rendering, and then a high-resolution game screen (for example, twice the original resolution) can be reconstructed through the image super-resolution processing method of the embodiment of the present application. In this scenario, the GPU rendering load is reduced, the FPS of the game is also improved, and the fluency is better.

[0092] The method of the embodiment of the present application can be used for a server or a terminal device. The terminal device can be any one or more of a mobile phone, a foldable electronic device, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), and a vehicle-mounted device. The embodiment of the present application does not impose any special restrictions on the specific type of the terminal device, and the terminal device can have a wired or wireless communication function.

[0093] The server can be located locally or in the cloud, and can be a physical device or a virtual device, such as a virtual machine, a container, etc., and has a wireless communication function, wherein the wireless communication function can be set in the chip (system) or other parts or components of the server. The wireless communication function can be realized, for example, through mobile communication technologies such as 2G / 3G / 4G / 5G, as well as Wi-Fi, Bluetooth, frequency modulation (FM), digital radio, satellite communication, etc. Communication can also be carried out through wired connection to achieve interaction with other devices.

[0094] The following Figure 2-Figure 8 , the image super-resolution processing method of the embodiment of the present application is introduced.

[0095] Figure 2 FIG. 1 is a flow chart showing an image super-resolution processing method according to an embodiment of the present application. The method can be used in a terminal device or a server, such as Figure 2 As shown, the method may include:

[0096] Step S201, determining an image to be processed.

[0097] The image to be processed may include a current frame image to be processed, such as a current frame in a sequence of images to be processed.

[0098] Furthermore, in the process of image super-resolution processing, simple single-frame super-resolution (i.e., the image of the input model is a single frame) is prone to cause incoherence between frames, resulting in low image reconstruction quality, while multi-frame super-resolution (i.e., the image of the input model is a continuous multi-frame) will result in excessive calculation and bring about multi-frame delay problems. In order to improve the reconstruction quality while ensuring the coherence between reconstructed frames, in an embodiment of the present application, the image to be processed may include information between the previous and next frames, and the image information of the previous frame is fused into the super-resolution processing process of the current frame image, thereby increasing context information.

[0099] The image to be processed may also include a spliced ​​image, wherein S201 includes:

[0100] A spliced ​​image is determined based on a current frame image, a frame image before the current frame image, and a motion vector from the previous frame image to the current frame image.

[0101] The current frame image is the image to be subjected to image super-resolution processing, and may be an image with a lower resolution. The motion vector from the previous frame image to the current frame image may be obtained based on rendering technology.

[0102] Among them, you can:

[0103] Perform motion compensation on the previous frame image using the motion vector to determine the compensated image;

[0104] The compensated image and the current frame image are spliced ​​to determine a spliced ​​image.

[0105] Based on the optical flow transformation (warp), the GridSample operator can be used to perform motion compensation on the previous frame image using the motion vector to determine the compensated image. If the number of channels of the compensated image is 3 and the number of channels of the current frame image is 3, after the compensated image and the current frame image are spliced, the number of channels of the spliced ​​image is 6. By making the input of the model a single-frame image, but integrating the information of the previous frame image into it, better reconstruction quality can be obtained while improving the image reconstruction speed, ensuring the real-time performance of super-resolution processing.

[0106] Step S202: Process the image to be processed by a neural network model to obtain a reconstructed image.

[0107] The processing may include extracting features from edge information of the image to be processed by a reparameterization module of the neural network model, and the edge information may include edge information of the image to be processed in an inclined direction, and the inclined direction may include an inclined direction of a preset angle, such as a 135° and a 45° direction. The resolution of the reconstructed image is greater than that of the image to be processed. For example, the resolution of the reconstructed image may be twice that of the image to be processed. The above-mentioned image to be processed may refer to a spliced ​​image in the image to be processed.

[0108] According to an embodiment of the present application, the image to be processed is processed by a neural network model to obtain a reconstructed image, so that the resolution of the reconstructed image is greater than that of the image to be processed, thereby realizing super-resolution processing of the image, wherein the edge information of the image to be processed in the inclined direction is extracted through the re-parameterization module of the neural network model, thereby improving the reconstruction quality of the image.

[0109] In order to reduce the complexity of learning the neural network model so that it only learns the detail residual part of the reconstructed image, in the above step S202, the image to be processed can be processed by the neural network model to obtain a processed image;

[0110] The spliced ​​image is processed through a neural network model to obtain a processed image;

[0111] Through the neural network model, the current frame image is upsampled to obtain an upsampled image;

[0112] Through the neural network model, a reconstructed image is obtained based on the processed image and the upsampled image.

[0113] The upsampling may be up-linear interpolation upsampling, and the processed image and the up-sampled image may be added to obtain a reconstructed image.

[0114] The structure of the neural network model is introduced below. The neural network model of the embodiment of the present application may include a reparameterized network and a first upsampling module. The structure of the reparameterized network can be found in Figure 3 , showing a schematic diagram of the structure of a re-parameterized network according to an embodiment of the present application. Figure 3 As shown, the input of the reparameterized network can be a spliced ​​image, and the output can be a processed image. The neural network model may include a reparameterized module (such as IECB in the figure), a first reparameterized residual module (such as IECBResBlock1 in the figure), a second reparameterized residual module (such as IECBResBlock2 in the figure), a third reparameterized residual module (such as IECBResBlock3 in the figure), a fourth reparameterized residual module (such as IECBResBlock4 in the figure), a first activation module (such as Leaky-ReLU1 in the figure), a second activation module (such as Leaky-ReLU2 in the figure), a convolution module (such as 3x3 conv in the figure), and a second upsampling module (such as pixel-shuffle in the figure).

[0115] In the figure, the structures of IECBResBlock1 to IECBResBlock4 can be the same, and the structures of Leaky-ReLU1 and Leaky-ReLU2 can also be the same.

[0116] Among them, the number of channels C1 of the input image of the reparameterization module can be 6, and the number of channels C2 of the output image can be 16. C2 can represent the number of intermediate layer channels of the neural network model, and its size can be adjusted as needed to control the number of parameters and calculation amount of the model.

[0117] The activation module can be a non-linear activation function.

[0118] The first parameterized residual module, the second parameterized residual module, the third parameterized residual module, and the fourth parameterized residual module are in a cascade relationship (such as the U-shaped structure in the figure), which can reduce the difficulty of intermediate feature extraction. The number of input and output channels between the first parameterized residual module to the fourth parameterized residual module is C2.

[0119] Through the convolution module (the convolution module can be used to perform a 3x3 convolution operation), the number of channels can be converted from C2 to C3 (for example, 12). The second upsampling module can cooperate with the convolution module to upsample the output of the activation module, and finally output a processed image with a channel number of 3.

[0120] Figure 4The structure diagram of the re-parameterized residual module according to an embodiment of the present application is shown. The first re-parameterized residual module, the second re-parameterized residual module, the third re-parameterized residual module, and the fourth re-parameterized residual module are residual structures, respectively, and are composed of a re-parameterized module and an activation module, such as Figure 4 As shown, Figure 3 IECBResBlock1~IECBResBlock4 in the figure are composed of two IECBs (i.e., reparameterization modules) and LeakyReLU (i.e., activation modules), in which the input of the first IECB is added to the output of the second IECB as the overall output of the reparameterized residual module, thereby forming a residual structure. The residual structure makes the neural network model more stable during the training process.

[0121] In the embodiment of the present application, it can be divided into a training phase and an inference phase. The above neural network model can be obtained through training, which is a trained neural network model. In the training phase and the inference phase, the structure of the re-parameterization module IECB can be different. Figure 5 , showing a structural diagram of a re-parameterization module according to an embodiment of the present application.

[0122] like Figure 5 The left half shows the structure of the re-parameterized module in the training phase. It may include multiple 3x3 convolution modules (such as 3x3 conv in the figure) and multiple 1x1 convolution modules (such as 1x1 conv in the figure), and may also include a 3x3 convolution module composed of preset operators. Among them, the preset operator may include an operator for feature extraction of edge information of the image in the oblique direction, and the oblique direction may include 135° and 45° directions, corresponding to grad-135 operator and grad-45 operator respectively, forming the convolution module grad-135conv and the convolution module grad-45conv in the figure.

[0123] The preset operator may also include one or more of the following: Laplace operator laplace (corresponding to laplace conv in the figure), operator sobel-x for extracting features from edge information of the image in the horizontal direction (corresponding to sobel-x conv in the figure), and operator sobel-y for extracting features from edge information of the image in the vertical direction (corresponding to sobel-y conv in the figure). Thus, as shown in the figure, the edge information of the image is extracted respectively through 5 branches, which greatly improves the edge information extraction capability of IECB, makes it pay more attention to the texture information in the image, and makes the image reconstruction effect better.

[0124] The parameters of the preset operator may be pre-set. Figure 6 , showing a schematic diagram of a preset operator according to an embodiment of the present application. Figure 6 As shown, the preset operators may include Laplace operator, Sobel-x operator for extracting features from edge information of an image in the horizontal direction, Sobel-y operator for extracting features from edge information of an image in the vertical direction, and grad-135 and grad-45 operators for extracting features from edge information of an image in the oblique direction. The preset operator may be an operator of size 3x3, and its parameters may be preset and remain fixed during the training process.

[0125] After the training of the neural network model is completed, the parameters of the reparameterization module in the neural network model in the reasoning stage can be obtained by fusing the weights of the preset operators in the reparameterization module. The preset operators include operators for extracting features from edge information in the tilt direction of the spliced ​​image. Since the parameters of the reparameterization module after fusion have the corresponding capabilities of the weights of the preset operators before fusion, and the parameter quantity is significantly reduced compared to before fusion, the backbone network based on IECB (i.e., the reparameterization network in the embodiment of the present application) not only has a strong feature extraction capability, but also has a smaller amount of calculation in the reasoning stage, which can improve the image reconstruction speed. Combined with Figure 3 The reparameterized network structure shown in the figure, in the inference stage, the reparameterized network consists of only 10 layers of 3x3 conv. At the input resolution of 540x960, the computation amount only reaches 22.25G floating point operations per second (FLOPS).

[0126] You can return to see Figure 5 , in the inference stage, we can Figure 5 The weights of all branches of the left half of IECB are fused and reparameterized (as shown in the figure Reparameter) as follows Figure 5 A 3x3 conv shown on the right.

[0127] Among them, each branch in the training stage can be fused first to obtain 7 3x3 conv. For example, the first 3x3 conv can be kept unchanged, and the 1x1 conv and 3x3 conv (for the preset operator, it can be the 3x3 convolution module corresponding to the preset operator) in the remaining 6 branches can be fused to obtain 1 3x3 conv for each branch. Then, the 7 3x3 conv are fused to obtain a 3x3 conv in the reasoning stage. The fusion method can be implemented based on the existing technology.

[0128] Based on the above-mentioned model structure and the structure of the IECB in the training phase, the training phase of the neural network model in the embodiment of the present application is introduced below. Figure 7, shows a flow chart of an image super-resolution processing method according to an embodiment of the present application. Figure 7 As shown, the method may also include:

[0129] Step S701, iteratively training the initial neural network model using training data.

[0130] The training data can be obtained by extracting frames from a video clip, and the test data in the training process can also be obtained by extracting frames from the same video clip. The training data can include a current frame training image, a previous frame training image of the current frame training image, and a motion vector from the previous frame training image to the current frame training image, and the test data can include a current frame test image, a previous frame test image of the current frame test image, and a motion vector from the previous frame test image to the current frame test image.

[0131] For example, in a game scene, you can record game clips (e.g., 2 to 5 seconds) of different game scenes at a preset resolution (e.g., 1080P) and determine the motion vectors rendered between each frame. You can extract frames from the recorded game clips and combine the images of the previous and next frames and the corresponding motion vectors into a data pair (x t-1 , x t , v t ) HR , that is, high-resolution labeled data pairs. Based on multiple data pairs, a high-resolution data set can be obtained. t-1 Can represent the previous frame image, x t Can represent x t-1 The next frame image, v t The corresponding motion vector can be represented.

[0132] The high-resolution dataset can be divided into a preset ratio (such as 8:2) according to different game scenes to obtain a training set and a test set.

[0133] The training data can be obtained from the training set. In each iterative training, a preset batch number (e.g., 64) of data pairs can be randomly selected from the training set, a region of interest (ROI) of a preset size (e.g., 256x256) is randomly generated for each data pair, and an image of the ROI region in the data pair is intercepted to obtain the data pair (x t-1 , x t , v t ) ROI .

[0134] We can also perform the same operation on the data pair (x t-1 , x t , v t ) ROIPerform random horizontal flipping with a preset probability (such as 0.5) to expand the training data. t Label data y for super-resolution processing gt , and then for the data pair (x t-1 , x t , v t ) ROI Downsample to a preset resolution (such as 128x128) to obtain the data pair (x t-1 , x t , v t ) LR As input data for training, and label data y gt Together they constitute the training data.

[0135] In the process of iteratively training the initial neural network model using training data, the preset batch number (x t-1 , x t , v t ) LR Input the neural network model to obtain the reconstructed image y after B training stages pred The value of the loss function may be calculated at each iteration. The loss function in the embodiment of the present application may be Charbonnier loss. The calculation method of the loss function value may refer to the following formula (1):

[0136]

[0137] Among them, Loss Charbonnier Can represent the value of the loss function, y pred It can represent the reconstructed image obtained in the training phase, y gt It can represent label data, and ε can represent preset parameters.

[0138] Since ε is introduced in Charbonnier loss 2 , making it differentiable even close to 0, insensitive to outliers, avoiding excessive error amplification, and more suitable for image super-resolution processing tasks.

[0139] An optimizer (such as the Adam optimizer) can be used to calculate gradients, update the parameters of the neural network model through back propagation, and update the learning rate of the optimizer using a learning rate strategy (such as the cosine annealing learning rate decay strategy), where the initial learning rate can be 2e-4 and the minimum learning rate can be 1e-7.

[0140] During the iterative training process, for IECB, the following updates can be made: Figure 5 The parameters of 3x3 conv and 1x1 conv in the default operator and the parameters of the convolution module corresponding to the default operator can be fixed.

[0141] Step S702: every preset number of iterative training times, use the test data to perform performance evaluation on the iteratively trained neural network model.

[0142] The preset number of iterative training times is, for example, 5000 times, and performance evaluation can be performed and model parameters can be saved every preset number of iterative training times.

[0143] The test data can be obtained from the test set, where data pairs (x t-1 , x t , v t ) HR , and select x t As label data y gt . We can use the data pair (x t-1 , x t , v t ) HR Downsampling is performed to obtain a data pair with a resolution 0.5 times the original one (x t-1 , x t , v t ) LR As input to the model in S702, the performance of the neural network model is evaluated.

[0144] In the process of evaluating the performance of the model, the reconstructed image y output by the neural network model during the test can be obtained based on the input data. pred For each data pair corresponding to y pred and label data y gt The peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) values ​​are calculated, and multiple PSNR values ​​and SSIM values ​​are averaged as the performance evaluation result of this model.

[0145] Step S703, taking the iteratively trained neural network model with the best performance evaluation result as the trained neural network model.

[0146] After the training is iterated for a preset number of times (eg, 500,000), the trained neural network model can be determined based on multiple performance evaluation results.

[0147] Among them, the PSNR value can be used as the first priority, and the PSNR values ​​corresponding to the models after multiple iterations are first compared. The neural network model with the largest corresponding PSNR value is used as the trained neural network model. When the PSNR values ​​are the same, the neural network model with the largest corresponding SSIM is used as the trained neural network model. In this way, the trained neural network model can be obtained. As mentioned above, in the inference stage, the weights of the preset operators in the reparameterization module of the trained neural network model can be fused to obtain the parameters of the reparameterization module.

[0148] Return to see Figure 2 ,After obtaining the trained neural network model, the trained neural network model can be ,used for image super-resolution processing in the inference stage.

[0149] The above-mentioned processing of the spliced ​​image by a neural network model to obtain a processed image may include:

[0150] The spliced ​​image is processed using a reparameterized network to obtain a processed image;

[0151] The above-mentioned neural network model is used to upsample the current frame image to obtain an upsampled image, including:

[0152] Performing upsampling processing on the current frame image by using the first upsampling module to obtain an upsampled image;

[0153] Therefore, the neural network model can obtain a reconstructed image based on the processed image and the upsampled image.

[0154] Among them, based on Figure 3 The structure of the reparameterized network shown in FIG. 1 may include:

[0155] The spliced ​​image is processed using a reparameterization module and a first activation module to obtain a first intermediate image.

[0156] The number of channels of the stitched image can be 6. The reparameterization module can be found in Figure 3 IECB in the example can be used to extract features from the edge information of the spliced ​​image, so that it can pay more attention to the texture information in the image during the super-resolution processing. The number of channels of the output image after feature extraction by IECB can be 16. After the first activation module (see Figure 3 After the nonlinear activation processing by Leaky-ReLU1), the number of channels of the output first intermediate image can be 16.

[0157] The first parameterized residual module may be used to perform feature extraction on the first intermediate image to obtain a second intermediate image.

[0158] The first parameterized residual module can be found in Figure 3 In IECBResBlock1, the number of channels of the second intermediate image can be 16.

[0159] The second parameterized residual module may be used to perform feature extraction on the second intermediate image to obtain a third intermediate image.

[0160] The second parameterized residual module can be found in Figure 3 In IECBResBlock2, the number of channels of the third intermediate image can be 16.

[0161] The third parameterized residual module may be used to perform feature extraction on the third intermediate image to obtain a fourth intermediate image.

[0162] Among them, the third parameterized residual module can be found in Figure 3 In IECBResBlock3, the number of channels of the fourth intermediate image can be 16.

[0163] The fourth parameterized residual module is used to perform feature extraction on the second intermediate image and the fourth intermediate image to obtain a fifth intermediate image.

[0164] Among them, the fourth parameterized residual module can be found in Figure 3 In IECBResBlock4, the number of channels of the fifth intermediate image can be 16.

[0165] The first intermediate image and the fifth intermediate image may be processed using a convolution module, a second activation module, and a second upsampling module to obtain processed images.

[0166] Among them, the convolution module can be found in Figure 3 In the 3x3 conv, the first intermediate image and the fifth intermediate image are convolved by the convolution module, and the number of channels can be converted from 16 to 12. The output is passed through the second activation module (see Figure 3 After nonlinear activation processing (Leaky-ReLU2 in ), it can be input into the second upsampling module (see Figure 3 The image is upsampled by pixel-shuffle (in ), and a processed image with 3 channels is obtained.

[0167] The reconstructed image can be obtained based on the sum of the processed image and the upsampled image through a neural network model.

[0168] In order to further improve the clarity of the reconstructed image, in the inference stage, the embodiment of the present application adds a sharpening module to the neural network model. The method may also include:

[0169] Sharpening is performed based on the reconstructed image to obtain a sharpened image.

[0170] Among them, the robust contrast adaptive sharpening (RCAS) technology can be used to sharpen the reconstructed image. Among them, the process based on RCAS sharpening in the embodiment of the present application can be implemented on the GPU in the form of convolution and tensor logic operations, and the speed of sharpening based on RCAS can be improved by GPU parallel computing. The clarity of the sharpened image is greater than that of the reconstructed image.

[0171] Figure 8 FIG. 2 shows an overall flow chart of image super-resolution processing according to an embodiment of the present application. Figure 8 As shown, the image super-resolution processing method of the embodiment of the present application can be divided into an upsampling part (as shown in the figure) and a sharpening part (sharpen) in the inference stage. As an example, in the upsampling part, the input of the neural network model includes the current frame image x with a resolution of 540p t , the previous frame image x of the current frame image with a resolution of 540p t-1 , and the rendered x t-1 to x t The motion vector v t . Can be based on x t-1 Using v t Perform motion compensation (Motion Compensation), such as optical flow transformation (Wrap), concatenate the obtained image with the current frame image, and input the concatenated image into the reparameterized network (IECBNet) to obtain a processed image with a resolution of 1080p. t The first upsampling module is used to perform bilinear interpolation upsampling (Upsample(bilinear)) to obtain the upsampled image, and the upsampled image and the processed image are added to obtain the reconstructed image y with a resolution of 1080p pred In order to further improve the clarity of the reconstructed image, in the inference stage, the RCAS technology (see RCASNet in the figure) can be used in the sharpening part to reconstruct the image y pred Perform sharpening processing to obtain a sharpened image y with a resolution of 1080p pred 'The image obtained by processing the final neural network model.

[0172] Table 1 is a performance diagram of a neural network model (which may be referred to as GSRNet) according to an embodiment of the present application.

[0173] Table 1

[0174] Model Input Params GFLOPS PSNR SSIM Latency(FP32) Latency(FP16) GSRNet 540P 19K 23.3135 34.4306 0.9444 4.4640ms 3.2765ms

[0175] As can be seen from Table 1, the parameters of the GSRNet model of this application are only 19K. When the resolution of the input image is only 540P, the computational complexity of the model is 23.3145 GFLOPS, and the number of parameters and computational complexity have reached a very light level. At the same time, the performance of the model has also reached a relatively high level, with PSNR reaching 34.4306 and SSIM reaching 0.9444 on the test set.

[0176] Regarding the inference speed of the neural network model of this application, we performed 1,000 inferences on the corresponding devices and calculated the average inference time. Among them, the average inference time (Latency) of GSRNet is 4.4640ms under Float32 precision, and the FPS reaches 224 frames; the average inference time of GSRNet under Float16 precision is 3.2765, and the FPS reaches 305 frames. These are only the inference speeds of the original model. If the GPU is used to accelerate the inference framework, the inference time will be lower. This application has been tested. The average inference time of the model on the GPU under Float16 precision is 2.9243ms, and the FPS reaches 342 frames. The GPU computing memory occupancy is only 619MB. It can be seen that the load of model calculation is negligible, which can greatly improve the frame rate of GPU rendering.

[0177] Fig. 9 FIG. 2 shows a structural diagram of an image super-resolution processing device according to an embodiment of the present application. Fig. 9 As shown, the device comprises:

[0178] A first determining module 901 is used to determine an image to be processed;

[0179] A second determination module 902 is used to process the image to be processed through a neural network model to obtain a reconstructed image;

[0180] The processing includes extracting features from edge information of the image to be processed through a reparameterization module of a neural network model, the edge information includes edge information of the image to be processed in an inclined direction, and the resolution of the reconstructed image is greater than that of the image to be processed.

[0181] In a possible implementation, the parameters of the re-parameterization module are obtained by fusing the weights of preset operators, where the preset operators include operators for extracting features of edge information of the image to be processed in an oblique direction.

[0182] In a possible implementation, the image to be processed includes a stitched image, and the first determining module 901 is configured to:

[0183] A spliced ​​image is determined based on a current frame image, a frame image before the current frame image, and a motion vector from the previous frame image to the current frame image.

[0184] In a possible implementation, determining a spliced ​​image based on a current frame image, a frame image before the current frame image, and a motion vector from the previous frame image to the current frame image includes:

[0185] Perform motion compensation on the previous frame image using the motion vector to determine the compensated image;

[0186] The compensated image and the current frame image are spliced ​​to determine a spliced ​​image.

[0187] In a possible implementation, the device further includes:

[0188] The sharpening module is used to perform sharpening based on the reconstructed image to obtain a sharpened image, and the clarity of the sharpened image is greater than that of the reconstructed image.

[0189] In a possible implementation, the image to be processed includes a current frame image, and the second determining module 902 is configured to:

[0190] The spliced ​​image is processed through a neural network model to obtain a processed image;

[0191] Through the neural network model, the current frame image is upsampled to obtain an upsampled image;

[0192] Through the neural network model, a reconstructed image is obtained based on the processed image and the upsampled image.

[0193] In a possible implementation, the neural network model includes a reparameterized network and a first upsampling module. The spliced ​​image is processed by the neural network model to obtain a processed image, including:

[0194] The spliced ​​image is processed using a reparameterized network to obtain a processed image;

[0195] Through the neural network model, the current frame image is upsampled to obtain the upsampled image, including:

[0196] The first up-sampling module is used to perform up-sampling processing on the current frame image to obtain an up-sampled image.

[0197] In a possible implementation, the reparameterized network includes a reparameterized module, a first reparameterized residual module, a second reparameterized residual module, a third reparameterized residual module, a fourth reparameterized residual module, a first activation module, a second activation module, a convolution module, and a second upsampling module. The first reparameterized residual module, the second reparameterized residual module, the third reparameterized residual module, and the fourth reparameterized residual module are in a cascade relationship. The spliced ​​image is processed using the reparameterized network to obtain a processed image, including:

[0198] Processing the spliced ​​image using the reparameterization module and the first activation module to obtain a first intermediate image;

[0199] Using a first parameterized residual module to extract features from the first intermediate image, to obtain a second intermediate image;

[0200] Using a second parameterized residual module to extract features from the second intermediate image, to obtain a third intermediate image;

[0201] extracting features from the third intermediate image using a third parameterized residual module to obtain a fourth intermediate image;

[0202] Using a fourth parameterized residual module to extract features from the second intermediate image and the fourth intermediate image, to obtain a fifth intermediate image;

[0203] The first intermediate image and the fifth intermediate image are processed by using a convolution module, a second activation module, and a second upsampling module to obtain processed images.

[0204] In a possible implementation, the first parameterized residual module, the second parameterized residual module, the third parameterized residual module, and the fourth parameterized residual module are residual structures, respectively consisting of a re-parameterized module and an activation module.

[0205] In a possible implementation, the weight of the preset operator is obtained in the training phase of the neural network model, the neural network model is a trained neural network model, and the device further includes:

[0206] An iterative training module, used to iteratively train the initial neural network model using training data;

[0207] A performance evaluation module is used to evaluate the performance of the neural network model after iterative training using test data every preset number of iterative training times;

[0208] The third determination module is used to use the iteratively trained neural network model with the best performance evaluation result as the trained neural network model.

[0209] In a possible implementation, the preset operator also includes one or more of the following: a Laplace operator, an operator for extracting features of edge information of the image to be processed in the horizontal direction, and an operator for extracting features of edge information of the image to be processed in the vertical direction.

[0210] According to an embodiment of the present application, the image to be processed is processed by a neural network model to obtain a reconstructed image, so that the resolution of the reconstructed image is greater than that of the image to be processed, thereby realizing super-resolution processing of the image, wherein the edge information of the image to be processed in the inclined direction is extracted through the re-parameterization module of the neural network model, thereby improving the reconstruction quality of the image.

[0211] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0212] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0213] The disclosed embodiment also proposes an image super-resolution processing device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0214] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0215] Fig.10 1 is a block diagram of a device 1900 for image super-resolution processing according to an exemplary embodiment. For example, the device 1900 may be provided as a server or a terminal device. Fig.10 , the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0216] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2000. TM , MacOS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0217] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the device 1900 to perform the above method.

[0218] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0219] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0220] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0221] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0222] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0223] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0224] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0225] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0226] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for image super-resolution processing, characterized in that: The method comprises: Determining an image to be processed, wherein the image to be processed includes a stitched image, and determining the image to be processed includes: Determine the spliced ​​image based on a current frame image, a frame image before the current frame image, and a motion vector from the previous frame image to the current frame image; The image to be processed is processed through a neural network model to obtain a reconstructed image; Wherein, the neural network model includes a reparameterized network, the reparameterized network includes a reparameterized module and a plurality of reparameterized residual modules, the plurality of reparameterized residual modules are in a cascade relationship, the plurality of reparameterized residual modules are respectively residual structures, each reparameterized residual module is further nested to include an activation module and two reparameterized modules, the input of the first reparameterized module in the reparameterized residual module is added to the output of the second reparameterized module in the subsequent order as the output of the reparameterized residual module, the parameters of the reparameterized module are obtained by fusing the weights of preset operators, the preset operators include operators for extracting features of edge information of the spliced ​​image in the oblique direction, and each operator corresponds to a different oblique direction; The processing includes: using the reparameterization network to process the stitched image to obtain a processed image, and determining a reconstructed image based on the processed image; wherein the reparameterization module is used to extract features from edge information of the stitched image, the edge information includes edge information of the stitched image in an inclined direction, and the resolution of the reconstructed image is greater than that of the image to be processed.

2. The method according to claim 1, characterized in that: The step of determining the spliced ​​image based on the current frame image, a frame image before the current frame image, and a motion vector from the previous frame image to the current frame image includes: Performing motion compensation on the previous frame image using the motion vector to determine a compensated image; The compensated image and the current frame image are spliced ​​to determine the spliced ​​image.

3. The method according to claim 1, characterized in that: The method further comprises: Sharpening is performed based on the reconstructed image to obtain a sharpened image, wherein the sharpened image has a clarity greater than that of the reconstructed image.

4. The method according to claim 1, characterized in that: The image to be processed includes a current frame image, and the image to be processed is processed by a neural network model to obtain a reconstructed image, including: Processing the spliced ​​image through a neural network model to obtain a processed image; Performing upsampling processing on the current frame image through a neural network model to obtain an upsampled image; The reconstructed image is obtained based on the processed image and the upsampled image through a neural network model.

5. The method according to claim 4, wherein the neural network model comprises a first upsampling module, and the upsampling process is performed on the current frame image through the neural network model to obtain an upsampled image, comprising: The first up-sampling module is used to up-sample the current frame image to obtain an up-sampled image.

6. The method according to claim 5, characterized in that The reparameterized network includes a reparameterized module, a first reparameterized residual module, a second reparameterized residual module, a third reparameterized residual module, a fourth reparameterized residual module, a first activation module, a second activation module, a convolution module, and a second upsampling module. The first reparameterized residual module, the second reparameterized residual module, the third reparameterized residual module, and the fourth reparameterized residual module are in a cascade relationship. The reparameterized network is used to process the spliced ​​image to obtain a processed image, including: Processing the spliced ​​image using the reparameterization module and the first activation module to obtain a first intermediate image; Using the first parameterized residual module to perform feature extraction on the first intermediate image to obtain a second intermediate image; performing feature extraction on the second intermediate image using the second parameterized residual module to obtain a third intermediate image; Using the third parameterized residual module to perform feature extraction on the third intermediate image to obtain a fourth intermediate image; Using the fourth parameterized residual module to perform feature extraction on the second intermediate image and the fourth intermediate image to obtain a fifth intermediate image; The first intermediate image and the fifth intermediate image are processed by using the convolution module, the second activation module and the second upsampling module to obtain the processed image.

7. The method according to claim 1, wherein the weight of the preset operator is obtained in a training phase of a neural network model, and the neural network model is a trained neural network model, and the method further comprises: Iteratively train the initial neural network model using training data; After every preset number of iterations, the performance of the neural network model after iterative training is evaluated using test data; The iteratively trained neural network model with the best performance evaluation result is used as the trained neural network model.

8. The method according to claim 1, characterized in that The preset operator also includes one or more of the following: a Laplace operator, an operator for extracting features from edge information of the image to be processed in the horizontal direction, and an operator for extracting features from edge information of the image to be processed in the vertical direction.

9. An image super-resolution processing device, characterized in that: The device comprises: A first determining module is used to determine an image to be processed, wherein the image to be processed includes a stitched image, and the first determining module is used to: Determine the spliced ​​image based on a current frame image, a frame image before the current frame image, and a motion vector from the previous frame image to the current frame image; The second determination module is used to process the image to be processed through a neural network model to obtain a reconstructed image; The neural network model includes a reparameterized network, the reparameterized network includes a reparameterized module and a plurality of reparameterized residual modules, the plurality of reparameterized residual modules are in a cascade relationship, the plurality of reparameterized residual modules are respectively residual structures, each reparameterized residual module is further nested to include an activation module and two reparameterized modules, the input of the first reparameterized module in the reparameterized residual module is added to the output of the second reparameterized module in the subsequent order as the output of the reparameterized residual module, the parameters of the reparameterized module are obtained by fusing the weights of preset operators, the preset operators include operators for extracting features of edge information of the spliced ​​image in the oblique direction, and each operator corresponds to a different oblique direction; The processing includes: using the reparameterization network to process the stitched image to obtain a processed image, and determining a reconstructed image based on the processed image; wherein the reparameterization module is used to extract features from edge information of the stitched image, the edge information includes edge information of the stitched image in an inclined direction, and the resolution of the reconstructed image is greater than that of the image to be processed.

10. An image super-resolution processing device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 8 when executing the instructions stored in the memory.

11. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images