Image processing, model training method and device, equipment, storage medium

Abstract

The embodiments of the present application disclose image processing and model training methods and devices, equipment, and storage media, wherein the method comprises: obtaining an image to be processed; calling at least one trained image reconstruction model; wherein the image reconstruction model is obtained after training based on a training sample set including multiple frames of sample images and a true value image corresponding to each frame, wherein the true value image is obtained by fusion of multiple frames of reference images, and the resolution of the true value image is greater than the resolution of the corresponding sample image; and super-resolution processing is performed on the image to be processed through the at least one image reconstruction model to obtain a target image.

Description

Technical Field

[0001] The embodiments of the present application relate to image processing technology, including but not limited to image processing, model training methods and devices, equipment, and storage media. Background Art

[0002] As an important form of information for perceiving the world, the richness and details of images directly determine the level of detail of the perceived content. When the pixel density on the unit scale of an image is higher, the image is clearer, the ability to express details is stronger, and the perceived information is richer, which is a high-resolution image. Image super-resolution reconstruction has been studied in many areas, such as remote sensing image field, satellite imaging field, medical image field, and some high-definition display fields.

[0003] However, in the related art, due to the defects of the original image, the clarity of the high-resolution image obtained by the super-resolution method is low. Summary of the invention

[0004] The image processing, model training method and apparatus, device, and storage medium provided in the embodiments of the present application can obtain a high-resolution image with high definition; wherein the image processing, model training method and apparatus, device, and storage medium are implemented as follows:

[0005] The image processing method provided by the embodiment of the present application includes: obtaining an image to be processed; calling at least one trained image reconstruction model; wherein the image reconstruction model is obtained after training based on a training sample set including multiple frames of sample images and a true value image corresponding to each frame, the true value image is obtained by fusing multiple frames of reference images, and the resolution of the true value image is greater than the resolution of the corresponding sample image; through the at least one image reconstruction model, super-resolution processing is performed on the image to be processed to obtain a target image.

[0006] The model training method provided in the embodiment of the present application includes: downsampling the original image set according to each preset downsampling parameter value to obtain a downsampled image set under the corresponding downsampling parameter value; generating a training sample set according to the downsampled image set and the true value image corresponding to the downsampled image; wherein the true value image is obtained by fusion of multiple frames of reference images, and the resolution of the true value image is greater than the resolution of the corresponding sample image; using the training sample set under each of the downsampling parameter values, the original deep learning model is trained to obtain an image reconstruction model under the corresponding downsampling parameter value.

[0007] The image processing device provided by the embodiment of the present application includes: an acquisition module, used to acquire an image to be processed; a calling module, used to call at least one trained image reconstruction model; wherein the image reconstruction model is obtained after training based on a training sample set including multiple frames of sample images and a true value image corresponding to each frame, the true value image is obtained by fusing multiple frames of reference images, and the resolution of the true value image is greater than the resolution of the corresponding sample image; a super-resolution processing module, used to perform super-resolution processing on the image to be processed through the at least one image reconstruction model to obtain a target image.

[0008] The model training device provided in the embodiment of the present application includes: a downsampling module, which is used to downsample the original image set according to each preset downsampling parameter value to obtain a downsampled image set under the corresponding downsampling parameter value; a sample generation module, which is used to generate a training sample set based on the downsampled image set and the true value image corresponding to the downsampled image; wherein the true value image is obtained by fusion of multiple frames of reference images, and the resolution of the true value image is greater than the resolution of the corresponding sample image; a model training module, which is used to train the original deep learning model using the training sample set under each of the downsampling parameter values ​​to obtain an image reconstruction model under the corresponding downsampling parameter value.

[0009] The electronic device provided in an embodiment of the present application includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps of the image processing method described in any of the embodiments of the present application are implemented, or when the processor executes the program, the steps of the model training method described in any of the embodiments of the present application are implemented.

[0010] The computer-readable storage medium provided in the embodiments of the present application has a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the image processing method described in any of the embodiments of the present application are implemented, or when the computer program is executed by a processor, the steps of the model training method described in any of the embodiments of the present application are implemented.

[0011] In the embodiment provided in the present application, the image reconstruction model for super-resolution processing of the image to be processed is obtained by training a training sample set including multiple frames of sample images and a true value image corresponding to each frame, and the true value image is formed by fusion of multiple frames of reference images, rather than directly taking a single frame of reference image as the true value image; in this way, the signal-to-noise ratio of the fused true value image is higher and the details are clearer, thereby improving the performance of the trained model, and then when the model is used, the image to be processed can be reconstructed into a target image with higher clarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A schematic diagram of an application scenario of the image reconstruction model;

[0013] Figure 2 A schematic diagram of another application scenario of the image reconstruction model;

[0014] Figure 3 A simplified schematic diagram of a video and audio playback interface;

[0015] Figure 4A This is a schematic diagram of the implementation flow of the model training method of the embodiment of the present application;

[0016] Figure 4B A schematic diagram of a method for generating a training sample set according to an embodiment of the present application;

[0017] Figure 5 A schematic diagram of the process of training multiple initial models (e.g., original deep learning models) in an embodiment of the present application;

[0018] Figure 6 This is another implementation flow diagram of the model training method according to an embodiment of the present application;

[0019] Figure 7 This is a schematic diagram of the implementation process of the image processing method of the embodiment of the present application;

[0020] Figure 8 This is another schematic diagram of the implementation process of the image processing method according to the embodiment of the present application;

[0021] Fig. 9 Schematic diagram of comparison between high-resolution reconstructed image 902 and original image 901;

[0022] Fig.10 This is a schematic diagram of the main process of generating real training data for a super-resolution network based on multi-frame fusion in an embodiment of the present application;

[0023] Fig.11 This is a schematic diagram of the implementation process of multi-frame fusion in the embodiment of the present application;

[0024] Fig.12 A schematic diagram of the implementation process of calculating the homography matrix in an embodiment of the present application;

[0025] Fig.13A This is a schematic diagram of the structure of an image processing device according to an embodiment of the present application;

[0026] Fig. 13B Another structural schematic diagram of the image processing device according to an embodiment of the present application;

[0027] Fig.14A This is a schematic diagram of the structure of the model training device of the embodiment of the present application;

[0028] Fig. 14B Another structural schematic diagram of the model training device according to an embodiment of the present application;

[0029] Fig.15 A hardware entity schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0032] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0033] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are merely used to distinguish similar or different objects, and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0034] Super-resolution processing is to process low-resolution images to obtain corresponding high-resolution images. For example, for a 540P resolution image, after super-resolution processing, a 1080P resolution image can be obtained. In the super-resolution processing process, first, the representation value of each pixel position in the low-resolution image is obtained; then, based on the representation value of each pixel position in the low-resolution image, each representation value is calculated based on the trained image reconstruction model, and multi-channel data is output to obtain multiple values ​​related to each representation value. These values ​​will be used as the representation values ​​of the new pixel positions on the super-resolution image; finally, based on the representation values ​​of these new pixel positions, a high-resolution image can be generated.

[0035] The application scenarios of the image reconstruction model described below in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. A person skilled in the art will appreciate that, with the emergence of new business scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0036] The trained image reconstruction model is configured in an image playback application, which is installed on a terminal with image display and / or video playback functions such as a mobile phone, tablet computer, personal computer (PC) or wearable device. The image playback application can be, for example, various applications that can play videos and / or various applications that can display images (such as camera applications). When a user opens the image playback application on the terminal, he can request to play a video or display an image.

[0037] Figure 1 A schematic diagram of an application scenario of the image reconstruction model is shown in FIG. Figure 1 As shown, when user 101 requests to play a video through an image playback application, terminal 102 will request the video from server 103 providing video playback service; after receiving the video returned by server 103, terminal 102 extracts the low-resolution video, determines each video frame 104 in the low-resolution video as a low-resolution image, and inputs the low-resolution image as an image to be processed into at least one image reconstruction model, and the at least one image reconstruction model performs super-resolution processing on the image to be processed, and obtains at least one high-resolution image frame. When there is one high-resolution image, the target image is obtained, and when there are multiple high-resolution images, these images are fused to obtain the target image. Each target image corresponds to a video frame. After obtaining the target image, the image playback application plays the target images in sequence according to factors such as the playback time sequence to achieve the playback of high-resolution video.

[0038] Similarly, when the terminal is targeting a single image, it can directly use the single image as the image to be processed and perform super-resolution processing before displaying the obtained target image to the user. In this way, on the one hand, the terminal only needs to download low-resolution video data, saving bandwidth occupation and terminal traffic; on the other hand, it can also save server storage space, and it can also facilitate users to watch higher-definition videos.

[0039] The user can request to switch between the low-resolution image and the high-resolution image by clicking a switch button. Of course, the terminal can also receive user operations through an application interface. Only when the user selects the high-resolution play button on the application interface, the terminal will perform super-resolution processing on the low-resolution image, obtain the high-resolution target image and play it to the user.

[0040] Figure 2 Schematic diagram of another application scenario of the image reconstruction model, the image reconstruction model is configured in the application server 201 that provides video playback service or image display service, and the terminal 202 is installed with the corresponding audio and video playback application. The application server 201 may store low-resolution videos or images. When the user 203 requests to watch these low-resolution videos or images through the video playback application installed on the terminal 202, the application server 201 can send the data of the video or image after super-resolution processing to the terminal 202, and display it to the user through the application installed on the terminal 202.

[0041] When watching a low-resolution video, if the user wants to watch a high-resolution video, the user can click on the high-resolution play button set on the application interface. At this time, the terminal 202 sends a K-fold high-resolution play request to the application server 201. The application server 201 responds to the high-resolution play request, determines the low-resolution video data, obtains the image to be processed (such as a low-resolution single picture or a video frame in a low-resolution video), performs super-resolution processing through at least one image reconstruction model, and finally outputs the target image 204. The target image is transmitted to the terminal 202, and the terminal 202 then displays the high-resolution target image through the audio and video playback application or plays the high-resolution target image in sequence according to the playback time sequence, so as to achieve the playback of the super-resolution video.

[0042] Figure 3 FIG. 3 is a schematic diagram of a video playback interface. After the user clicks the “super resolution” playback button 301 in the lower right corner, the video player performs the above Figure 1 The corresponding method or the relevant application server follows the above Figure 2 In a corresponding manner, the video or image is super-resolution processed to provide the user with a super-resolution video or image. The "super-resolution" button 301 in the lower right corner can be a plurality of buttons of other types, such as an icon button 302 of "2x super-resolution" and an icon button 303 of "4x super-resolution".

[0043] In some embodiments, the server may also switch between low-resolution images and high-resolution images based on the network resource information between the server and the terminal. When the network resources meet the conditions (for example, when the bandwidth is sufficient), the server automatically converts the low-resolution image into a high-resolution target image and sends it to the user; when the network resources do not meet the conditions (for example, when the bandwidth is small), the server sends the low-resolution image.

[0044] The image processing method of the embodiment of the present application is described in detail from two aspects. The two aspects include, on the one hand, model training, and on the other hand, a process of super-resolution processing of the image to be processed based on at least one trained image reconstruction model.

[0045] The model training method provided in the embodiment of the present application can be applied to an electronic device, which can be either a terminal or a server; similarly, the image processing method provided in the embodiment of the present application can be applied to the electronic device.

[0046] Figure 4A Schematic diagram of the implementation flow of the model training method of the embodiment of the present application, such as Figure 4A As shown, the method may at least include the following steps 401 to 403:

[0047] Step 401 : downsampling the original image set according to each preset downsampling parameter value to obtain a downsampled image set under the corresponding downsampling parameter value.

[0048] Each preset downsampling parameter value may include one or more different downsampling parameter values. When one is included, the corresponding image reconstruction model obtained by training is one; when multiple are included, the corresponding image reconstruction models obtained by training are multiple. For multiple, for example, Figure 5 As shown, each preset downsampling parameter value is: 2×2 binning, 3×3 binning and 4×4 binning; according to 2×2 binning, each image 501 in the original image set is downsampled, and a downsampled image set 502 is obtained; according to 3×3 binning, each image 501 in the original image set is downsampled, and a downsampled image set 503 is obtained; according to 4×4 binning, each image 501 in the original image set is downsampled, and a downsampled image set 504 is obtained. It should be noted that binning means merging, and 2×2 binning means merging 2×2 pixels into 1 pixel, for example, taking the pixel mean of these 4 pixels as the pixel value of the pixel, so binning can also be understood as downsampling.

[0049] In some embodiments, for downsampling the original image in the original image set according to the downsampling parameter value to obtain the downsampled image, the electronic device can implement it as follows: using the downsampling parameter value as the size of the Gaussian kernel, performing Gaussian blur processing on the original image to obtain a Gaussian blurred image; and performing bicubic interpolation on the Gaussian blurred image according to the downsampling parameter value to obtain the downsampled image.

[0050] Step 402, generating a training sample set corresponding to the downsampled image set according to each of the downsampled image sets and a ground truth image (GT) corresponding to the downsampled image; wherein the ground truth image is obtained by fusing multiple frames of reference images, and the resolution of the ground truth image is greater than the resolution of the corresponding sample image.

[0051] It can be understood that in supervised learning, each downsampled image corresponds to a frame of true value image; in semi-supervised learning, each image of some downsampled images in the set corresponds to a frame of true value image.

[0052] When implementing step 402, the device may first determine the second displacement of each downsampled image relative to the corresponding true value image, and then shift each downsampled image pixel by pixel according to the corresponding second displacement to align with the corresponding true value image, thereby obtaining a corresponding sample image; each sample image and the corresponding true value image are used as a data pair to generate the training sample set. In this way, through the alignment method, the trained image reconstruction model can obtain a target image that is not locally blurred or deformed when performing super-resolution processing on a low-resolution image.

[0053] It should be noted that the method for determining the second displacement is the same as the method for determining the first displacement. For example, the electronic device may first detect feature points of two frames of images and extract feature description information of each feature point; then, based on the feature description information of each feature point of the two frames of images, the two frames of images are feature-matched, thereby finding multiple feature point matching pairs between the two frames of images; based on the pixel coordinates of the feature points of these feature point matching pairs, the homography matrix between the two frames of images is determined; finally, based on the homography matrix, the displacement of one frame of image relative to another frame of image is determined, that is, the displacement of each pixel point in one frame of image relative to the corresponding pixel point in another frame of image.

[0054] In some embodiments, before aligning the downsampled image with the corresponding true value image, the device may first post-process the downsampled image, and then align the post-processed downsampled image with the corresponding true value image. The post-processing will not be elaborated in detail here, and the specific implementation method of the post-processing can be referred to as follows.

[0055] Step 403: Use the training sample set under each of the downsampling parameter values ​​to train the original deep learning model to obtain an image reconstruction model under the corresponding downsampling parameter value.

[0056] The structure of the original deep learning model can be varied, for example, the model can be a recurrent neural network (RNN), a recursive neural network (RNN) or a convolutional neural network (CNN).

[0057] It can be understood that the super-resolution conversion ratio of the image reconstruction model obtained by training the training sample set under different downsampling parameter values ​​is different. Figure 5 For example, a training sample set 512 obtained under a downsampling parameter value of 2×2binning includes a true image and a sample image aligned therewith, and an image reconstruction model 522 obtained by training has a 2-fold super-resolution conversion ratio, that is, a 2-fold magnification capability. When used, the model 522 can transform each pixel position in a low-resolution image to be processed into 2×2 pixel positions; similarly, for a training sample set 513 obtained under a downsampling parameter value of 3×3binning, an image reconstruction model 523 obtained by training has a 3-fold magnification capability; for a training sample set 514 obtained under a downsampling parameter value of 4×4binning, an image reconstruction model 524 obtained by training has a 4-fold magnification capability.

[0058] In an embodiment of the present application, a model training method is provided, in which, in a training sample set used to train an original deep learning model, the true value image corresponding to the sample image is obtained by fusion of multiple reference frames, rather than using only a single reference frame as the true value image; in this way, the obtained true value image has a higher signal-to-noise ratio and clearer details, so that the image reconstruction model obtained by training has better performance, and then when the model is used, it can reconstruct the image to be processed into a higher quality target image. That is, the output target image has a higher signal-to-noise ratio and clearer details.

[0059] It is found in the research that the true value image obtained by fusing multiple reference images may be slightly offset from the corresponding downsampled image, and this offset may cause the output target image of the trained image reconstruction model to be locally blurred or deformed when processing the processed image. In order to solve the above problem, in some embodiments, for the above step 402, a training sample set is generated according to the downsampled image set and the true value image corresponding to the downsampled image, such as Figure 4B As shown, a terminal or server or other device can implement the following steps 4021 to 4023:

[0060] Step 4021: Determine a second displacement of each of the downsampled images relative to the corresponding true value image.

[0061] During implementation, the device may first find multiple feature point matching pairs between the downsampled image and the corresponding true value image; then, based on these feature point matching pairs, solve the homography matrix between the two through direct linear transformation (DLT); finally, determine the displacement of the downsampled image relative to the corresponding true value image according to the homography matrix.

[0062] Step 4022, shifting each of the downsampled images pixel by pixel according to the corresponding second shift to align with the corresponding true value image, thereby obtaining a corresponding sample image;

[0063] Step 4023: Use each of the sample images and the corresponding true value image as a data pair to generate the training sample set.

[0064] This application embodiment further provides a model training method, Figure 6 This is another implementation flow diagram of the model training method of the present application embodiment, such as Figure 6 As shown, the method may at least include the following steps 601 to 610:

[0065] Step 601, obtaining N frames of candidate images, where N is an integer greater than 1.

[0066] Generally speaking, the candidate images are high-resolution images. The N frames of candidate images can be multiple frames of images captured by a camera (such as a SLR camera or a mobile phone with a high-definition shooting function, etc.) of an actual scene. Of course, the N frames of candidate images can also be simulated high-resolution images. Compared with the latter, the image reconstruction model obtained according to the former has better universality. For the real captured image to be processed, the high-resolution image (i.e., the target image) output after reconstruction has better image quality.

[0067] Step 602: Determine the definition of each candidate image.

[0068] Clarity can be characterized by various image parameters. In some embodiments, the clarity is characterized by sharpness, and sharpness estimation is performed on each candidate image to obtain the sharpness of the corresponding image. For example, the sharpness of the image can be determined by the sharpness estimation model shown in the following formula (1):

[0069]

[0070] In formula (1), the entire image is divided into k1*k2 windows, where I max，k，l and Imin，k，l They represent the maximum brightness value and the minimum brightness value in the k*lth window respectively.

[0071] Step 603: determine the images whose clarity meets a specific condition among the N frames of candidate images as the reference images, thereby obtaining multiple frames of reference images.

[0072] In some embodiments, an image whose sharpness is greater than a specific threshold among the N frames of candidate images is determined as the reference image.

[0073] For example, the camera continuously collects 30 frames of candidate images, selects 8 frames with relatively high sharpness as reference images, and fuses them to obtain the true image. It can be understood that the purpose of multi-frame fusion is to obtain a true image with a higher signal-to-noise ratio and clearer details. Therefore, the higher the clarity of the image used for fusion, the clearer the fused true image will be, and the performance of the trained model will be better. That is to say, when the model is used online, it can reconstruct a high-resolution target image with clearer details and a higher signal-to-noise ratio.

[0074] Step 604: Select one frame of the multiple reference images as a first reference image.

[0075] During implementation, any frame of the multiple frames of images may be selected as the first reference image, that is, the image to be referenced for alignment.

[0076] Step 605 : determining a first displacement of each other reference image in the multiple reference image frames except the first reference image relative to the first reference image.

[0077] The method for determining the first displacement here is the same as the method for determining the second displacement described above, and therefore will not be described in detail here.

[0078] Step 606, shifting each of the other reference images pixel by pixel according to the corresponding first shift to align with the first reference image, thereby obtaining a corresponding second reference image;

[0079] Step 607: Fuse the first reference image with each of the second reference images to obtain a true value image.

[0080] It should be noted that the image fusion method, that is, the method for implementing step 607, can be various. The device can implement image fusion in the spatial domain or in the frequency domain.

[0081] When implemented in the spatial domain, for example, the sharpness of each area of ​​the first reference image and each of the second reference images can be determined first; based on the sharpness of each area, the weight of each pixel position in the corresponding area can be determined; the first reference image and each of the second reference images can be weightedly averaged according to the pixel positions and the weights of the pixel positions to obtain the true value image.

[0082] The numerical value can be various. For example, the numerical value can be: the brightness value V of the pixel position, or the brightness value L of the pixel position, or the brightness channel data of the pixel position (i.e., the Y channel value), or any one or more of the red (Red, R), green (Green, G), and blue (Blue, B) values ​​of the pixel position, or the grayscale value of each channel of a multi-spectral camera, or the grayscale value of each channel of a special camera (infrared camera, ultraviolet camera, depth camera), etc.

[0083] It can be understood that sharpness and weight have a certain mapping relationship. The greater the sharpness of the image area, the greater the weight of the pixel position in the area. In this way, when performing image fusion, the influence of the blurred area in the reference image on the clarity of the true value image can be reduced.

[0084] It is impossible for every area of ​​an image to have the same clarity. The focused area is clear and the out-of-focus area is blurry. If the weight is not used and the numerical average is directly performed, the image information of the out-of-focus area will affect the clarity of the corresponding area of ​​the true value image more. Therefore, the weight of the blurred area is set lower, and the weight of the high-definition area is set higher, so that each area of ​​the fused true value image can be as clear as possible.

[0085] When image fusion is realized in the frequency domain, in some embodiments, the device can use wavelet transform to perform multi-layer filtering on the first reference image and the second reference image to be fused, decompose to form high-frequency sub-images and low-frequency sub-images corresponding to each image; perform high-frequency component fusion according to the high-frequency sub-images corresponding to each image to form high-frequency component fusion coefficients; perform low-frequency component fusion according to the low-frequency sub-images corresponding to each image to form low-frequency component fusion coefficients; perform inverse wavelet transform according to the high-frequency components corresponding to the high-frequency component fusion coefficients and the low-frequency components corresponding to the low-frequency component fusion coefficients to generate a true value image. In other embodiments, the device can also perform discrete Fourier transform (DFT) or discrete cosine transform (DCT) on each reference image to convert the image signal to the frequency domain, average the image in the frequency domain, and then inverse transform it to the spatial domain to obtain a true value image.

[0086] Step 608: Downsampling the original image set according to each preset downsampling parameter value to obtain a downsampled image set under the corresponding downsampling parameter value.

[0087] It should be noted that the execution order of the step of obtaining the true value image (corresponding to step 601 to step 607) and the step of obtaining the downsampled image set (i.e., step 608) is not limited. Step 608 may be executed first to obtain the downsampled image set under different downsampling parameter values, and then the true value image may be determined through steps 601 to step 607. Step 608 may also be executed first, and then steps 601 to step 607 may be executed. The step of determining the true value image and the step of determining the sampled image set may also be executed in parallel.

[0088] Step 609, generating a training sample set according to the downsampled image set and the true value images corresponding to the downsampled images;

[0089] Step 610, using the training sample set under each of the downsampling parameter values, the original deep learning model is trained to obtain an image reconstruction model under the corresponding downsampling parameter value.

[0090] The present application provides an image processing method. Figure 7 Schematic diagram of the implementation process of the image processing method of the present application embodiment. Figure 7 As shown, the method may include the following steps 701 to 703:

[0091] Step 701: Obtain an image to be processed.

[0092] As described in the above application scenario, the image to be processed can be any video frame image in a low-resolution video, or can be a single low-resolution image, such as an image taken by a user through a terminal.

[0093] Step 702, calling at least one trained image reconstruction model; wherein the image reconstruction model is obtained after training based on a training sample set including multiple frames of sample images and a true value image corresponding to each frame, the true value image is obtained by fusing multiple frames of reference images, and the resolution of the true value image is greater than the resolution of the corresponding sample image.

[0094] It is understandable that different image reconstruction models have different magnifications. However, in the online stage, that is, when these models are in use, in order to reduce the amount of calculation, not all image reconstruction models will be called, but some models will be called. In some embodiments, the device can determine the magnification of the image to be processed; according to the magnification to be processed, at least one image reconstruction model that matches the magnification to be processed is selected from the trained multiple image reconstruction models. In this way, the user's visual needs for image resolution can be met, the amount of calculation can be reduced, and the real-time requirements of high-resolution image display or high-resolution video display can be met.

[0095] Step 703: Perform super-resolution processing on the image to be processed by using the at least one image reconstruction model to obtain a target image; wherein the resolution of the target image is greater than the resolution of the image to be processed.

[0096] It can be understood that the at least one image reconstruction model may be one, or two or more. When the at least one image reconstruction model is one model, the model is directly used to convert the image to be processed, and the high-resolution image (referred to as high-resolution image) output by the model is used as the target image. When the at least one image reconstruction model is two or more, in some embodiments, the device can use each model of the at least one image reconstruction model to perform super-resolution processing on the image to be processed to obtain a high-resolution image output by the corresponding model; each of the high-resolution images is fused to obtain the target image.

[0097] In an embodiment of the present application, an image reconstruction model for super-resolution processing of an image to be processed is obtained by training a training sample set including multiple frames of sample images and a true value image corresponding to each frame, wherein the true value image is fused by multiple frames of reference images, rather than directly using a single frame of reference image as the true value image; in this way, the signal-to-noise ratio of the fused true value image is higher and the details are clearer, thereby improving the performance of the trained model, and then when the model is used, the image to be processed can be reconstructed into a higher quality target image, that is, the target image obtained has a higher signal-to-noise ratio and clearer details.

[0098] The present application embodiment provides another image processing method. Figure 8 Schematic diagram of the implementation process of the image processing method of the present application embodiment. Figure 8 As shown, the method may include the following steps 801 to 805:

[0099] Step 801, obtaining an image to be processed;

[0100] Step 802: Determine the magnification ratio of the image to be processed.

[0101] Typically, a user selects a desired magnification through an application interface, that is, the device receives an operation instruction indicating the magnification, and determines the magnification of the image to be processed according to the operation instruction.

[0102] Step 803, selecting, according to the magnification to be obtained, from a plurality of trained image reconstruction models, at least one image reconstruction model that matches the magnification to be obtained; wherein the image reconstruction model is obtained by training a training sample set including a plurality of frames of sample images and a true value image corresponding to each frame, the true value image is obtained by fusion of a plurality of frames of reference images, and the resolution of the true value image is greater than the resolution of the corresponding sample image;

[0103] Step 804, using each model of the at least one image reconstruction model, performing super-resolution processing on the image to be processed to obtain a high-resolution image output by the corresponding model;

[0104] Step 805: fuse each of the high-resolution images to obtain the target image.

[0105] From the above description, it can be seen that different image reconstruction models have different magnification capabilities and the corresponding magnification ratios are different. For example, the pre-trained image reconstruction models include model 1, model 2 and model 3; among them, model 1 has the ability of 2x magnification, model 2 has the ability of 4x magnification, and model 3 has the ability of 6x magnification. Assuming that the magnification ratio of the image to be processed is 2.3, if only model 1 is selected to reconstruct the image to be processed, the clarity of the target image obtained will not meet the user's needs; and if only model 2 is selected to reconstruct the image to be processed, the target image obtained will be too clear and will not meet the user's needs; based on this, in order to make the target image meet the image effect actually corresponding to the magnification ratio as much as possible, model 1 and model 2 can be selected to reconstruct the image to be processed respectively, and then the high-resolution images output by each are fused to obtain the target image.

[0106] In some embodiments, if the magnification to be magnified is different from the magnification of each model in the multiple image reconstruction models, the device can select two models closest to the magnification to be magnified as the at least one image reconstruction model to reconstruct the image to be processed respectively, and then fuse the high-resolution images output by the two models to obtain the target image.

[0107] In other embodiments, if the magnification to be processed is the same as the magnification of a model among the multiple image reconstruction models, the model can be used as the at least one image reconstruction model; accordingly, the image to be processed is reconstructed using the model, and the output high-resolution image is used as the target image.

[0108] Single image super-resolution (SISR) refers to the use of signal processing methods to restore low-resolution images to high-resolution images without changing the shooting hardware equipment.

[0109] Super-resolution methods mainly use data-driven deep learning models to reconstruct the required details to obtain accurate super-resolution.

[0110] Therefore, the quality of data pairs greatly affects the quality of high-resolution reconstruction by deep learning models. In previous deep learning models of single image super-resolution based on deep learning, bicubic downsampling or Gaussian blurred point downsampling are the two most commonly used degradation models. The process is to use the high-resolution image (HR Image) taken by SLR as the training truth of the deep learning model, and the low-resolution image (LR Image) obtained by bicubic downsampling or Gaussian blurred point downsampling of the HR image as the training input of the model to form a data pair.

[0111] The above two degradation models perform well in processing simulated datasets with the same degradation process. However, in complex real imaging systems, these degradation models cannot accurately simulate the real degradation process, resulting in a significant decrease in the performance of super-resolution algorithms on real images, such as images collected by smartphones.

[0112] Therefore, how to construct training data pairs that conform to the degradation model of the actual imaging system is the key to further improving the performance of the super-resolution deep learning model and reconstructing more realistic high-resolution images.

[0113] The related methods for generating super-resolution network training data are basically single-frame simulation methods, which use Bicubic or Gaussian fuzzy downsampling to obtain low-resolution images. Super-resolution networks trained with training data obtained by these two degradation models often perform well in processing simulated data with the same degradation process, but the performance of processing real images is significantly reduced. In recent years, in order to better simulate the complex real degradation process, there are also studies that add more other degradation factors (such as noise, blur, quantization) on the basis of downsampling to get closer to the real degradation process.

[0114] Although various simulated degradation methods have improved the reconstruction performance of deep learning-based super-resolution algorithms on noisy or blurred low-resolution images, they still cannot accurately model the degradation process in real imaging systems. Fig. 9 As shown, in the relevant literature, some researchers have raised the following question: Why does the high-resolution reconstructed image 902 obtained by the current state-of-the-art (SOTA) super-resolution model (such as ESRGAN) for smartphones look very unreal compared with the original image 901?

[0115] Based on this, in a related technology, the residual channel attention mechanism is used to effectively enhance useful features, suppress noise and ensure that multi-scale high-frequency image features can be accurately restored. However, the training data is to reduce the high-resolution image I by bicubic interpolation. HR To generate the corresponding low-resolution image I LR ,This approach works better for images simulated by bicubic interpolation, but the effect for ,real images will be greatly reduced.

[0116] In another related technology, high / low resolution images are actually captured by changing the field of view, so as to implicitly model the real degradation model in a data-driven manner. However, there are three problems with this: ① The high-resolution image captured in a single frame still has noise and blur, which is not enough to serve as the ground truth of the super-resolution neural network; ② The high-resolution and low-resolution images are captured twice, which not only makes it more difficult to align the image content, but also makes it very difficult to align the brightness and color. Local alignment failure will cause the result of super-resolution processing to appear locally blurred or even deformed; ③ For the Zoom application of mobile phones, images need to be captured separately at each Zoom focal length, which is a huge workload and difficult to achieve.

[0117] Based on this, an exemplary application of an embodiment of the present application in a practical application scenario will be described below.

[0118] In order to solve the above technical problems, the embodiment of the present application provides a method for generating real training data for super-resolution networks based on multi-frame fusion. On the one hand, through multi-frame fusion, compared with high-definition images collected by a single frame, the obtained Ground Truth signal-to-noise ratio can be higher and the details can be clearer; on the other hand, by using real collected images, there is no need for simulation, but the degradation effects (noise, quantization, etc.) produced by the real collection equipment (smartphones, SLRs, etc.) can be directly reflected on the image. At the same time, the combination of real device collection and simulation post-processing makes it possible to collect image pairs with multiple magnifications.

[0119] Fig.10 The main process of generating real training data for super-resolution network based on multi-frame fusion is shown. It mainly includes the following steps 1 to 4:

[0120] Step 1: Binning, that is, downsampling. Here, Gaussian blurring and then double-triple interpolation are used to obtain the Binning result.

[0121] Step 2: Multi-frame fusion: Use multiple frames of continuously acquired images to synthesize a training Ground Truth with higher clarity and lower noise.

[0122] Step 3: Post-processing: This is an optional step that simulates the image compression encoding process.

[0123] Step 4: Alignment. There may be a slight offset between the multi-frame fused image and the binning image. To prevent local blur or deformation of the model, the post-processed images need to be aligned.

[0124] The detailed description of the above steps is as follows:

[0125] For Binning in step 1:

[0126] Binning is the downsampling process, as shown in formula (2):

[0127] D b {u}=(H binning *H blur )(u)(2);

[0128] In formula (1), H blur represents Gaussian blur, * represents convolution operation, the Gaussian kernel size is the same as the downsample size. If the downsample is 2×2, the Gaussian kernel size is also 2×2, H binning Indicates the process of merging adjacent pixels, that is, downsampling. Here, the adjacent pixel averaging method is used for downsampling, simulating the hardware downsampling (hardware binning) process of the sampling device. b represents the binning parameter, that is, the size of downsample. If downsample is 2×2, the binning parameter b=2, which means averaging the adjacent 2×2 pixels.

[0129] For multi-frame fusion in step 2:

[0130] Multi-frame synthesis uses a large number of input frames to obtain a GroundTruth image with lower noise and clearer details. Fig.11 As shown, the process includes: selecting a reference frame (sharpness estimation), image registration and multi-frame averaging.

[0131] Among them, for sharpness estimation, a sharpness reference model based on local brightness features can be used to estimate the sharpness of the image, as shown in the following formula (3):

[0132]

[0133] The entire image is divided into k1*k2 windows, where I max，k，l and I min，k，l They represent the maximum brightness value and the minimum brightness value in the k*lth window respectively.

[0134] Among them, for image registration, the homography matrix estimation based on SIFT feature point detection can be used for image registration to estimate the Kth image Y K The homography matrix H to the reference frame image Y0 K , as shown in the following formula (4):

[0135]

[0136] In formula (3), (x, y) is Y K The coordinates of any point in the 3×3 matrix are H K , the converted point (x`, y`) is the coordinate of (x, y) after being aligned to the reference frame image Y0, w`=w=1.

[0137] So we can use the homography matrix H K Calculate Y K The displacement of each point relative to Y0 [mv xk , mv yk ], constituted by Y0 and Y K The offset vector diagram of two channels of the same size has the following main processes: Fig.12 As shown, the steps include steps 121 to 124:

[0138] Step 121, respectively, images Y0 and Y K Perform feature point detection;

[0139] Step 122, determining a feature point description of a feature point of each image;

[0140] Step 123, based on the feature point description, the two frames of images are matched with feature points;

[0141] Step 124: Calculate the homography matrix based on the feature point matching results.

[0142] Scale-invariant feature transform (SIFT) is the original algorithm for key point detection. Its essence is to find key points (feature points) in different scale spaces, calculate the size, direction, and scale information of the key points, and use this information to form key points to describe the feature points. The key points found by SIFT are very prominent and stable feature points that will not change due to factors such as lighting, affine transformation, and noise. After obtaining the feature points, the gradient histogram of the points around the feature points is used to form the feature vector This feature vector is the description of the current feature point. k Find the feature points of the two pictures and get two sets of feature points and

[0143] The process of feature point matching is to find Y0 and Y in the two sets of feature points obtained above. k The four or more feature point pairs that are closest to each other on the two images. In implementation, the closest feature point pairs can be searched using the Euclidean distance shown in the following formula (5).

[0144]

[0145] After finding the closest feature point pair, the homography matrix H can be solved by direct linear transformation (DLT) K The coefficient of Y K The displacement of each point relative to Y0 [mv x , mv y ]. Assume that through feature point matching, the obtained K The coordinates of the feature points on are (x1, y1), (x2, y2), ..., (x t ,y t ), the corresponding feature point coordinates on Y0 are (x`1, y`1), (x`2, y`2), ..., (x` t ,y` t ), apply the homography matrix to the corresponding point pairs, and rewrite the equation to obtain the following equation (6):

[0146] or AH K =0 (6);

[0147] In equation (6), A is a matrix with twice the number of rows of corresponding point pairs. The coefficients of these corresponding point pair equations are stacked into a matrix, and the singular value decomposition (SVD) algorithm can be used to find the least squares solution of H, thereby calculating each frame Y K The displacement relative to Y0 [mv xk , mv yk After estimating the displacement, we can convert Y K Pixel by pixel according to [mv xk , mv yk ] to obtain the result of aligning the kth frame to the reference frame, denoted as Y k` .

[0148] For multi-frame averaging, after registration, the images are numerically averaged on the Y channel according to the pixel position to obtain the final multi-frame synthesis result.

[0149] For the post-processing in step three, the following explanation is given here.

[0150] This step is optional, depending on whether the trained model is processing the image super-resolution process in the YUV->YUV domain or the compressed jpg->jpg format. If it is a YUV->YUV super-resolution model, there is no need to simulate the image encoding compression process; if it is a jpg->jpg super-resolution model, it is also necessary to consider reconstructing the details destroyed by the image encoding compression.

[0151] As shown in the following formula (7):

[0152] Xn compress =C c {Xn} (7);

[0153] In formula (7), n represents the binning parameter, C C {*} indicates image compression with compression strength c. The image can be encoded and compressed using the JPEG2000 compression standard. To ensure the robustness of the model, for the jpg->jpg super-resolution model, the compression encoding strength, that is, the quantization parameters (QP) are randomly selected from 0, 10, 20, 30, and 40, that is, covering different degrees of compression artifacts from no compression to strong compression.

[0154] For the alignment in step 4, here is the explanation:

[0155] There may be a slight offset between the image after multi-frame fusion and the image after Binning. To prevent the model from being locally blurred or deformed, the image registration method in step 2 is used to register Xn compression To Ground Truth Y gt Align to obtain the final super-resolution neural network training input image Xn SR input .

[0156] In the embodiment of the present application, a method based on multi-frame fusion to generate real training data for super-resolution neural networks is adopted; on the one hand, through multi-frame fusion, compared with high-definition images collected by a single frame, the obtained Ground Truth signal-to-noise ratio can be higher and the details can be clearer; on the other hand, by using real collected images, the degradation effects (noise, blur, quantization, etc.) produced by the real collection device (smartphone, single shot, etc.) can be directly reflected on the image without simulation. At the same time, the combination of real device collection and simulation post-processing makes it possible to collect image pairs with multiple magnifications.

[0157] An embodiment of the present application proposes a method for generating real training data for a super-resolution neural network using multi-frame synthesis, so that the performance of a super-resolution model based on deep learning on truly acquired images is better than that of data previously generated using simulated degradation, thereby improving the performance upper limit of deep learning applied to super-resolution algorithms, and taking into account artifacts generated by image compression encoding when processing LR images, thereby expanding the training data to the application of super-resolution models for processing encoded images such as jpg images.

[0158] In the embodiments of the present application, the image registration method includes but is not limited to the following methods:

[0159] As a possible implementation method, image registration can use Speed ​​Up Robust Feature (SURF), corner points or other features to detect and describe feature points.

[0160] As a possible implementation method, the optical flow vector of each pixel from the current frame to the reference frame is solved according to the brightness information around each point in the adjacent frames, and then the motion vector of the pixel is calculated according to the optical flow vector.

[0161] In the embodiment of the present application, the multi-frame averaging method includes but is not limited to the following methods:

[0162] As a possible implementation method, multi-frame averaging can also be performed as a weighted average according to the local sharpness estimation of the image.

[0163] As a possible implementation method, multi-frame averaging can also be performed in the frequency domain, such as performing a wavelet transform (WT) or a discrete Fourier transform, or performing a discrete cosine transform to convert the image signal to the frequency domain, perform image averaging, and then convert it back to the spatial domain.

[0164] In the embodiment of the present application, the post-processing image compression encoding method includes but is not limited to the following methods:

[0165] As a possible implementation method, compression coding can also adopt the H.265 / HEVC video coding standard for data application of video super-resolution models.

[0166] As a possible implementation manner, compression coding may also adopt JPEG-XR or MPEG series (such as MPEG-2) image compression standards.

[0167] Based on the foregoing embodiments, the image processing device and model training device provided in the embodiments of the present application may include the modules included and the units included in the modules, which may be implemented by a processor in an electronic device; of course, they may also be implemented by a specific logic circuit; in the implementation process, the processor may be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0168] Fig.13A Schematic diagram of the structure of the image processing device according to the embodiment of the present application. Fig.13A As shown, the device 130 includes an acquisition module 131, a calling module 132 and a super-resolution processing module 133, wherein:

[0169] An acquisition module 131 is used to acquire an image to be processed;

[0170] A calling module 132 is used to call at least one trained image reconstruction model; wherein the image reconstruction model is obtained by training based on a training sample set including multiple frames of sample images and a true value image corresponding to each frame, the true value image is obtained by fusing multiple frames of reference images, and the resolution of the true value image is greater than the resolution of the corresponding sample image;

[0171] The super-resolution processing module 133 is used to perform super-resolution processing on the image to be processed through the at least one image reconstruction model to obtain a target image.

[0172] In some embodiments, the calling module 132 is used to: determine the magnification of the image to be processed; and select at least one image reconstruction model that matches the magnification from multiple trained image reconstruction models based on the magnification; wherein different image reconstruction models correspond to different magnifications.

[0173] In some embodiments, the super-resolution processing module 133 is used to: use each model of the at least one image reconstruction model to perform super-resolution processing on the image to be processed to obtain a high-resolution image output by the corresponding model; and fuse each of the high-resolution images to obtain the target image.

[0174] In some embodiments, Fig. 13B As shown, the device 130 also includes a downsampling module 134, a sample generation module 135 and a model training module 136; wherein the downsampling module 134 is used to downsample the original image set according to each preset downsampling parameter value to obtain a downsampled image set under the corresponding downsampling parameter value; the sample generation module 135 is used to generate a training sample set based on the downsampled image set and the true value image corresponding to the downsampled image; the model training module 136 is used to train the original deep learning model using the training sample set under each of the downsampling parameter values ​​to obtain an image reconstruction model under the corresponding downsampling parameter value.

[0175] In some embodiments, the downsampling module 134 is used to use the downsampling parameter value as the size of the Gaussian kernel to perform Gaussian blur processing on the original image to obtain a Gaussian blurred image; and perform bicubic interpolation on the Gaussian blurred image according to the downsampling parameter value to obtain the downsampled image.

[0176] In some embodiments, the clarity is characterized by sharpness, and the acquisition module 131 is also used to: acquire N frames of candidate images, where N is an integer greater than 1; determine the clarity of each of the candidate images; and determine the image whose clarity meets specific conditions among the N frames of candidate images as the reference image.

[0177] In some embodiments, the acquisition module 131 is used to: estimate the sharpness of each of the candidate images to obtain the sharpness of the corresponding image; and determine the image whose sharpness is greater than a specific threshold among the N frames of candidate images as the reference image.

[0178] In some embodiments, the acquisition module 131 is further used to: select a frame image from the multiple frames of reference images as a first reference image; determine a first displacement of each other reference image in the multiple frames of reference images except the first reference image relative to the first reference image; shift each of the other reference images pixel by pixel according to the corresponding first displacement to align with the first reference image, thereby obtaining a corresponding second reference image; and fuse the first reference image with each of the second reference images to obtain the true value image.

[0179] In some embodiments, the acquisition module 131 is used to: perform feature point detection on each of the reference images and extract feature description information of each feature point; perform feature point matching on the other reference image of the i-th frame and the first reference image according to the feature description information of each feature point of the other reference image of the i-th frame and the first reference image to obtain a feature point matching pair set; wherein i is an integer greater than 0 and less than or equal to the total number of the other reference images; determine the homography matrix between the two frames of images according to the pixel coordinates of the feature points in the feature point matching pair set; and determine the first displacement of the other reference image of the i-th frame relative to the first reference image according to the homography matrix.

[0180] The present application embodiment provides a model training device, Fig.14A Schematic diagram of the structure of the model training device of the embodiment of the present application, as shown in Fig.14A As shown, the device 140 includes: a downsampling module 141, a sample generation module 142 and a model training module 143; wherein,

[0181] The downsampling module 141 is used to perform downsampling processing on the original image set according to each preset downsampling parameter value to obtain a downsampled image set under the corresponding downsampling parameter value;

[0182] A sample generation module 142 is used to generate a training sample set according to the downsampled image set and the true value image corresponding to the downsampled image; wherein the true value image is obtained by fusing multiple frames of reference images, and the resolution of the true value image is greater than the resolution of the corresponding sample image;

[0183] The model training module 143 is used to train the original deep learning model using the training sample set under each of the downsampling parameter values ​​to obtain an image reconstruction model under the corresponding downsampling parameter value.

[0184] In some embodiments, the downsampling module 141 is used to: use the downsampling parameter value as the size of the Gaussian kernel to perform Gaussian blur processing on the original image to obtain a Gaussian blurred image; and perform bicubic interpolation on the Gaussian blurred image according to the downsampling parameter value to obtain the downsampled image.

[0185] In some embodiments, Fig. 14B As shown, the device 140 also includes a determination module 144, which is used to: obtain N frames of candidate images, where N is an integer greater than 1; determine the clarity of each of the candidate images; and determine the image whose clarity meets a specific condition among the N frames of candidate images as the reference image.

[0186] In some embodiments, the clarity is characterized by sharpness, and the determination module 144 is used to: estimate the sharpness of each of the candidate images to obtain the sharpness of the corresponding image; and determine the image whose sharpness is greater than a specific threshold among the N frames of candidate images as the reference image.

[0187] In some embodiments, Fig. 14B As shown, the device 140 also includes an image fusion module 145, which is used to: select a frame image from the multiple reference image frames as a first reference image; determine a first displacement of each other reference image in the multiple reference image frames except the first reference image relative to the first reference image; shift each of the other reference images pixel by pixel according to the corresponding first displacement to align with the first reference image, thereby obtaining a corresponding second reference image; and fuse the first reference image with each of the second reference images to obtain the true value image.

[0188] In some embodiments, the image fusion module 145 is used to: perform feature point detection on each of the reference images and extract feature description information of each feature point; perform feature point matching on the other reference image of the i-th frame and the first reference image according to the feature description information of each feature point of the other reference image of the i-th frame and the first reference image to obtain a feature point matching pair set; wherein i is an integer greater than 0 and less than or equal to the total number of the other reference images; determine the homography matrix between the two frames of images according to the pixel coordinates of the feature points in the feature point matching pair set; and determine the first displacement of the other reference image of the i-th frame relative to the first reference image according to the homography matrix.

[0189] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0190] It should be noted that in the embodiment of the present application, if the above-mentioned image processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0191] Correspondingly, if Fig.15 As shown, the electronic device 150 provided in an embodiment of the present application may include: a memory 151 and a processor 152, wherein the memory 151 stores a computer program that can be executed on the processor 152, and the processor 152 implements the steps in the method provided in the above embodiment when executing the program.

[0192] The memory 151 is configured to store instructions and applications executable by the processor 152, and can also cache data to be processed or processed by the processor 152 and various modules in the electronic device 150 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0193] Correspondingly, the computer-readable storage medium provided in the embodiments of the present application stores a computer program thereon, and when the computer program is executed by a processor, the steps in the method provided in the above embodiments are implemented.

[0194] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0195] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0196] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0197] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0198] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0199] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0200] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0201] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0202] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0203] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0204] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0205] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method comprises: Get the image to be processed; Calling at least one trained image reconstruction model; wherein the image reconstruction model is obtained by training based on a training sample set including multiple frames of sample images and a true value image corresponding to each frame, the true value image is obtained by fusing multiple frames of reference images, and the resolution of the true value image is greater than the resolution of the corresponding sample image; Performing super-resolution processing on the image to be processed by using the at least one image reconstruction model to obtain a target image; The method for obtaining the reference image includes: Obtain N frames of candidate images, where N is an integer greater than 1; Determining the clarity of each of the candidate images; Determine an image whose definition meets a specific condition among the N frames of candidate images as the reference image; The clarity is represented by sharpness, and the determining the clarity of each candidate image includes: estimating the sharpness of each candidate image to obtain the sharpness of the corresponding image; Accordingly, the step of determining an image whose definition meets a specific condition among the N frames of candidate images as the reference image includes: An image whose sharpness is greater than a specific threshold among the N frames of candidate images is determined as the reference image.

2. The method according to claim 1, characterized in that The calling of at least one trained image reconstruction model comprises: Determining the magnification of the image to be processed; According to the magnification to be obtained, at least one image reconstruction model matching the magnification to be obtained is selected from a plurality of trained image reconstruction models; wherein different image reconstruction models correspond to different magnifications.

3. The method according to claim 1, characterized in that The step of performing super-resolution processing on the image to be processed by using the at least one image reconstruction model to obtain a target image comprises: Using each model of the at least one image reconstruction model, super-resolution processing is performed on the image to be processed to obtain a high-resolution image output by the corresponding model; Each of the high-resolution images is fused to obtain the target image.

4. The method according to claim 2, characterized in that: The training process of the multiple image reconstruction models includes: According to each preset downsampling parameter value, the original image set is downsampled to obtain a downsampled image set under the corresponding downsampling parameter value; Generate a training sample set according to the downsampled image set and the true value images corresponding to the downsampled images; The original deep learning model is trained using the training sample set under each of the downsampling parameter values ​​to obtain an image reconstruction model under the corresponding downsampling parameter value.

5. The method according to claim 4, characterized in that Downsampling the original image in the original image set according to the downsampling parameter value to obtain the downsampled image, including: Using the downsampling parameter value as the size of the Gaussian kernel, performing Gaussian blur processing on the original image to obtain a Gaussian blurred image; According to the downsampling parameter value, bicubic interpolation is performed on the Gaussian blurred image to obtain the downsampling image.

6. The method according to any one of claims 1 to 5, characterized in that: The method for obtaining the true value image includes: Selecting one frame of the multiple reference images as a first reference image; Determine a first displacement of each other reference image in the multiple reference image frames except the first reference image relative to the first reference image; Displacing each of the other reference images pixel by pixel according to the corresponding first displacement to align with the first reference image, thereby obtaining a corresponding second reference image; The first reference image is fused with each of the second reference images to obtain the true value image.

7. The method according to claim 6, characterized in that The determining a first displacement of each other reference image in the multiple reference image frames except the first reference image relative to the first reference image comprises: Performing feature point detection on each of the reference images, and extracting feature description information of each feature point; According to the feature description information of each feature point of the other reference image and the first reference image in the i-th frame, feature point matching is performed on the other reference image in the i-th frame and the first reference image to obtain a feature point matching pair set; wherein i is an integer greater than 0 and less than or equal to the total number of the other reference images; Determining a homography matrix between two frames of images according to pixel coordinates of feature points in the feature point matching pair set; A first displacement of the other reference image in the i-th frame relative to the first reference image is determined according to the homography matrix.

8. A model training method, characterized in that: The method comprises: According to each preset downsampling parameter value, the original image set is downsampled to obtain a downsampled image set under the corresponding downsampling parameter value; Generate a training sample set according to the downsampled image set and the true value image corresponding to the downsampled image; wherein the true value image is obtained by fusing multiple frames of reference images, and the resolution of the true value image is greater than the resolution of the corresponding sample image; Using the training sample set under each of the downsampling parameter values, the original deep learning model is trained to obtain an image reconstruction model under the corresponding downsampling parameter value; The method for obtaining the reference image includes: Obtain N frames of candidate images, where N is an integer greater than 1; Determining the clarity of each of the candidate images; Determine an image whose definition meets a specific condition among the N frames of candidate images as the reference image; The clarity is represented by sharpness, and the determining the clarity of each candidate image includes: estimating the sharpness of each candidate image to obtain the sharpness of the corresponding image; Accordingly, the step of determining an image whose definition meets a specific condition among the N frames of candidate images as the reference image includes: An image whose sharpness is greater than a specific threshold among the N frames of candidate images is determined as the reference image.

9. An image processing device, characterized in that include: An acquisition module, used for acquiring an image to be processed; A calling module, used to call at least one trained image reconstruction model; wherein the image reconstruction model is obtained after training based on a training sample set including multiple frames of sample images and a true value image corresponding to each frame, the true value image is obtained by fusing multiple frames of reference images, and the resolution of the true value image is greater than the resolution of the corresponding sample image; A super-resolution processing module, used to perform super-resolution processing on the image to be processed through the at least one image reconstruction model to obtain a target image; The acquisition module is also used to acquire N frames of candidate images, where N is an integer greater than 1; determine the clarity of each of the candidate images; and determine the image whose clarity meets a specific condition among the N frames of candidate images as the reference image; wherein the clarity is characterized by sharpness, and the determination of the clarity of each of the candidate images includes: estimating the sharpness of each of the candidate images to obtain the sharpness of the corresponding image; accordingly, determining the image whose clarity meets a specific condition among the N frames of candidate images as the reference image includes: determining the image whose sharpness is greater than a specific threshold among the N frames of candidate images as the reference image.

10. A model training device, characterized in that: The device comprises: A downsampling module is used to downsample the original image set according to each preset downsampling parameter value to obtain a downsampled image set under the corresponding downsampling parameter value; A sample generation module, used to generate a training sample set according to the downsampled image set and the true value image corresponding to the downsampled image; wherein the true value image is obtained by fusing multiple frames of reference images, and the resolution of the true value image is greater than the resolution of the corresponding sample image; A model training module, used to train the original deep learning model using the training sample set under each of the downsampling parameter values ​​to obtain an image reconstruction model under the corresponding downsampling parameter value; A determination module is used to obtain N frames of candidate images, where N is an integer greater than 1; determine the clarity of each of the candidate images; and determine an image whose clarity meets a specific condition among the N frames of candidate images as the reference image; wherein the clarity is characterized by sharpness, and the determination of the clarity of each of the candidate images includes: estimating the sharpness of each of the candidate images to obtain the sharpness of the corresponding image; accordingly, determining an image whose clarity meets a specific condition among the N frames of candidate images as the reference image includes: determining an image whose sharpness is greater than a specific threshold among the N frames of candidate images as the reference image.

11. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, it implements the steps of any one of the image processing methods of claims 1 to 7, or, when the processor executes the program, it implements the steps of the model training method of claim 8.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the image processing method described in any one of claims 1 to 7, or, when the computer program is executed by a processor, it implements the steps of the model training method described in claim 8.

Images