Image processing method, device, storage medium and terminal

Abstract

The present disclosure relates to an image processing method, device, storage medium and terminal, the method comprising: acquiring a low-quality image of a target scene by a terminal; inputting the low-quality image into a pre-trained image super-resolution model to obtain a super-resolution image corresponding to the low-quality image; the image super-resolution model is used to acquire at least one detail image corresponding to the low-quality image, and acquire the super-resolution image according to the at least one detail image, different detail images corresponding to different image scales. That is, the present disclosure can acquire at least one detail image of different image scales corresponding to the low-quality image, and then obtain the super-resolution image corresponding to the low-quality image according to the at least one detail image. In this way, the super-resolution image is clear in different scale detail images, thereby improving the quality of the super-resolution image.

Description

Technical Field

[0001] This disclosure relates to the field of terminal technology, and in particular to an image processing method, apparatus, storage medium and terminal. Background Technology

[0002] The demand for smartphone photography is growing, but smartphone cameras are limited by the physical size of the phone, and the image sensor size is relatively small, resulting in lower resolution images. Especially at higher magnification, the image clarity is poor and cannot meet user requirements.

[0003] In related technologies, multi-frame super-resolution algorithms are often used to recover the original high-resolution image from a low-quality image without increasing the hardware cost of the smartphone. However, the details in the high-resolution image recovered by existing multi-frame super-resolution algorithms are not clear enough, resulting in poor image quality. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides an image processing method, apparatus, storage medium, and terminal.

[0005] According to a first aspect of the present disclosure, an image processing method is provided, comprising:

[0006] Acquire low-quality images of the target scene through the terminal;

[0007] The low-quality image is input into a pre-trained image super-resolution model to obtain a super-resolution image corresponding to the low-quality image; the image super-resolution model is used to obtain at least one detail image corresponding to the low-quality image, and to obtain the super-resolution image based on at least one detail image, wherein different detail images correspond to different image scales.

[0008] Optionally, the image super-resolution model includes a first feature extraction sub-model, a second feature extraction sub-model, a third feature extraction sub-model, a first detail extraction sub-model, a second detail extraction sub-model, and a third detail extraction sub-model. The output of the first feature extraction sub-model is coupled to the input of the first detail extraction sub-model and the input of the second feature extraction sub-model, respectively. The output of the second feature extraction sub-model is coupled to the input of the second detail extraction sub-model and the input of the third feature extraction sub-model, respectively. The output of the third feature extraction sub-model is coupled to the input of the third detail extraction sub-model.

[0009] Optionally, the step of inputting the low-quality image into a pre-trained image super-resolution model to obtain the super-resolution image corresponding to the low-quality image includes:

[0010] The low-quality image is input into the first feature extraction sub-model to obtain the first scale feature image corresponding to the low-quality image;

[0011] The first-scale feature image is input into the second feature extraction sub-model to obtain the second-scale feature image corresponding to the low-quality image;

[0012] The second-scale feature image is input into the third feature extraction sub-model to obtain the third-scale feature image corresponding to the low-quality image;

[0013] The first scale feature image is input into the first detail extraction sub-model to obtain the first detail image corresponding to the low-quality image;

[0014] The second scale feature image is input into the second detail extraction sub-model to obtain the second detail image corresponding to the low-quality image;

[0015] The third-scale feature image is input into the third detail extraction sub-model to obtain the third detail image corresponding to the low-quality image;

[0016] Based on the first detail image, the second detail image, and the third detail image, obtain the super-resolution image corresponding to the low-quality image.

[0017] Optionally, the image super-resolution model is trained in the following manner:

[0018] Acquire multiple sample images and a ground truth image corresponding to each sample image, wherein the sample images include multiple frames of low-quality images continuously acquired at the same location in a preset scene;

[0019] The target neural network model is trained using multiple sample images and multiple ground truth images to obtain the image super-resolution model.

[0020] Optionally, training the target neural network model based on multiple sample images and multiple ground truth images to obtain the image super-resolution model includes:

[0021] The model training steps are executed repeatedly until the trained target neural network meets the preset stopping iteration condition based on the ground image and the predicted image. The trained target neural network model is then used as the image super-resolution model. The predicted image is the image output by the trained target neural network model after the sample image is input into it.

[0022] The model training steps include:

[0023] The target neural network model is input into multiple sample images, and the predicted image corresponding to each sample image is output.

[0024] If, based on the ground truth image and the predicted image, it is determined that the target neural network model does not meet the preset stopping iteration condition, a target loss value is determined based on the ground truth image and the predicted image, and the parameters of the target neural network model are updated based on the target loss value to obtain a trained target neural network model, which is then used as a new target neural network model.

[0025] Optionally, the step of inputting multiple sample images into the target neural network model and outputting a predicted image corresponding to each sample image includes:

[0026] For each sample image, the sample image is input into the target neural network model to obtain multiple sample detail images corresponding to the sample image, and the prediction image corresponding to the sample image is obtained based on the multiple sample detail images.

[0027] Optionally, obtaining the predicted image corresponding to the sample image based on the plurality of sample detail images includes:

[0028] The predicted image corresponding to the sample image is obtained by adding multiple sample detail images together.

[0029] Optionally, the target loss value includes a total loss value and multiple detail loss values; determining the target loss value based on the ground truth image and the predicted image includes:

[0030] The total loss value is determined based on the predicted image and the ground truth image;

[0031] Based on the multiple sample detail images and the ground truth image, determine multiple detail loss values;

[0032] The parameters for updating the target neural network model based on the target loss value include:

[0033] The parameters of the target neural network model are updated based on the total loss value and the detailed loss value.

[0034] Optionally, the sample detail images include a first sample detail image, a second sample detail image, and a third sample detail image; the detail loss values ​​include a first detail loss value, a second detail loss value, and a third detail loss value; determining the multiple detail loss values ​​based on the multiple detail images and the ground truth image includes:

[0035] Acquire the first sample detail image, the second sample detail image, and the third sample detail image;

[0036] The first detail loss value is determined based on the first sample detail image and the ground truth image;

[0037] The second detail loss value is determined based on the second sample detail image and the ground truth image;

[0038] The third detail loss value is determined based on the third sample detail image and the ground truth image.

[0039] Optionally, the target neural network model includes a first neural network model, a second neural network model, a third neural network model, a fourth neural network model, a fifth neural network model, and a sixth neural network model; obtaining the first sample detail image, the second sample detail image, and the third sample detail image includes:

[0040] The sample image is input into the first neural network model to obtain the first scale sample image corresponding to the sample image;

[0041] The first-scale sample image is input into the second neural network model to obtain the second-scale sample image corresponding to the sample image;

[0042] The second-scale sample image is input into the third neural network model to obtain the third-scale sample image corresponding to the sample image;

[0043] The first scale sample image is input into the fourth neural network model to obtain the first sample detail image corresponding to the sample image;

[0044] The second-scale sample image is input into the fifth neural network model to obtain the second sample detail image corresponding to the sample image;

[0045] The third-scale sample image is input into the sixth neural network model to obtain the third sample detail image corresponding to the sample image.

[0046] Optionally, updating the parameters of the target neural network model based on the total loss value and the detail loss value includes:

[0047] Based on the total loss value, update the parameters of the first neural network model, the second neural network model, the third neural network model, the fourth neural network model, the fifth neural network model, and the sixth neural network model;

[0048] Based on the first detail loss value, update the parameters of the first neural network model and the fourth neural network model;

[0049] Based on the second detail loss value, update the parameters of the second neural network model and the fifth neural network model;

[0050] The parameters of the third neural network model and the sixth neural network model are updated based on the third detail loss value.

[0051] Optionally, before determining the target loss value based on the ground truth image and the predicted image, the method further includes:

[0052] Obtain the edge image corresponding to the sample image;

[0053] Determining the target loss value based on the ground truth image and the predicted image includes:

[0054] The target loss value is determined based on the ground truth image, the predicted image, and the edge image.

[0055] Optionally, determining the target loss value based on the ground truth image, the predicted image, and the edge image includes:

[0056] Based on the edge image, determine the target weight value;

[0057] Based on the ground truth image and the predicted image, determine the undetermined loss value;

[0058] The product of the undetermined loss value and the target weight value is taken as the target loss value.

[0059] Optionally, before training the target neural network model based on the multiple sample images and the multiple ground truth images to obtain the image super-resolution model, the method further includes:

[0060] For each sample image, each frame of the sample image is degraded to obtain a low-quality sample image corresponding to the sample image;

[0061] The step of training the target neural network model based on multiple sample images and multiple ground truth images to obtain the image super-resolution model includes:

[0062] The target neural network model is trained using multiple low-quality sample images and multiple ground truth images to obtain the image super-resolution model.

[0063] Optionally, before training the target neural network model based on the multiple sample images and the multiple ground truth images to obtain the image super-resolution model, the method further includes:

[0064] For each of the sample images, the other frame images in the sample image except the first frame image are randomly translated to obtain the translated image corresponding to the sample image. For each frame image in the sample image, the pixel difference between the target frame image and the frame image is obtained. The target frame image is the image in the translated image that corresponds to the frame image. The frame weight corresponding to the frame image is determined according to the pixel difference.

[0065] The step of training the target neural network model based on multiple sample images and multiple ground truth images to obtain the image super-resolution model includes:

[0066] The target neural network model is trained based on multiple sample images and multiple frame weights to obtain the image super-resolution model.

[0067] According to a second aspect of the present disclosure, an image processing apparatus is provided, comprising:

[0068] The low-quality image acquisition module is configured to acquire low-quality images of the target scene through the terminal;

[0069] The super-resolution image acquisition module is configured to input the low-quality image into a pre-trained image super-resolution model to obtain a super-resolution image corresponding to the low-quality image; the image super-resolution model is used to acquire at least one detail image corresponding to the low-quality image, and to acquire the super-resolution image based on at least one detail image, wherein different detail images correspond to different image scales.

[0070] Optionally, the image super-resolution model includes a first feature extraction sub-model, a second feature extraction sub-model, a third feature extraction sub-model, a first detail extraction sub-model, a second detail extraction sub-model, and a third detail extraction sub-model. The output of the first feature extraction sub-model is coupled to the input of the first detail extraction sub-model and the input of the second feature extraction sub-model, respectively. The output of the second feature extraction sub-model is coupled to the input of the second detail extraction sub-model and the input of the third feature extraction sub-model, respectively. The output of the third feature extraction sub-model is coupled to the input of the third detail extraction sub-model.

[0071] Optionally, the super-resolution image acquisition module is further configured to:

[0072] The low-quality image is input into the first feature extraction sub-model to obtain the first scale feature image corresponding to the low-quality image;

[0073] The first-scale feature image is input into the second feature extraction sub-model to obtain the second-scale feature image corresponding to the low-quality image;

[0074] The second-scale feature image is input into the third feature extraction sub-model to obtain the third-scale feature image corresponding to the low-quality image;

[0075] The first scale feature image is input into the first detail extraction sub-model to obtain the first detail image corresponding to the low-quality image;

[0076] The second scale feature image is input into the second detail extraction sub-model to obtain the second detail image corresponding to the low-quality image;

[0077] The third-scale feature image is input into the third detail extraction sub-model to obtain the third detail image corresponding to the low-quality image;

[0078] Based on the first detail image, the second detail image, and the third detail image, obtain the super-resolution image corresponding to the low-quality image.

[0079] Optionally, the device further includes:

[0080] The model training module is configured to acquire multiple sample images and ground truth images corresponding to each sample image, wherein the sample images include multiple frames of low-quality images continuously acquired at the same location in a preset scene; and to train the target neural network model based on the multiple sample images and the multiple ground truth images to obtain the image super-resolution model.

[0081] Optionally, the model training module is further configured to:

[0082] The model training steps are executed repeatedly until the trained target neural network meets the preset stopping iteration condition based on the ground image and the predicted image. The trained target neural network model is then used as the image super-resolution model. The predicted image is the image output by the trained target neural network model after the sample image is input into it.

[0083] The model training steps include:

[0084] The target neural network model is input into multiple sample images, and the predicted image corresponding to each sample image is output.

[0085] If, based on the ground truth image and the predicted image, it is determined that the target neural network model does not meet the preset stopping iteration condition, a target loss value is determined based on the ground truth image and the predicted image, and the parameters of the target neural network model are updated based on the target loss value to obtain a trained target neural network model, which is then used as a new target neural network model.

[0086] Optionally, the model training module is further configured to:

[0087] For each sample image, the sample image is input into the target neural network model to obtain multiple sample detail images corresponding to the sample image, and the prediction image corresponding to the sample image is obtained based on the multiple sample detail images.

[0088] Optionally, the model training module is further configured to:

[0089] The predicted image corresponding to the sample image is obtained by adding multiple sample detail images together.

[0090] Optionally, the target loss value includes a total loss value and multiple detail loss values; the model training module is further configured to:

[0091] The total loss value is determined based on the predicted image and the ground truth image;

[0092] Based on the multiple sample detail images and the ground truth image, determine multiple detail loss values;

[0093] The parameters for updating the target neural network model based on the target loss value include:

[0094] The parameters of the target neural network model are updated based on the total loss value and the detailed loss value.

[0095] Optionally, the sample detail images include a first sample detail image, a second sample detail image, and a third sample detail image; the detail loss values ​​include a first detail loss value, a second detail loss value, and a third detail loss value; the model training module is further configured to:

[0096] Acquire the first sample detail image, the second sample detail image, and the third sample detail image; determine the first detail loss value based on the first sample detail image and the ground truth image;

[0097] The second detail loss value is determined based on the second sample detail image and the ground truth image;

[0098] The third detail loss value is determined based on the third sample detail image and the ground truth image.

[0099] Optionally, the target neural network model includes a first neural network model, a second neural network model, a third neural network model, a fourth neural network model, a fifth neural network model, and a sixth neural network model; the model training module is further configured to:

[0100] The sample image is input into the first neural network model to obtain the first scale sample image corresponding to the sample image;

[0101] The first-scale sample image is input into the second neural network model to obtain the second-scale sample image corresponding to the sample image;

[0102] The second-scale sample image is input into the third neural network model to obtain the third-scale sample image corresponding to the sample image;

[0103] The first scale sample image is input into the fourth neural network model to obtain the first sample detail image corresponding to the sample image;

[0104] The second-scale sample image is input into the fifth neural network model to obtain the second sample detail image corresponding to the sample image;

[0105] The third-scale sample image is input into the sixth neural network model to obtain the third sample detail image corresponding to the sample image.

[0106] Optionally, the model training module is further configured to:

[0107] Based on the total loss value, update the parameters of the first neural network model, the second neural network model, the third neural network model, the fourth neural network model, the fifth neural network model, and the sixth neural network model;

[0108] Based on the first detail loss value, update the parameters of the first neural network model and the fourth neural network model;

[0109] Based on the second detail loss value, update the parameters of the second neural network model and the fifth neural network model;

[0110] The parameters of the third neural network model and the sixth neural network model are updated based on the third detail loss value.

[0111] Optionally, the model training module is further configured to:

[0112] Obtain the edge image corresponding to the sample image;

[0113] Determining the target loss value based on the ground truth image and the predicted image includes:

[0114] The target loss value is determined based on the ground truth image, the predicted image, and the edge image.

[0115] Optionally, the model training module is further configured to:

[0116] Based on the edge image, determine the target weight value;

[0117] Based on the ground truth image and the predicted image, determine the undetermined loss value;

[0118] The product of the undetermined loss value and the target weight value is taken as the target loss value.

[0119] Optionally, the model training module is further configured to:

[0120] For each sample image, each frame of the sample image is degraded to obtain a low-quality sample image corresponding to the sample image;

[0121] The step of training the target neural network model based on multiple sample images and multiple ground truth images to obtain the image super-resolution model includes:

[0122] The target neural network model is trained using multiple low-quality sample images and multiple ground truth images to obtain the image super-resolution model.

[0123] Optionally, the model training module is further configured to:

[0124] For each of the sample images, the other frame images in the sample image except the first frame image are randomly translated to obtain the translated image corresponding to the sample image. For each frame image in the sample image, the pixel difference between the target frame image and the frame image is obtained. The target frame image is the image in the translated image that corresponds to the frame image. The frame weight corresponding to the frame image is determined according to the pixel difference.

[0125] The step of training the target neural network model based on multiple sample images and multiple ground truth images to obtain the image super-resolution model includes:

[0126] The target neural network model is trained based on multiple sample images and multiple frame weights to obtain the image super-resolution model.

[0127] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the steps of the image processing method provided in the first aspect of the present disclosure.

[0128] According to a fourth aspect of the present disclosure, a terminal is provided, comprising:

[0129] A memory on which computer programs are stored;

[0130] A processor is configured to execute the computer program in the memory to implement the steps of the image processing method provided in the first aspect of this disclosure.

[0131] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: acquiring a low-quality image of a target scene through a terminal; inputting the low-quality image into a pre-trained image super-resolution model to obtain a super-resolution image corresponding to the low-quality image; the image super-resolution model is used to acquire at least one detail image corresponding to the low-quality image, and to acquire the super-resolution image based on at least one detail image, wherein different detail images correspond to different image scales. In other words, this disclosure can acquire at least one detail image at different image scales corresponding to a low-quality image, and then obtain the super-resolution image corresponding to the low-quality image based on the at least one detail image. Thus, the detail images at different scales of the super-resolution image are relatively clear, thereby improving the quality of the super-resolution image.

[0132] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0133] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0134] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment;

[0135] Figure 2 This is a schematic diagram illustrating an image super-resolution model according to an exemplary embodiment;

[0136] Figure 3 This is a schematic diagram illustrating a feature extraction sub-model according to an exemplary embodiment;

[0137] Figure 4 This is a schematic diagram illustrating a detail extraction sub-model according to an exemplary embodiment;

[0138] Figure 5 This is a flowchart illustrating a training method for an image super-resolution model according to an exemplary embodiment;

[0139] Figure 6 This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment;

[0140] Figure 7 This is a block diagram illustrating another image processing apparatus according to an exemplary embodiment;

[0141] Figure 8 This is a block diagram illustrating a terminal according to an exemplary embodiment. Detailed Implementation

[0142] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0143] First, the application scenarios of this disclosure will be introduced. Multi-frame super-resolution algorithms refer to the process of continuously and rapidly acquiring multiple frames of low-quality images (up to 10 frames within 0.5 seconds) during mobile phone photography, and then fusing these images through a super-resolution algorithm to obtain a high-quality image (super-resolution image). The inventors discovered that although, according to the Nyquist sampling theorem, a high-resolution image can be recovered from a sufficiently large number of low-quality images, in reality, due to interference from sensor noise, jitter, and unknown blur during the imaging process, the damaged high-frequency image information becomes very difficult to recover. Furthermore, image details vary in scale, and existing multi-frame super-resolution algorithms produce super-resolution images with lower clarity at certain scales, resulting in poor super-resolution image quality.

[0144] To overcome the technical problems existing in the above-mentioned related technologies, this disclosure provides an image processing method, apparatus, storage medium and terminal. The method can acquire multiple detail images of different image scales corresponding to a low-quality image, and then obtain a super-resolution image corresponding to the low-quality image based on the multiple detail images. In this way, the detail images of the super-resolution image at different scales are relatively clear, thereby improving the quality of the super-resolution image.

[0145] The present disclosure will now be described in conjunction with specific embodiments.

[0146] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment, such as... Figure 1 As shown, the method may include:

[0147] S101. Obtain low-quality images of the target scene through the terminal.

[0148] The low-quality image may include multiple frames of images captured continuously in the target scene. For example, after receiving a shooting command triggered by the user, the terminal can capture 10 frames of images continuously.

[0149] S102. Input the low-quality image into the pre-trained image super-resolution model to obtain the super-resolution image corresponding to the low-quality image.

[0150] The image super-resolution model is used to obtain at least one detail image corresponding to the low-quality image, and to obtain the super-resolution image based on the at least one detail image. Different detail images correspond to different image scales.

[0151] Figure 2 This is a schematic diagram illustrating an image super-resolution model according to an exemplary embodiment, such as... Figure 2 As shown, the image super-resolution model may include a first feature extraction sub-model, a second feature extraction sub-model, a third feature extraction sub-model, a first detail extraction sub-model, a second detail extraction sub-model, and a third detail extraction sub-model. The output of the first feature extraction sub-model is coupled to the input of the first detail extraction sub-model and the input of the second feature extraction sub-model, respectively. The output of the second feature extraction sub-model is coupled to the input of the second detail extraction sub-model and the input of the third feature extraction sub-model, respectively. The output of the third feature extraction sub-model is coupled to the input of the third detail extraction sub-model.

[0152] In one possible implementation, the low-quality image can be input into the first feature extraction sub-model to obtain a first-scale feature image corresponding to the low-quality image; the first-scale feature image can be input into the second feature extraction sub-model to obtain a second-scale feature image corresponding to the low-quality image; the second-scale feature image can be input into the third feature extraction sub-model to obtain a third-scale feature image corresponding to the low-quality image; the first-scale feature image can be input into the first detail extraction sub-model to obtain a first detail image corresponding to the low-quality image; the second-scale feature image can be input into the second detail extraction sub-model to obtain a second detail image corresponding to the low-quality image; the third-scale feature image can be input into the third detail extraction sub-model to obtain a third detail image corresponding to the low-quality image; and a super-resolution image corresponding to the low-quality image can be obtained based on the first detail image, the second detail image, and the third detail image.

[0153] It should be noted that the specific structure of the image super-resolution model described above is merely illustrative. This image super-resolution model may also include more feature extraction sub-models and detail extraction sub-models to obtain more detail images corresponding to the low-quality image. This disclosure does not limit the number of feature extraction sub-models and detail extraction sub-models included in the image super-resolution model. Furthermore, the steps described above for obtaining the first detail image, the second detail image, and the third detail image are merely illustrative. This disclosure does not limit the execution order of the above steps. For example, the first-scale feature image can also be simultaneously input into the first detail extraction sub-model and the second feature extraction sub-model; this disclosure does not limit this approach.

[0154] in, Figure 3 This is a schematic diagram illustrating a feature extraction sub-model according to an exemplary embodiment, such as... Figure 3 As shown, this feature extraction sub-model can include conv (convolutional neural network), relu (activation function), and etl-add (elementwise addition). Figure 4 This is a schematic diagram illustrating a detail extraction sub-model according to an exemplary embodiment, such as... Figure 4 As shown, the detail extraction sub-model can include conv, relu, and etl-add.

[0155] In this step, after acquiring the low-quality image, it can be input into the image super-resolution model. The first feature extraction sub-model within the super-resolution model extracts features from the low-quality image, obtaining a first-scale feature image corresponding to the low-quality image. Then, the first-scale feature image can be downsampled. For example, if the pixel size of the first-scale feature image is 512*512, it can be downsampled to 256*256. The second feature extraction sub-model within the image super-resolution model then... Feature extraction is performed on the downsampled first-scale feature image to obtain the second-scale feature image corresponding to the low-quality image. Finally, the second-scale feature image can be downsampled. For example, if the pixel value of the second-scale feature image is 256*256, it can be downsampled to reduce the pixel value to 128*128. The third feature extraction sub-model in the image super-resolution model is then used to extract features from the downsampled second-scale feature image to obtain the third-scale feature image corresponding to the low-quality image.

[0156] For the obtained first-scale feature image, it can be upsampled first, and then the first detail extraction sub-model in the image super-resolution model can be used to extract features from the first-scale feature image to obtain the first detail image corresponding to the low-quality image. For the obtained second-scale feature image, it can be upsampled first, and then the second detail extraction sub-model in the image super-resolution model can be used to extract features from the second-scale feature image to obtain the second detail image corresponding to the low-quality image. For the obtained third-scale feature image, it can be upsampled first, and then the third detail extraction sub-model in the image super-resolution model can be used to extract features from the third-scale feature image to obtain the third detail image corresponding to the low-quality image.

[0157] Furthermore, after obtaining the first detail image, the second detail image, and the third detail image, the first detail image, the second detail image, and the third detail image can be added together to obtain the super-resolution image corresponding to the low-quality image. For example, the pixels of the first detail image, the second detail image, and the third detail image can be added together to obtain the super-resolution image corresponding to the low-quality image.

[0158] Using the above method, multiple detail images at different image scales corresponding to a low-quality image can be obtained. Then, a super-resolution image corresponding to the low-quality image can be obtained based on these multiple detail images. In this way, the detail images of the super-resolution image at different scales are relatively clear, thereby improving the quality of the super-resolution image.

[0159] Figure 5 This is a flowchart illustrating a training method for an image super-resolution model according to an exemplary embodiment, such as... Figure 5 As shown, the method may include:

[0160] S501. Obtain multiple sample images and the ground truth image corresponding to each sample image.

[0161] The sample image may include multiple low-quality images continuously captured at the same location in a preset scene. For example, multiple sample images can be acquired by a terminal. In order to avoid the terminal shaking during the acquisition of the sample image, which would cause large differences between multiple frames of the acquired sample image, the terminal can be fixed in the same position when acquiring the sample image, for example, by using a tripod.

[0162] It should be noted that, in order to ensure the accuracy of the image super-resolution model, the number of image frames in the sample image can be the same as the number of image frames in the low-quality image in the actual application. For example, if the sample image is 10 consecutively acquired images at the same location in a preset scene, then the low-quality image can also be 10 images of the target scene.

[0163] In this step, after obtaining the multiple sample images, each sample image is denoised using existing denoising methods. The denoised sample image is then used as the ground truth image corresponding to that sample image. To improve the accuracy of the ground truth image, images can be captured multiple times at the same location in the preset scene, with multiple frames captured each time. Then, any one of the captured frames can be used as a sample image, and the ground truth image corresponding to that sample image can be obtained based on these multiple captured frames. For example, images can be captured 50 times in the preset scene, with 10 frames captured each time. The first 10 frames are used as sample images, and the ground truth image corresponding to the sample image is obtained based on the 500 frames captured over 50 times. For instance, the average pixel value of each channel in the 500 frames can be obtained, and the ground truth image corresponding to the sample image can be obtained based on this average value.

[0164] In one possible implementation, after obtaining multiple sample images, for each sample image, each frame of that sample image can be degraded to obtain a corresponding low-quality sample image. The target neural network model is then trained based on multiple low-quality sample images and multiple ground truth images to obtain the image super-resolution model. This degradation process can include downsampling, Gaussian blur, motion blur, defocus blur, etc. For example, if the sample image includes 10 frames, these 10 frames can be randomly degraded. For instance, the first frame can be downsampled, the second frame can be motion blurred, the fifth frame can be Gaussian blurred, and the tenth frame can be defocused. Thus, since the sample images used to train the image super-resolution model include various low-quality sample images, even when the quality of these low-quality sample images is poor, the image super-resolution model can still obtain a high-quality super-resolution image.

[0165] S502. Train the target neural network model based on multiple sample images and multiple ground truth images to obtain the image super-resolution model.

[0166] In this step, after obtaining the multiple sample images and the ground truth image corresponding to each sample image, the model training steps can be executed repeatedly until the trained target neural network meets the preset stopping iteration condition based on the ground truth image and the predicted image. The trained target neural network model is then used as the image super-resolution model, and the predicted image is the image output by the sample image after inputting it into the trained target neural network model.

[0167] The model training steps may include:

[0168] S1. Input multiple sample images into the target neural network model and output the predicted image corresponding to each sample image.

[0169] In this step, for each sample image, the sample image can be input into the target neural network model to obtain multiple sample detail images corresponding to the sample image, and the predicted image corresponding to the sample image can be obtained based on the multiple sample detail images. Alternatively, the multiple sample detail images can be added together to obtain the predicted image corresponding to the sample image.

[0170] S2. If it is determined from the ground truth image and the predicted image that the target neural network model does not meet the preset stopping iteration condition, the target loss value is determined from the ground truth image and the predicted image, the parameters of the target neural network model are updated from the target loss value, the trained target neural network model is obtained, and the trained target neural network model is used as the new target neural network model.

[0171] The target loss value may include a total loss value and multiple detail loss values. The total loss value and multiple detail loss values ​​can be calculated by the loss function corresponding to the target neural network. The loss function can be a loss function commonly used in the prior art. The preset stopping iteration condition can be a stopping iteration condition commonly used in the prior art. This disclosure does not limit the loss function and the preset stopping iteration condition.

[0172] In this step, the total loss value can be determined based on the predicted image and the ground truth image, multiple detail loss values ​​can be determined based on multiple sample detail images and the ground truth image, and the parameters of the target neural network model can be updated based on the total loss value and the detail loss value.

[0173] The sample detail image may include a first sample detail image, a second sample detail image, and a third sample detail image; the detail loss value may include a first detail loss value, a second detail loss value, and a third detail loss value. The target neural network model may include a first neural network model, a second neural network model, a third neural network model, a fourth neural network model, a fifth neural network model, and a sixth neural network model.

[0174] In one possible implementation, the first sample detail image, the second sample detail image, and the third sample detail image can be acquired. Based on the first sample detail image and the ground truth image, the first detail loss value is determined. Based on the second sample detail image and the ground truth image, the second detail loss value is determined. Based on the third sample detail image and the ground truth image, the third detail loss value is determined.

[0175] Specifically, the sample image can be input into the first neural network model to obtain a first-scale sample image; the first-scale sample image can be input into the second neural network model to obtain a second-scale sample image; the second-scale sample image can be input into the third neural network model to obtain a third-scale sample image; the first-scale sample image can be input into the fourth neural network model to obtain a first-scale sample detail image; the second-scale sample image can be input into the fifth neural network model to obtain a second-scale sample detail image; and the third-scale sample image can be input into the sixth neural network model to obtain a third-scale sample detail image.

[0176] In one possible implementation, before determining the target loss value based on the ground truth image and the predicted image, an edge image corresponding to the sample image can be obtained. The target loss value is then determined based on the ground truth image, the predicted image, and the edge image. For example, the edge image corresponding to the sample image can be obtained using the Canny edge detection algorithm. After obtaining the edge image corresponding to the sample image, a target weight value can be determined based on the edge image. A pending loss value is determined based on the ground truth image and the predicted image, and the product of the pending loss value and the target weight value is used as the target loss value. Specifically, after obtaining the edge image corresponding to the sample image, the edge image can be normalized to obtain its corresponding grayscale value. Then, the target weight value corresponding to this grayscale value can be determined through a pre-set weight association relationship, which can include the correspondence between different grayscale values ​​and weight values. In this way, the weight values ​​of textured regions (with higher grayscale values) in the image can be set to be larger, while the weight values ​​of flat regions (with lower grayscale values) can be set to be smaller. This allows the super-resolution model to reflect the high-frequency details of the image, resulting in a better quality super-resolution image even when the low-quality image is affected by high-frequency details.

[0177] It should be noted that the method described above for determining the target loss value can also be applied to the total loss value, the first detailed loss value, the second detailed loss value, and the third detailed loss value, and this disclosure does not limit this.

[0178] Furthermore, after obtaining the total loss value, the first detail loss value, the second detail loss value, and the third detail loss value, the parameters of the first neural network model, the second neural network model, the third neural network model, the fourth neural network model, the fifth neural network model, and the sixth neural network model can be updated based on the total loss value; the parameters of the first neural network model and the fourth neural network model can be updated based on the first detail loss value; the parameters of the second neural network model and the fifth neural network model can be updated based on the second detail loss value; and the parameters of the third neural network model and the sixth neural network model can be updated based on the third detail loss value. The loss functions corresponding to the first neural network model, the second neural network model, the third neural network model, the fourth neural network model, the fifth neural network model, and the sixth neural network model can be loss functions commonly used in the prior art. The loss functions corresponding to different neural network models can be the same or different, and this disclosure does not impose any limitations on this.

[0179] It should be noted that the method of updating the parameters of the neural network model based on the loss value described above can refer to the existing update methods, which will not be elaborated here.

[0180] In one possible implementation, for each sample image, all frames in the sample image except the first frame are randomly translated to obtain a translated image corresponding to the sample image. For each frame in the sample image, the pixel difference between the target frame image and the target frame image is obtained. The target frame image is the image corresponding to the target frame image in the translated image. The frame weight corresponding to the target frame image is determined based on the pixel difference. The target neural network model is trained based on multiple sample images and multiple frame weights to obtain the image super-resolution model. The random translation can be moving the image to any angle.

[0181] For example, if the sample image comprises 10 frames, the second to tenth frames of the sample image can be randomly shifted to obtain the shifted image corresponding to the sample image. Then, the pixel difference between the first frame of the sample image and the first frame of the shifted image can be obtained, as can the pixel difference between the second frame of the sample image and the second frame of the shifted image, and so on, until the pixel difference between the tenth frame of the sample image and the tenth frame of the shifted image is obtained. Finally, the alignment degree of the multiple frames in the sample image can be determined based on the pixel difference values. For example, a smaller pixel difference value indicates better alignment of the multiple frames in the sample image, while a larger pixel difference value indicates poorer alignment. If the alignment of multiple frames in the sample image is relatively good, the frame weight of each frame in the sample image can be set to a preset average weight value. If the alignment of multiple frames in the sample image is relatively poor, the frame weight of the first frame in the sample image can be set to a preset maximum weight value, and the frame weights of the other frames in the sample image can be set to preset minimum weight values. In this way, even if the low-quality image alignment of the input image super-resolution model fails, a better quality super-resolution image can still be obtained.

[0182] It should be noted that the frame weight setting method in the above example is only for illustration, and this disclosure does not limit the frame weight setting method.

[0183] Using the above method, multiple sample detail images at different image scales corresponding to the sample image can be obtained. A prediction image corresponding to the sample image is then obtained from these multiple detail images. The target neural network model is trained using both the prediction image and the ground truth image corresponding to the sample image to obtain an image super-resolution model. This trained image super-resolution model can represent the detail images of low-quality images at different image scales, thus improving its accuracy. Furthermore, before training the image super-resolution model, the sample image can be downgraded or shifted, allowing the model to adapt to more types of low-quality or misaligned images, further improving its accuracy.

[0184] Figure 6 This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment, such as... Figure 6 As shown, the device may include:

[0185] The low-quality image acquisition module 601 is configured to acquire low-quality images of the target scene through a terminal;

[0186] The super-resolution image acquisition module 602 is configured to input the low-quality image into a pre-trained image super-resolution model to obtain a super-resolution image corresponding to the low-quality image; the image super-resolution model is used to acquire at least one detail image corresponding to the low-quality image, and to acquire the super-resolution image based on at least one detail image, wherein different detail images correspond to different image scales.

[0187] Optionally, the image super-resolution model includes a first feature extraction sub-model, a second feature extraction sub-model, a third feature extraction sub-model, a first detail extraction sub-model, a second detail extraction sub-model, and a third detail extraction sub-model. The output of the first feature extraction sub-model is coupled to the input of the first detail extraction sub-model and the input of the second feature extraction sub-model, respectively. The output of the second feature extraction sub-model is coupled to the input of the second detail extraction sub-model and the input of the third feature extraction sub-model, respectively. The output of the third feature extraction sub-model is coupled to the input of the third detail extraction sub-model.

[0188] Optionally, the super-resolution image acquisition module is further configured to:

[0189] The low-quality image is input into the first feature extraction sub-model to obtain the first scale feature image corresponding to the low-quality image;

[0190] The first-scale feature image is input into the second feature extraction sub-model to obtain the second-scale feature image corresponding to the low-quality image;

[0191] The second-scale feature image is input into the third-scale feature extraction sub-model to obtain the third-scale feature image corresponding to the low-quality image;

[0192] The first scale feature image is input into the first detail extraction sub-model to obtain the first detail image corresponding to the low-quality image;

[0193] The second-scale feature image is input into the second detail extraction sub-model to obtain the second detail image corresponding to the low-quality image;

[0194] The third-scale feature image is input into the third detail extraction sub-model to obtain the third detail image corresponding to the low-quality image;

[0195] Based on the first detail image, the second detail image, and the third detail image, obtain the super-resolution image corresponding to the low-quality image.

[0196] Optionally, Figure 7 This is a block diagram of another image processing apparatus according to an exemplary embodiment, such as Figure 7 As shown, the device also includes:

[0197] The model training module 603 is configured to acquire multiple sample images and ground truth images corresponding to each sample image, wherein the sample images include multiple frames of low-quality images continuously acquired at the same location in a preset scene; and to train the target neural network model based on the multiple sample images and the multiple ground truth images to obtain the image super-resolution model.

[0198] Optionally, the model training module 603 is also configured as follows:

[0199] The model training steps are executed iteratively until the trained target neural network meets the preset stopping iteration condition based on the ground truth image and the predicted image. The trained target neural network model is then used as the image super-resolution model. The predicted image is the image output by the trained target neural network model after the sample image is input into it.

[0200] The training steps for this model include:

[0201] Input multiple sample images into the target neural network model, and output the predicted image corresponding to each sample image;

[0202] If, based on the ground truth image and the predicted image, it is determined that the target neural network model does not meet the preset stopping iteration condition, the target loss value is determined based on the ground truth image and the predicted image, the parameters of the target neural network model are updated based on the target loss value, the trained target neural network model is obtained, and the trained target neural network model is used as the new target neural network model.

[0203] Optionally, the model training module 603 is also configured as follows:

[0204] For each sample image, the sample image is input into the target neural network model to obtain multiple sample detail images corresponding to the sample image, and the prediction image corresponding to the sample image is obtained based on the multiple sample detail images.

[0205] Optionally, the model training module 603 is also configured as follows:

[0206] The predicted image corresponding to the sample image is obtained by summing multiple detailed images of the sample.

[0207] Optionally, the target loss value includes a total loss value and multiple detail loss values; the model training module 603 is also configured to:

[0208] The total loss value is determined based on the predicted image and the ground truth image;

[0209] Based on multiple detailed images of the sample and the ground truth image, determine multiple detail loss values;

[0210] The parameters for updating the target neural network model based on the target loss value include:

[0211] The parameters of the target neural network model are updated based on the total loss value and the detailed loss value.

[0212] Optionally, the sample detail image includes a first sample detail image, a second sample detail image, and a third sample detail image; the detail loss value includes a first detail loss value, a second detail loss value, and a third detail loss value; the model training module 603 is further configured to:

[0213] Obtain the detailed image of the first sample, the detailed image of the second sample, and the detailed image of the third sample;

[0214] The first detail loss value is determined based on the first sample detail image and the ground truth image;

[0215] The second detail loss value is determined based on the second sample detail image and the ground truth image;

[0216] The third detail loss value is determined based on the third sample detail image and the ground truth image.

[0217] Optionally, the target neural network model includes a first neural network model, a second neural network model, a third neural network model, a fourth neural network model, a fifth neural network model, and a sixth neural network model; the model training module is further configured as follows:

[0218] The sample image is input into the first neural network model to obtain the first scale sample image corresponding to the sample image;

[0219] The first-scale sample image is input into the second neural network model to obtain the second-scale sample image corresponding to the first sample image.

[0220] The second-scale sample image is input into the third neural network model to obtain the third-scale sample image corresponding to the sample image;

[0221] The first-scale sample image is input into the fourth neural network model to obtain the first sample detail image corresponding to the sample image;

[0222] The second-scale sample image is input into the fifth neural network model to obtain the second sample detail image corresponding to the sample image;

[0223] The third-scale sample image is input into the sixth neural network model to obtain the third sample detail image corresponding to the sample image;

[0224] The model training module 603 is also configured as follows:

[0225] Based on the total loss value, update the parameters of the first neural network model, the second neural network model, the third neural network model, the fourth neural network model, the fifth neural network model, and the sixth neural network model;

[0226] Based on the first detail loss value, update the parameters of the first neural network model and the fourth neural network model;

[0227] Based on the second detail loss value, update the parameters of the second neural network model and the fifth neural network model;

[0228] Based on the third detail loss value, update the parameters of the third neural network model and the sixth neural network model.

[0229] Optionally, the model training module 603 is also configured as follows:

[0230] Obtain the edge image corresponding to the sample image;

[0231] Determining the target loss value based on the ground truth image and the predicted image includes:

[0232] The target loss value is determined based on the ground truth image, the predicted image, and the edge image.

[0233] Optionally, the model training module 603 is also configured as follows:

[0234] Based on the edge image, determine the target weight value;

[0235] Based on the ground truth image and the predicted image, determine the value of the undetermined loss.

[0236] The product of the undetermined loss value and the target weight value is taken as the target loss value.

[0237] Optionally, the model training module 603 is also configured as follows:

[0238] For each sample image, each frame of that sample image is degraded to obtain the corresponding low-quality sample image.

[0239] The target neural network model is trained using multiple sample images and multiple ground truth images to obtain the image super-resolution model, which includes:

[0240] The target neural network model is trained using multiple low-quality images of the sample and multiple ground truth images to obtain the image super-resolution model.

[0241] Optionally, the model training module 603 is also configured as follows:

[0242] For each sample image, the other frames in the sample image except the first frame image are randomly translated to obtain the translated image corresponding to the sample image. For each frame image in the sample image, the pixel difference between the target frame image and the frame image is obtained. The target frame image is the image in the translated image that corresponds to the frame image. The frame weight corresponding to the frame image is determined based on the pixel difference.

[0243] The target neural network model is trained using multiple sample images and multiple ground truth images to obtain the image super-resolution model, which includes:

[0244] The target neural network model is trained using multiple sample images and multiple frame weights to obtain the image super-resolution model.

[0245] The above-mentioned device can acquire multiple detail images at different image scales corresponding to a low-quality image, and then obtain a super-resolution image corresponding to the low-quality image based on these multiple detail images. In this way, the detail images of the super-resolution image at different scales are relatively clear, thereby improving the quality of the super-resolution image.

[0246] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0247] This disclosure also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the steps of the image processing method provided in this disclosure.

[0248] Figure 8 This is a block diagram illustrating a terminal 800 according to an exemplary embodiment. For example, terminal 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc.

[0249] Reference Figure 8 The terminal 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0250] Processing component 802 typically controls the overall operation of terminal 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the image processing method described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0251] Memory 804 is configured to store various types of data to support operation on terminal 800. Examples of this data include instructions for any application or method operating on terminal 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0252] Power component 806 provides power to various components of terminal 800. Power component 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to terminal 800.

[0253] Multimedia component 808 includes a screen that provides an output interface between the terminal 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the terminal 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0254] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when terminal 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0255] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0256] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of terminal 800. For example, sensor assembly 814 can detect the on / off state of terminal 800, the relative positioning of components such as the display and keypad of terminal 800, changes in position of terminal 800 or one of its components, the presence or absence of user contact with terminal 800, orientation or acceleration / deceleration of terminal 800, and temperature changes of terminal 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0257] Communication component 816 is configured to facilitate wired or wireless communication between terminal 800 and other devices. Terminal 800 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0258] In an exemplary embodiment, terminal 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the image processing method described above.

[0259] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of a terminal 800 to complete the image processing method described above. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0260] In another exemplary embodiment, a computer program product is also provided, which includes a computer program executable by a programmable device, the computer program having a code portion for performing the image processing method described above when executed by the programmable device.

[0261] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of this disclosure. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0262] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, include: Acquire low-quality images of the target scene through the terminal; The low-quality image is input into a pre-trained image super-resolution model to obtain the super-resolution image corresponding to the low-quality image; The image super-resolution model is used to obtain at least one detail image corresponding to the low-quality image, and to obtain the super-resolution image based on at least one detail image, wherein different detail images correspond to different image scales; The image super-resolution model includes a first feature extraction sub-model, a second feature extraction sub-model, a third feature extraction sub-model, a first detail extraction sub-model, a second detail extraction sub-model, and a third detail extraction sub-model. The output of the first feature extraction sub-model is coupled to the input of the first detail extraction sub-model and the input of the second feature extraction sub-model. The output of the second feature extraction sub-model is coupled to the input of the second detail extraction sub-model and the input of the third feature extraction sub-model. The output of the third feature extraction sub-model is coupled to the input of the third detail extraction sub-model. The step of inputting the low-quality image into a pre-trained image super-resolution model to obtain the super-resolution image corresponding to the low-quality image includes: The low-quality image is input into the first feature extraction sub-model to obtain the first scale feature image corresponding to the low-quality image; The first-scale feature image is input into the second feature extraction sub-model to obtain the second-scale feature image corresponding to the low-quality image; The second-scale feature image is input into the third feature extraction sub-model to obtain the third-scale feature image corresponding to the low-quality image; The first scale feature image is input into the first detail extraction sub-model to obtain the first detail image corresponding to the low-quality image; The second scale feature image is input into the second detail extraction sub-model to obtain the second detail image corresponding to the low-quality image; The third-scale feature image is input into the third detail extraction sub-model to obtain the third detail image corresponding to the low-quality image; Based on the first detail image, the second detail image, and the third detail image, obtain the super-resolution image corresponding to the low-quality image.

2. The method according to claim 1, characterized in that, The image super-resolution model is trained in the following way: Acquire multiple sample images and a ground truth image corresponding to each sample image, wherein the sample images include multiple frames of low-quality images continuously acquired at the same location in a preset scene; The target neural network model is trained using multiple sample images and multiple ground truth images to obtain the image super-resolution model.

3. The method according to claim 2, characterized in that, The step of training the target neural network model based on multiple sample images and multiple ground truth images to obtain the image super-resolution model includes: The model training steps are executed repeatedly until the trained target neural network meets the preset stopping iteration condition based on the ground image and the predicted image. The trained target neural network model is then used as the image super-resolution model. The predicted image is the image output by the trained target neural network model after the sample image is input into it. The model training steps include: The target neural network model is input into multiple sample images, and the predicted image corresponding to each sample image is output. If, based on the ground truth image and the predicted image, it is determined that the target neural network model does not meet the preset stopping iteration condition, a target loss value is determined based on the ground truth image and the predicted image, and the parameters of the target neural network model are updated based on the target loss value to obtain a trained target neural network model, which is then used as a new target neural network model.

4. The method according to claim 3, characterized in that, The step of inputting multiple sample images into the target neural network model and outputting a predicted image corresponding to each sample image includes: For each sample image, the sample image is input into the target neural network model to obtain multiple sample detail images corresponding to the sample image, and the prediction image corresponding to the sample image is obtained based on the multiple sample detail images.

5. The method according to claim 4, characterized in that, The step of obtaining the predicted image corresponding to the sample image based on multiple sample detail images includes: The predicted image corresponding to the sample image is obtained by adding multiple sample detail images together.

6. The method according to claim 4, characterized in that, The target loss value includes a total loss value and multiple detail loss values; determining the target loss value based on the ground truth image and the predicted image includes: The total loss value is determined based on the predicted image and the ground truth image; Based on the multiple sample detail images and the ground truth image, determine multiple detail loss values; The parameters for updating the target neural network model based on the target loss value include: The parameters of the target neural network model are updated based on the total loss value and the detailed loss value.

7. The method according to claim 6, characterized in that, The sample detail images include a first sample detail image, a second sample detail image, and a third sample detail image; the detail loss values ​​include a first detail loss value, a second detail loss value, and a third detail loss value; determining the multiple detail loss values ​​based on the multiple detail images and the ground truth image includes: Acquire the first sample detail image, the second sample detail image, and the third sample detail image; The first detail loss value is determined based on the first sample detail image and the ground truth image; The second detail loss value is determined based on the second sample detail image and the ground truth image; The third detail loss value is determined based on the third sample detail image and the ground truth image.

8. The method according to claim 7, characterized in that, The target neural network model includes a first neural network model, a second neural network model, a third neural network model, a fourth neural network model, a fifth neural network model, and a sixth neural network model; obtaining the first sample detail image, the second sample detail image, and the third sample detail image includes: The sample image is input into the first neural network model to obtain the first scale sample image corresponding to the sample image; The first-scale sample image is input into the second neural network model to obtain the second-scale sample image corresponding to the sample image; The second-scale sample image is input into the third neural network model to obtain the third-scale sample image corresponding to the sample image; The first scale sample image is input into the fourth neural network model to obtain the first sample detail image corresponding to the sample image; The second-scale sample image is input into the fifth neural network model to obtain the second sample detail image corresponding to the sample image; The third-scale sample image is input into the sixth neural network model to obtain the third sample detail image corresponding to the sample image.

9. The method according to claim 8, characterized in that, The step of updating the parameters of the target neural network model based on the total loss value and the detail loss value includes: Based on the total loss value, update the parameters of the first neural network model, the second neural network model, the third neural network model, the fourth neural network model, the fifth neural network model, and the sixth neural network model; Based on the first detail loss value, update the parameters of the first neural network model and the fourth neural network model; Based on the second detail loss value, update the parameters of the second neural network model and the fifth neural network model; The parameters of the third neural network model and the sixth neural network model are updated based on the third detail loss value.

10. The method according to claim 3, characterized in that, Before determining the target loss value based on the ground truth image and the predicted image, the method further includes: Obtain the edge image corresponding to the sample image; Determining the target loss value based on the ground truth image and the predicted image includes: The target loss value is determined based on the ground truth image, the predicted image, and the edge image.

11. The method according to claim 10, characterized in that, Determining the target loss value based on the ground truth image, the predicted image, and the edge image includes: Based on the edge image, determine the target weight value; Based on the ground truth image and the predicted image, determine the undetermined loss value; The product of the undetermined loss value and the target weight value is taken as the target loss value.

12. The method according to claim 2, characterized in that, Before training the target neural network model based on multiple sample images and multiple ground truth images to obtain the image super-resolution model, the method further includes: For each of the sample images, each frame of the sample image is degraded to obtain a low-quality sample image corresponding to the sample image. The step of training the target neural network model based on multiple sample images and multiple ground truth images to obtain the image super-resolution model includes: The target neural network model is trained using multiple low-quality sample images and multiple ground truth images to obtain the image super-resolution model.

13. The method according to claim 2, characterized in that, Before training the target neural network model based on multiple sample images and multiple ground truth images to obtain the image super-resolution model, the method further includes: For each of the sample images, the other frame images in the sample image except the first frame image are randomly translated to obtain the translated image corresponding to the sample image. For each frame image in the sample image, the pixel difference between the target frame image and the frame image is obtained. The target frame image is the image in the translated image that corresponds to the frame image. The frame weight corresponding to the frame image is determined according to the pixel difference. The step of training the target neural network model based on multiple sample images and multiple ground truth images to obtain the image super-resolution model includes: The target neural network model is trained based on multiple sample images and multiple frame weights to obtain the image super-resolution model.

14. An image processing apparatus, characterized in that, include: The low-quality image acquisition module is configured to acquire low-quality images of the target scene through the terminal; The super-resolution image acquisition module is configured to input the low-quality image into a pre-trained image super-resolution model to obtain a super-resolution image corresponding to the low-quality image; the image super-resolution model is used to acquire at least one detail image corresponding to the low-quality image, and to acquire the super-resolution image based on at least one detail image, wherein different detail images correspond to different image scales; The image super-resolution model includes a first feature extraction sub-model, a second feature extraction sub-model, a third feature extraction sub-model, a first detail extraction sub-model, a second detail extraction sub-model, and a third detail extraction sub-model. The output of the first feature extraction sub-model is coupled to the input of the first detail extraction sub-model and the input of the second feature extraction sub-model. The output of the second feature extraction sub-model is coupled to the input of the second detail extraction sub-model and the input of the third feature extraction sub-model. The output of the third feature extraction sub-model is coupled to the input of the third detail extraction sub-model. The super-resolution image acquisition module is configured to input the low-quality image into the first feature extraction sub-model to obtain a first-scale feature image corresponding to the low-quality image; input the first-scale feature image into the second feature extraction sub-model to obtain a second-scale feature image corresponding to the low-quality image; input the second-scale feature image into the third feature extraction sub-model to obtain a third-scale feature image corresponding to the low-quality image; input the first-scale feature image into the first detail extraction sub-model to obtain a first detail image corresponding to the low-quality image; input the second-scale feature image into the second detail extraction sub-model to obtain a second detail image corresponding to the low-quality image; input the third-scale feature image into the third detail extraction sub-model to obtain a third detail image corresponding to the low-quality image; and acquire the super-resolution image corresponding to the low-quality image based on the first detail image, the second detail image, and the third detail image.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-13.

16. A terminal, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-13.

Images