Image super-resolution method and device, electronic equipment and storage medium

Abstract

The application discloses an image super-resolution method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence. The method is applied to an electronic equipment, the electronic equipment is deployed with an image super-resolution model, the image super-resolution model adopts a UNet structure, the UNet structure comprises a plurality of skip connections, and a spatial attention module is configured on each skip connection in the plurality of skip connections. The method comprises the following steps: acquiring a to-be-processed image, inputting the to-be-processed image into the image super-resolution model, and obtaining a target image corresponding to the to-be-processed image output by the image super-resolution model, wherein the resolution of the target image is higher than that of the to-be-processed image. The application increases the spatial attention mechanism on the global skip connection of the UNet structure based on the UNet structure of the image super-resolution model, so that the image super-resolution effect is more stable, and the generated target image is more real and natural.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and more particularly, to an image super-resolution method and device, an electronic device, and a storage medium. BACKGROUND

[0002] In photographing of an electronic device (such as a smart phone), improving image definition has always been a key issue. Among them, image super-resolution refers to reconstructing an input low-resolution image, that is, a small-size image with relatively few pixels and poor visual effect, into a clear image with relatively many pixels and good visual effect. However, the current image super-resolution technology has the problem of detail error reconstruction, resulting in unnatural generated images. SUMMARY

[0003] In view of the above problems, the present application provides an image super-resolution method, device, electronic device, and storage medium to solve the above problems.

[0004] In a first aspect, an image super-resolution method is provided by the embodiments of the present application, which is applied to an electronic device, the electronic device is deployed with an image super-resolution model, the image super-resolution model adopts a semantic segmentation network UNet structure, the UNet structure includes a plurality of skip connections, and a spatial attention module is configured on each skip connection in the plurality of skip connections. The method comprises: acquiring a to-be-processed image; inputting the to-be-processed image into the image super-resolution model to obtain a target image corresponding to the to-be-processed image output by the image super-resolution model, wherein the resolution of the target image is higher than that of the to-be-processed image.

[0005] In a second aspect, an image super-resolution device is provided by the embodiments of the present application, which is applied to an electronic device, the electronic device is deployed with an image super-resolution model, the image super-resolution model adopts a semantic segmentation network UNet structure, the UNet structure includes a plurality of skip connections, and a spatial attention module is configured on each skip connection in the plurality of skip connections. The device comprises: a to-be-processed image acquisition module configured to acquire a to-be-processed image; and a target image obtaining module configured to input the to-be-processed image into the image super-resolution model to obtain a target image corresponding to the to-be-processed image output by the image super-resolution model, wherein the resolution of the target image is higher than that of the to-be-processed image.

[0006] In a third aspect, an electronic device is provided by the embodiments of the present application, which comprises a memory and a processor, the memory is coupled to the processor, and the memory stores instructions. When the instructions are executed by the processor, the processor executes the above method.

[0007] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, which stores program codes. The program codes can be invoked by a processor to execute the above method.

[0008] The image super-resolution method, device, electronic device and storage medium provided by the embodiments of the present application can deploy an image super-resolution model on an electronic device, design the image super-resolution model to adopt a UNet structure, and configure a spatial attention module on each of a plurality of skip connections in the UNet structure. Since the skip connections are responsible for transmitting texture details in an image, the image super-resolution model can pay more attention to the details of the image region by configuring the spatial attention module on the skip connections. Based on this, the image super-resolution effect can be more stable, and the generated target image can be more realistic and natural. BRIEF DESCRIPTION OF DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0010] Figure 1 A flowchart of an image super-resolution method provided by an embodiment of the present application is shown;

[0011] Figure 2 A first schematic diagram of a UNet structure provided by an embodiment of the present application is shown;

[0012] Figure 3 A second schematic diagram of a UNet structure provided by an embodiment of the present application is shown;

[0013] Figure 4 A structure schematic diagram of an image super-resolution model provided by an embodiment of the present application is shown;

[0014] Figure 5 A flowchart of an image super-resolution method provided by an embodiment of the present application is shown;

[0015] Figure 6 A loss calculation schematic diagram of an image super-resolution model provided by an embodiment of the present application is shown;

[0016] Figure 7 A structure schematic diagram of an image super-resolution model provided by an embodiment of the present application is shown; Figure 5 A flowchart of step S220 of the image super-resolution method is shown;

[0017] Figure 8 A flowchart of an image super-resolution method provided by an embodiment of the present application is shown.

[0018] Figure 9 A flowchart of an image super-resolution method provided by an embodiment of the present application is shown.

[0019] Figure 10 A flowchart of an image super-resolution method provided by an embodiment of the present application is shown. Figure 9 A flowchart of step S430 of the image super-resolution method is shown.

[0020] Figure 11 A flowchart of an image super-resolution method provided by an embodiment of the present application is shown.

[0021] Figure 12 A block diagram of an image super-resolution device provided by an embodiment of the present application is shown.

[0022] Figure 13 A block diagram of an electronic device for executing an image super-resolution method according to an embodiment of the present application is shown.

[0023] Figure 14 A storage unit for storing or carrying program code for implementing an image super-resolution method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application.

[0025] Among them, in the photographing of electronic devices (such as smart phones), improving the clarity of images (such as portraits) has always been a key issue, and one of the hotspots is the improvement of face clarity. With the wide application of artificial intelligence (AI) deep learning technology, convolutional neural network (CNN), especially generative adversarial network (GAN) technology, has made great achievements in image restoration and face repair (face super-resolution FaceSR) field, and has shown strong face repair and detail restoration ability.

[0026] Among them, in the photographing of electronic devices (such as smart phones), blind face super-resolution (blind faceSR) which is easy to deploy and controllable in effect is a single image face super-resolution, and all face information only comes from the camera ISP output image of the electronic device.

[0027] One common problem with super-resolution imaging of blind faces is the lack of realism in the reconstruction. Severely distorted face images exhibit significant issues such as smearing or fake textures. The main reason lies in the low-resolution output image, especially the severely distorted low-resolution image, which loses a large amount of effective detail information (texture information, contour lines, etc.). Models trained with GANs can supplement texture based on residual textures (texture enhancement), but when texture information is completely lost, it is easy to learn pseudo-textures or regular textures (artifacts), as well as textureless (smeared) or noisy and dirty objects, which are misidentified as textures and enhanced (artifacts).

[0028] Another issue with blind face super-resolution is the effect. In non-severely blurred (distorted) face scenes, the input face still has (residual parts) its real and effective texture details. It is necessary to use CNN models to appropriately enhance or generate to fill in the lost details. In practical applications, there is still the problem of over-enhancing or generating details in local areas of the face, resulting in the appearance of fake textures and dirt in the face area.

[0029] The reason for this is that a key feature of CNNs is the sharing of convolutional kernel parameters, which significantly reduces the number of parameters in the entire network, thereby improving computational efficiency. However, at the same time, weight sharing also limits the network's ability to recognize different regions of the face in low-level pixel-level super-resolution reconstruction tasks, making it unable to spatially distinguish between different regions (corresponding to facial features, skin, hair, etc.).

[0030] To address the aforementioned problems, the inventors, through long-term research, discovered and proposed the image super-resolution method, apparatus, electronic device, and storage medium provided in the embodiments of this application. By proposing an image super-resolution model based on the UNet structure and adding a spatial attention mechanism to the global skip connections of the UNet structure, the image super-resolution effect can be made more stable, and the generated target image more realistic and natural. The specific image super-resolution method will be described in detail in the subsequent embodiments.

[0031] Please see Figure 1 , Figure 1 A flowchart illustrating an embodiment of the image super-resolution method provided in this application is shown. This method proposes an image super-resolution model based on a UNet structure, adding a spatial attention mechanism to the global skip connections of the UNet structure. This results in a more stable image super-resolution effect and a more realistic and natural generated target image. In a specific embodiment, this image super-resolution method is applied to, for example... Figure 12 The image super-resolution device 200 and the electronic device 100 equipped with the image super-resolution device 200 are shown. Figure 13). The following will take an electronic device as an example to illustrate the specific process of the embodiment. It should be understood that the electronic device to which the embodiment is applied can include a smart phone, a tablet computer, a wearable electronic device, etc., which is not limited here. The following will be described in detail with reference to the flowchart shown in Figure 1 The image super-resolution method can specifically include the following steps:

[0032] Step S110: Obtain a to-be-processed image.

[0033] The inventors have found that a network suitable for deployment on an electronic device (such as a smart phone) is implemented using a semantic segmentation network UNet structure. Based on this, the image super-resolution model deployed on the electronic device of the embodiment can also use the UNet network structure. Specifically, the image super-resolution model deployed on the electronic device is limited by hardware resources, and the model size is limited. Therefore, the image super-resolution model usually uses a more efficient UNet network structure. Therefore, the embodiment selects to improve the UNet network structure to obtain more robust effect improvement on the image and avoid the problem of local area contamination.

[0034] First, the inventors tried to assemble a model structure composed of an encoder and a decoder in the UNet structure, which is equivalent to removing the skip connection of the UNet structure, similar to the network structure of an autoencoder, as shown in Figure 2 Under the training framework of GAN, the image super-resolution (such as face super-resolution) effect presented mainly shows that: 1, the eye contour is distorted; 2, the original beard shape is lost, and a regular point beard is generated; 3, the skin texture is lost, and a regular repeated texture is generated. Then, under the same training conditions (training data, training parameters, loss, etc.), after restoring the original skip connection on the UNet network, as shown in Figure 3 The image super-resolution (such as face super-resolution) effect processed is obviously improved, that is, the texture details on the face can be preserved and enhanced through the skip connection. However, the ordinary UNet structure still causes regular texture in the local area of the image, abnormal enhancement of hair, and local skin texture abnormalities (pseudo-texture).

[0035] Comparison revealed that image texture detail information (high-frequency components) is transmitted through skip connections between the encoder and decoder. Simultaneously, high-frequency classification is enhanced or generated between the encoder and decoder of UNet. However, due to parameter sharing in CNN convolution operations, the network's ability to handle spatial differentiation is weakened, leading to unstable local processing results. The network lacks the ability to perform differentiated pixel processing across regions, resulting in misidentification of image regions and incorrect enhancement of layout areas (e.g., forehead shadows are misidentified as hair, generating extra hair strands).

[0036] Among them, such as Figure 4 As shown, to address the problem of misidentification of image regions leading to incorrect enhancement of layout regions, the inventors conceived of adding a "spatial attention mechanism" to the network, enabling the image super-resolution model to possess superior spatial region recognition and processing capabilities. Furthermore, considering that when using the UNet structure for image super-resolution, the skip connections across the encoder and decoder are responsible for transmitting image texture details, and that current UNet issues often result in mis-enhanced local details or pseudo-texture problems, this embodiment places the spatial attention module within the skip connections. This structure allows the image super-resolution model to more effectively focus on the detailed representation of image regions: for example, in the eye region, this embodiment can consider the detailed differentiation of the iris and eyeball regions, the morphological representation of different hair regions such as hair, eyebrows, and beard, and the differentiated representation of different skin regions, including the forehead, cheeks, nose, and around the mouth.

[0037] Based on the above description, this embodiment deploys an image super-resolution model in an electronic device. The image super-resolution model adopts a UNet structure, which includes multiple skip connections (multiple skip connections between the encoder and decoder). Each of the multiple skip connections is configured with a spatial attention module.

[0038] The UNet structure is a typical encoding-decoding structure. The encoder part uses a pooling layer to gradually down-sample, and the decoder uses deconvolution to gradually up-sample. High-frequency information (details of the image) and low-frequency information (outlines of the image) in the original input image are gradually recovered. As a result, a low-resolution feature map is finally mapped to a pixel-level segmentation result map. Specifically, the left side of the UNet structure is an encoder responsible for feature extraction. Optionally, the encoder can be composed of four down-sampling modules, each of which can be composed of two 3*3 convolution kernels for convolution and a 2*2 pooling kernel for maximum pooling. The right side of the UNet structure is a decoder for restoring the original resolution. Optionally, the decoder can be composed of four up-sampling modules, each of which is composed of a fusion operation of the feature map generated by up-sampling and the left feature map, a 3*3 convolution kernel for convolution operation, and a generated feature map. Based on the above structure, deep features and shallow features can be fused to allow a low-resolution image containing high-level abstract features to become a high-resolution image while retaining high-level abstract features, and then perform a fusion operation with a low-level surface feature high-resolution image on the left.

[0039] Meanwhile, in order to further make up for the information lost in the down-sampling of the encoding stage, the UNet structure adds multiple skip-connects between the encoder and the decoder, which can enable the UNet structure to fuse the feature maps of the corresponding positions of the encoder in the channel during each level of up-sampling. Through the fusion of low-level features and high-level features, the UNet structure can retain more high-resolution detail information contained in the high-level feature map, thereby improving the image segmentation accuracy.

[0040] The spatial attention module is a commonly used module in deep learning, which can help the model learn important regions or features in images or videos. Optionally, the spatial attention module can be composed of a convolution layer and a gate mechanism (Gate). The convolution layer is used to extract the features of the input data, and the gate mechanism calculates the importance of each pixel according to these features. The gate mechanism can be a sigmoid function, a softmax function, or other custom functions, which are not limited here. After calculating the importance of each pixel, the spatial attention module can perform a weighted average on the input data to obtain important regions or features. This weighted average can be achieved through a convolution layer or other means, which are not limited here. When embedded in an image super-resolution model, the spatial attention module can help the image super-resolution model automatically focus on important regions or features, improving the performance and generalization ability of the image super-resolution model.

[0041] In this embodiment, the to-be-processed image can be acquired. Optionally, the to-be-processed image can be acquired by a camera of the electronic device, can be acquired from a local album of the electronic device, can be acquired from a server in communication with the electronic device, or can be acquired from another electronic device in communication with the electronic device, and the like, which is not limited herein.

[0042] Optionally, the number of to-be-processed images can be one or more, which is not limited herein.

[0043] Step S120: inputting the to-be-processed image into the image super-resolution model to obtain a target image corresponding to the to-be-processed image output by the image super-resolution model, wherein the resolution of the target image is higher than that of the to-be-processed image.

[0044] In this embodiment, when the to-be-processed image is acquired, the to-be-processed image can be input into the image super-resolution model to obtain a target image corresponding to the to-be-processed image output by the image super-resolution model. It can be understood that the resolution of the target image processed by the image super-resolution model is higher than that of the to-be-processed image.

[0045] As an implementable manner, when the to-be-processed image is acquired, the to-be-processed image can be directly input into the image super-resolution model to obtain a target image corresponding to the to-be-processed image output by the image super-resolution model.

[0046] As another implementable manner, when the to-be-processed image is acquired, the to-be-processed image can be stored in a target storage area of the electronic device, and then the to-be-processed image can be read from the target storage area by the image super-resolution model, and a target image corresponding to the to-be-processed image output by the image super-resolution model can be acquired.

[0047] As an implementable manner, if the number of to-be-processed images is one, when one to-be-processed image is acquired, the one to-be-processed image can be input into the image super-resolution model to obtain one target image corresponding to the one to-be-processed image output by the image super-resolution model.

[0048] As another implementable manner, if the number of to-be-processed images is multiple, when multiple to-be-processed images are acquired, the multiple to-be-processed images can be sequentially input into the image super-resolution model to obtain target images corresponding to the multiple to-be-processed images respectively output by the image super-resolution model.

[0049] In some embodiments, after obtaining the to-be-processed image, the to-be-processed image can be preprocessed (such as denoising, filtering, etc.), a preprocessed to-be-processed image is obtained, the preprocessed to-be-processed image is input into the image super-resolution model, and a target image corresponding to the to-be-processed image output by the image super-resolution model is obtained.

[0050] The image super-resolution method provided by an embodiment of the present application deploys an image super-resolution model on an electronic device, designs the image super-resolution model to adopt a UNet structure, the UNet structure includes a plurality of skip connections, and each skip connection in the plurality of skip connections is configured with a spatial attention module. Since the skip connection is responsible for transmitting texture details in an image, the image super-resolution model can pay more attention to the details of the image region by configuring the spatial attention module on the skip connection. Based on this, a to-be-processed image is obtained, the to-be-processed image is input into the image super-resolution model, and a target image corresponding to the to-be-processed image output by the image super-resolution model is obtained, which can make the image super-resolution effect more stable and the generated target image more realistic and natural.

[0051] Please refer to Figure 5 , Figure 5 A flowchart of an image super-resolution method provided by an embodiment of the present application is shown. The method is applied to an electronic device, the electronic device is deployed with an image super-resolution model, the image super-resolution model adopts a UNet structure, the UNet structure includes a plurality of skip connections, and each skip connection in the plurality of skip connections is configured with a spatial attention module. The following will be described in detail with respect to the flowchart shown in Figure 5 The image super-resolution method can specifically include the following steps:

[0052] Step S210: Obtain a training data set, wherein the training data set includes a plurality of first images and second images corresponding to the plurality of first images respectively, the resolution of the second image is higher than that of the first image having a corresponding relationship, and the second image is divided into a plurality of categories by a segmentation mask.

[0053] In the embodiment, a training data set can be obtained. The training data set can include a plurality of first images and second images corresponding to the plurality of first images respectively, the resolution of the second image is higher than that of the first image having a corresponding relationship, and the second image is divided into a plurality of categories by a segmentation mask.

[0054] As an example, it is assumed that the training data set includes 3 first images (first image 1, first image 2 and first image 3) and 3 second images (second image 1, second image 2 and second image 3), wherein the first image 1 has a corresponding relationship with the second image 1, the first image 2 has a corresponding relationship with the second image 2, and the third image 3 has a corresponding relationship with the second image 3. Then, the resolution of the second image 1 is higher than that of the first image 1, the resolution of the second image 2 is higher than that of the first image 2, and the resolution of the second image 3 is higher than that of the first image 3. In addition, the second image 1, the second image 2 and the second image 3 are all divided into multiple categories by a segmentation mask.

[0055] In the image segmentation, the mask refers to an image binary mask used to mark and separate the region of interest. In image segmentation, it is often necessary to cut the original image (such as the second image in the embodiment) into different parts for further analysis and processing, and the mask is a mask used to depict the outline of the target region. When performing image segmentation, manual annotation or electronic device automatic recognition method is often used to generate the mask of the target region. These masks can be binary images or grayscale images, which can be represented by black and white colors (or grayscale values). In a binary mask, the target region is marked as white, and the background is black. In a grayscale image, the pixel points of the target region are usually set to a higher grayscale value, and the background is a lower grayscale value. Through the generation and application of the mask, the target part and the background region in the original image (such as the second image in the embodiment) can be separated out, so as to facilitate subsequent analysis, recognition, positioning and other operations.

[0056] As a way, if the second image is a face image, the forehead category, chin category, head category and the like can be divided from the second image by the mask, which is not limited here.

[0057] As another way, if the second image is a landscape image, the sky category, sea category, sand category and the like can be divided from the second image by the mask, which is not limited here.

[0058] As another way, if the second image is a building image, the eave category, wall category, column category and the like can be divided from the second image by the mask, which is not limited here.

[0059] In some embodiments, the training data set can be collected by the camera of the electronic device, can be obtained from the local album of the electronic device, can be obtained from the server in communication with the electronic device, or can be obtained from other electronic devices in communication with the electronic device, and the like, which is not limited here.

[0060] As one feasible approach, when an electronic device acquires an image, it can determine the image's resolution. If the image has a high resolution (high-resolution image), it can be designated as the second image, and the resolution of the image can be reduced to obtain a low-resolution image, which is then designated as the corresponding first image. Conversely, if the image has a low resolution (low-resolution image), it can be designated as the first image, and the resolution of the image can be increased to obtain a high-resolution image, which is then designated as the corresponding second image.

[0061] Step S220: Train the image super-resolution model based on the training dataset, and guide the attention map output by the spatial attention module through the second image to obtain the trained image super-resolution model.

[0062] like Figure 6 As shown in this embodiment, with a training dataset available, the image super-resolution model can be trained based on this dataset. The attention map output by the spatial attention module is then guided by a second image to obtain the trained image super-resolution model. Based on this, by adding attention map guidance, the image super-resolution task becomes more stable, resulting in a more realistic and natural image restoration effect.

[0063] In some implementations, if the second image is a face image, a facial feature region segmentation map can be obtained by dividing the second image into multiple categories using a segmentation mask. Based on this, during the training of the image super-resolution model using the training dataset, the facial feature region segmentation map can be used to guide the attention map output by the spatial attention module to obtain the trained image super-resolution model.

[0064] In some implementations, when a training dataset is available, multiple first images can be used as input parameters of the image super-resolution model, and the second images corresponding to each of the multiple first images can be used as output parameters of the image super-resolution model. The image super-resolution model can be trained using a training strategy equivalent to GAN. During the training process, the attention map output by the spatial attention module can be guided by the second images to obtain the trained image super-resolution model.

[0065] Wherein, since each pixel value in the attention map output by the spatial attention module represents the enhancement weight (attention degree) of the feature map at its corresponding position. At the same time, since it is impossible to measure the corresponding values of each region in the attention map with label values (artificially specified), the label values of the segmentation map cannot be directly used to supervise the learning of the attention map. Based on this, in the present embodiment, since the second image is divided into multiple categories by the segmentation mask, during the training of the image super-resolution model based on the training data set, the attention map output by the spatial attention module can be guided by the second image to obtain the trained image super-resolution model. In this way, on the one hand, the inter-class distance can be widened, and on the other hand, the weight difference of different positions (same region, different positions, such as forehead and cheek of skin class) within the class can be preserved.

[0066] In some embodiments, training the image super-resolution model based on the training data set and guiding the attention map output by the spatial attention module by the second image to obtain the trained image super-resolution model can include: training the image super-resolution model based on the training data set, and guiding the attention map output by the spatial attention module by the second image to obtain the trained image super-resolution model when the image super-resolution model meets a preset training condition.

[0067] Wherein, when the image super-resolution model meets the preset training condition, it can be considered that the image super-resolution model has reached the later stage of the training process, and the attention map loss can be added for supervision in the later stage of the training process. In the early stage of the training process, the parameter update of the spatial attention module on the UNet main body and the skip connection is performed by end-to-end supervised learning, so that the attention map values of different regions fluctuate within a reasonable range of the belonging category.

[0068] As an implementable way, when the number of training times reaches a number threshold, it can be determined that the image super-resolution model meets the preset training condition. Optionally, the number of training times of the image super-resolution model trained by the training data set can be counted, and the training number is compared with the number threshold. When it is determined that the training number reaches the number threshold, it can be considered that the image super-resolution model has reached the later stage of the training process, and then it can be determined that the image super-resolution model meets the preset training condition.

[0069] As another implementable manner, the image super-resolution model is determined to satisfy the preset training condition when the training duration reaches a duration threshold. Optionally, the training duration of the image super-resolution model trained by the training data set is counted, and the training duration is compared with the duration threshold. When it is determined that the training duration reaches the duration threshold, it is considered that the image super-resolution model has reached the later stage of the training process, and then it is determined that the image super-resolution model satisfies the preset training condition.

[0070] As another implementable manner, the image super-resolution model is determined to satisfy the preset training condition when the number of remaining training data in the training data set is less than a number threshold. Optionally, the number of remaining training data of the image super-resolution model trained by the training data set is counted, and the number of remaining training data is compared with the number threshold. When it is determined that the number of remaining training data is less than the number threshold, it is considered that the image super-resolution model has reached the later stage of the training process, and then it is determined that the image super-resolution model satisfies the preset training condition.

[0071] As another implementable manner, the image super-resolution model is determined to satisfy the preset training condition when the confidence of the output result of the image super-resolution model reaches a confidence threshold. Optionally, the confidence of the output result of the image super-resolution model in the process of training the image super-resolution model by the training data set is detected, and the confidence is compared with the confidence threshold. When it is determined that the confidence reaches the confidence threshold, it is considered that the image super-resolution model has reached the later stage of the training process, and then it is determined that the image super-resolution model satisfies the preset training condition.

[0072] Of course, in the present embodiment, whether the image super-resolution model satisfies the preset training condition can also be determined by the above-mentioned manners together, or whether the image super-resolution model satisfies the preset training condition can also be determined by other more manners, which will not be described here.

[0073] Please refer to Figure 7 , Figure 7 The image super-resolution method of the present application is shown Figure 5 The flowchart of step S220 of the image super-resolution method is shown. The following will be described in detail with respect to the flow shown in Figure 7 The method can specifically include the following steps:

[0074] Step S221: determining that the attention map output by the spatial attention module corresponds to a numerical set in the same category region of the segmentation mask, and determining a variance value in the numerical set.

[0075] Wherein, the attention map can be classified by using the segmentation mask: the numerical set in the same category region of the attention map corresponding to the segmentation mask is determined, the variance statistic in the set is constrained, and the variance value participates in the loss loss calculation (attention map loss) to drive the variance value in a single region to be lower than a certain set value. In this way, on the one hand, the inter-class distance can be widened, and on the other hand, the weight difference of different positions in the same class (the same region, different positions, such as the forehead and cheek of the skin class) can be preserved.

[0076] Correspondingly, in the embodiment, the numerical set in the same category region of the attention map output by the spatial attention module corresponding to the segmentation mask can be determined, and the variance value in the numerical set can be determined. For example, the numerical set in the forehead region of the attention map output by the spatial attention module corresponding to the segmentation mask can be determined, and the variance value in the numerical set can be determined; the numerical set in the cheek region of the attention map output by the spatial attention module corresponding to the segmentation mask can be determined, and the variance value in the numerical set can be determined, and the like, which are not limited herein.

[0077] Step S222: loss calculation is performed on the attention map based on the variance value, so that the variance value in the same category region is lower than the variance threshold.

[0078] In the embodiment, in the case where the variance value in the numerical set in the same category region is obtained, loss calculation can be performed on the attention map based on the variance value, so that the variance value in the same category region (a single category region) is lower than the variance threshold.

[0079] Step S230: obtaining an image to be processed.

[0080] Step S240: inputting the image to be processed into the image super-resolution model to obtain a target image corresponding to the image to be processed output by the image super-resolution model, wherein the resolution of the target image is higher than that of the image to be processed.

[0081] Wherein, the specific description of steps S230-S240 can be referred to steps S110-S120, which will not be repeated here.

[0082] The image super-resolution method provided by the embodiment of the present application can improve the resolution of the image to be processed, and the image super-resolution method provided by the embodiment of the present application can improve the resolution of the image to be processed. Figure 1In the image super-resolution method shown, the embodiment further obtains a training data set, where the training data set includes a plurality of first images and a plurality of second images corresponding to the plurality of first images respectively, the resolution of the second image is higher than that of the corresponding first image, the second image is divided into a plurality of categories by a segmentation mask, the image super-resolution model is trained based on the training data set, and the attention map output by the spatial attention module is guided by the second image, thereby obtaining the trained image super-resolution model, so as to increase the guidance of the segmentation mask to the attention map, which can, on the one hand, pull apart the distance between different categories, and on the other hand, can retain the weight difference of different positions in the same category, thereby improving the super-resolution effect of the subsequently trained image super-resolution model on different categories.

[0083] Please refer to Figure 8 , Figure 8 A flowchart of an image super-resolution method provided by an embodiment of the present application is shown. The method is applied to an electronic device, the electronic device includes a camera, and the electronic device is deployed with an image super-resolution model, the image super-resolution model adopts a UNet structure, the UNet structure includes a plurality of skip connections, and each skip connection in the plurality of skip connections is configured with a spatial attention module. In the following, the flow shown will be described in detail, and the image super-resolution method can specifically include the following steps: Figure 8

[0084] Step S310: obtaining a first to-be-confirmed image collected by the camera, and obtaining the resolution of the first to-be-confirmed image.

[0085] In the embodiment, the electronic device includes a camera. Optionally, the camera can include a front camera, can include a rear camera, can include a rotating camera, can include a telescopic camera, etc., which is not limited here.

[0086] In some embodiments, the camera can be used for image collection, the image collected by the camera is obtained as the first to-be-confirmed image, and the resolution corresponding to the first to-be-confirmed image is obtained. Optionally, in the case where the first to-be-confirmed image collected by the camera is obtained, the attribute information of the first to-be-confirmed image can be obtained, and the resolution of the first to-be-confirmed image can be obtained based on the attribute information of the first to-be-confirmed image.

[0087] Step S320: if the resolution of the first to-be-confirmed image is lower than a resolution threshold, the first to-be-confirmed image is determined as the to-be-processed image.

[0088] ​In some embodiments, the electronic device can be pre-configured and store a resolution threshold used as a basis for judging the resolution of an image. It can be understood that, in the case that the resolution of an image is lower than the resolution threshold, the image can be considered as a low-resolution image, and the image super-resolution model can be used to perform image super-resolution on the image to improve the resolution of the image; in the case that the resolution of an image is equal to or higher than the resolution threshold, the image can be considered as a high-resolution image, and the image super-resolution processing can not be performed on the image.

[0089] In the present embodiment, in the case that the resolution of the first to-be-confirmed image is obtained, the resolution of the first to-be-confirmed image can be compared with the resolution threshold to determine whether the resolution of the first to-be-confirmed image is lower than the resolution threshold. Optionally, if the resolution of the first to-be-confirmed image is lower than the resolution threshold, it can be determined that the image super-resolution processing needs to be performed on the first to-be-confirmed image, and the first to-be-confirmed image can be determined as a to-be-processed image; if the resolution of the first to-be-confirmed image is equal to or higher than the resolution threshold, it can be determined that the image super-resolution processing does not need to be performed on the first to-be-confirmed image, and the first to-be-confirmed image can not be determined as a to-be-processed image.

[0090] Step S330: inputting the to-be-processed image into the image super-resolution model to obtain a target image corresponding to the to-be-processed image output by the image super-resolution model, wherein the resolution of the target image is higher than the resolution of the to-be-processed image.

[0091] The specific description of step S330 can be referred to step S120, which will not be repeated here.

[0092] Compared with the image super-resolution method shown in Figure 1 The present embodiment further obtains a first to-be-confirmed image collected by the camera of the electronic device, and obtains the resolution of the first to-be-confirmed image. If the resolution of the first to-be-confirmed image is lower than the resolution threshold, the first to-be-confirmed image is determined as a to-be-processed image, so that the image super-resolution processing can be performed on the low-resolution image collected by the camera of the electronic device, and the effect of the collected image can be improved.

[0093] Please refer to Figure 9 , Figure 9 A flowchart of an image super-resolution method provided by an embodiment of the present application is shown. The method is applied to an electronic device, the electronic device includes a camera, and the electronic device is deployed with an image super-resolution model, the image super-resolution model adopts a UNet structure, the UNet structure includes a plurality of skip connections, and each skip connection in the plurality of skip connections is configured with a spatial attention module. In the following, the image super-resolution method will be described in detail. Figure 9The flowchart shown is described in detail, and the image super-resolution method can specifically include the following steps:

[0094] Step S410: Obtain a second to-be-confirmed image collected by the camera, and obtain image content included in the second to-be-confirmed image.

[0095] In this embodiment, the electronic device includes a camera. Optionally, the camera can include a front camera, can include a rear camera, can include a rotating camera, can include a telescopic camera, and the like, which is not limited here.

[0096] In some embodiments, image collection can be performed by the camera, the image collected by the camera is obtained as a second to-be-confirmed image, and the image content included in the second to-be-confirmed image is obtained. Optionally, in the case where the second to-be-confirmed image collected by the camera is obtained, image recognition can be performed on the to-be-confirmed image to obtain the image content included in the second to-be-confirmed image.

[0097] Step S420: If the image content includes a face, the second to-be-confirmed image is determined as the to-be-processed image.

[0098] In some embodiments, in the case where the image content included in the second to-be-confirmed image is obtained, it can be determined whether the image content includes a face. If it is determined that the image content includes a face, the second to-be-confirmed image can be determined as a to-be-processed image, so that the subsequent image super-resolution model is used to process the second to-be-confirmed image, so that the face super-resolution effect is more stable, the generated face is more real and natural, and the side effect problem such as pseudo-texture is resisted. If it is determined that the image content does not include a face, the second to-be-confirmed image can not be determined as a to-be-processed image.

[0099] Step S430: input the to-be-processed image into the image super-resolution model to obtain a target image corresponding to the to-be-processed image output by the image super-resolution model, wherein the resolution of the target image is higher than the resolution of the to-be-processed image.

[0100] For specific description of step S430, please refer to step S120, which will not be repeated here.

[0101] Please refer to Figure 10 , Figure 10 The image super-resolution method of the present application is shown Figure 9 The flowchart of step S430 of the image super-resolution method shown. In the following Figure 10 The flowchart shown is described in detail, and the method can specifically include the following steps:

[0102] Step S431: cropping a to-be-processed sub-image from the to-be-processed image based on the face in the to-be-processed image, wherein the to-be-processed sub-image comprises the face.

[0103] In some embodiments, if it is determined that the image content comprises a face, the second to-be-confirmed image can be determined as a to-be-processed image, and a to-be-processed sub-image comprising the face can be cropped from the to-be-processed image based on the face in the to-be-processed image. Optionally, the size of the to-be-processed sub-image is smaller than the size of the to-be-processed image, and the to-be-processed sub-image can comprise only the face or can comprise the face and a part of other images, which is not limited herein.

[0104] As an implementable manner, if it is determined that the image content comprises a face, the second to-be-confirmed image can be determined as a to-be-processed image, and the to-be-processed image can be cropped with the center of the face in the to-be-processed image as a center point until the face is just included in the cropped to-be-processed image, and the to-be-processed sub-image comprising the face just included in the cropped to-be-processed image can be determined.

[0105] Step S432: inputting the to-be-processed sub-image into the image super-resolution model to obtain a target sub-image corresponding to the to-be-processed sub-image output by the image super-resolution model.

[0106] In some embodiments, after the to-be-processed sub-image is obtained, the to-be-processed sub-image can be input into the image super-resolution model to obtain a target sub-image corresponding to the to-be-processed sub-image output by the image super-resolution model. It can be understood that, since the size of the to-be-processed sub-image is smaller than the size of the to-be-processed image, the data processing amount of the image super-resolution model can be effectively reduced and the processing speed of the image super-resolution can be improved by performing the super-resolution processing on the to-be-processed sub-image through the image super-resolution model compared with performing the super-resolution processing on the to-be-processed image through the image super-resolution model.

[0107] Step S333: obtaining the target image based on the target sub-image and the to-be-processed image.

[0108] In some embodiments, after the target sub-image is obtained, the target image can be obtained based on the target sub-image and the to-be-processed image.

[0109] As an implementable manner, after the target sub-image is obtained, the target image can be obtained by performing fusion processing on the target sub-image and the to-be-processed image.

[0110] As another implementable manner, after the target sub-image is obtained, the target image can be obtained by replacing the image in the corresponding region of the to-be-processed image with the target sub-image.

[0111] The image super-resolution method provided in an embodiment of the present application can improve the effect of the captured image compared with the image super-resolution method shown in Figure 1 The embodiment further acquires a second to-be-confirmed image captured by the camera of the electronic device, acquires image content included in the second to-be-confirmed image, and determines the second to-be-confirmed image as a to-be-processed image if the image content includes a face. Thus, the face captured by the camera of the electronic device can be subjected to super-resolution processing, and the effect of the captured image can be improved.

[0112] Please refer to Figure 11 , Figure 11 A flowchart of an image super-resolution method provided in an embodiment of the present application is shown. The method is applied to an electronic device, and the electronic device is deployed with an image super-resolution model. The image super-resolution model adopts a UNet structure, and the UNet structure includes multiple skip connections. Each skip connection is configured with a spatial attention module. The following will be described in detail with reference to the flowchart shown in Figure 11 The image super-resolution method can specifically include the following steps:

[0113] Step S510: Acquire a to-be-processed video, and extract a to-be-processed image with a resolution lower than a resolution threshold from the to-be-processed video.

[0114] In the embodiment, a to-be-processed video can be acquired, and a to-be-processed image with a resolution lower than a resolution threshold can be extracted from the to-be-processed video. Optionally, the to-be-processed video can include a monitoring video, a vehicle-mounted video, a video recorded by the camera of the electronic device, etc., which is not limited herein.

[0115] In some embodiments, after the to-be-processed video is acquired, multiple image frames included in the to-be-processed video can be acquired, and resolutions corresponding to the multiple image frames respectively can be acquired. Then, the resolutions corresponding to the multiple image frames respectively are compared with the resolution threshold to determine whether the resolutions corresponding to the multiple image frames respectively are lower than the resolution threshold, to obtain a comparison result. Finally, according to the comparison result, an image frame with a resolution lower than the resolution threshold among the multiple image frames is determined as the to-be-processed image.

[0116] Step S520: Input the to-be-processed image into the image super-resolution model, and obtain a target image corresponding to the to-be-processed image output by the image super-resolution model, wherein the resolution of the target image is higher than the resolution of the to-be-processed image.

[0117] The specific description of step S520 can be referred to step S120, which will not be repeated here.

[0118] Step S530: Perform fusion processing on the target image and the to-be-processed video, and obtain a target video.

[0119] In some embodiments, in the case of obtaining the target image, the test question fuses the target image and the to-be-processed video to obtain a target video. Based on this, the low-resolution image frames in the to-be-processed video can be image super-resolution to improve the resolution, so that the image frames in the finally obtained target video are all high-resolution image frames, which can improve the effect of the obtained video.

[0120] As an implementable way, in the case of obtaining the target image, the target image can replace the image frame in the to-be-processed video as the to-be-processed image to obtain the target video.

[0121] Compared with the image super-resolution method shown in FIG. 1, the image super-resolution method provided by an embodiment of the present application further includes obtaining a to-be-processed video, extracting a to-be-processed image with a resolution lower than a resolution threshold from the to-be-processed video, performing super-resolution processing on the to-be-processed image to obtain a target image, and fusing the target image and the to-be-processed video to obtain a target video. Thus, the low-resolution image frames in the video can be super-resolution processed to improve the overall effect of the video. Figure 1 Compared with the image super-resolution method shown in FIG. 1, the image super-resolution method provided by an embodiment of the present application further includes obtaining a to-be-processed video, extracting a to-be-processed image with a resolution lower than a resolution threshold from the to-be-processed video, performing super-resolution processing on the to-be-processed image to obtain a target image, and fusing the target image and the to-be-processed video to obtain a target video. Thus, the low-resolution image frames in the video can be super-resolution processed to improve the overall effect of the video.

[0122] Please refer to Figure 12 , Figure 12 A block diagram of an image super-resolution device provided by an embodiment of the present application is shown. The image super-resolution device 200 is applied to an electronic device, the electronic device is deployed with an image super-resolution model, the image super-resolution model adopts a UNet structure, the UNet structure includes a plurality of skip connections, and each skip connection in the plurality of skip connections is configured with a spatial attention module. In the following, the block diagram shown in FIG. 2 will be described, the image super-resolution device 200 includes a to-be-processed image obtaining module 210 and a target image obtaining module 220, wherein: Figure 12

[0123] The to-be-processed image obtaining module 210 is configured to obtain a to-be-processed image.

[0124] Further, the electronic device includes a camera, and the to-be-processed image obtaining module 210 includes a first to-be-confirmed image obtaining sub-module and a first to-be-processed image obtaining sub-module, wherein:

[0125] The first to-be-confirmed image obtaining sub-module is configured to obtain a first to-be-confirmed image collected by the camera and obtain a resolution of the first to-be-confirmed image.

[0126] The first to-be-processed image obtaining sub-module is configured to determine the first to-be-confirmed image as the to-be-processed image if the resolution of the first to-be-confirmed image is lower than a resolution threshold.

[0127] ​Further, the electronic device comprises a camera, and the to-be-processed image acquisition module 210 comprises a second to-be-confirmed image acquisition submodule and a second to-be-processed image acquisition submodule, wherein:

[0128] The second to-be-confirmed image acquisition submodule is configured to acquire a second to-be-confirmed image collected by the camera and acquire image content included in the second to-be-confirmed image.

[0129] The second to-be-processed image acquisition submodule is configured to determine the second to-be-confirmed image as the to-be-processed image if the image content includes a face.

[0130] Further, the to-be-processed image acquisition module 210 comprises a third to-be-processed image acquisition submodule, wherein:

[0131] The third to-be-processed image acquisition submodule is configured to acquire a to-be-processed video and extract a to-be-processed image with a resolution lower than a resolution threshold from the to-be-processed video.

[0132] The target image acquisition module 220 is configured to input the to-be-processed image into the image super-resolution model and acquire a target image corresponding to the to-be-processed image output by the image super-resolution model, wherein the resolution of the target image is higher than the resolution of the to-be-processed image.

[0133] Further, the target image acquisition module 220 comprises a to-be-processed sub-image acquisition submodule, a target sub-image acquisition submodule, and a target image acquisition submodule, wherein:

[0134] The to-be-processed sub-image acquisition submodule is configured to cut a to-be-processed sub-image from the to-be-processed image based on a face in the to-be-processed image, wherein the to-be-processed sub-image includes the face.

[0135] The target sub-image acquisition submodule is configured to input the to-be-processed sub-image into the image super-resolution model and acquire a target sub-image corresponding to the to-be-processed sub-image output by the image super-resolution model.

[0136] The target image acquisition submodule is configured to acquire the target image based on the target sub-image and the to-be-processed image.

[0137] Further, the image super-resolution device 200 further comprises a training data set acquisition module and a model training module, wherein:

[0138] The training data set acquisition module is configured to acquire a training data set, wherein the training data set comprises a plurality of first images and second images corresponding to the plurality of first images respectively, the resolution of the second image is higher than that of the first image with a corresponding relationship, and the second image is divided into a plurality of categories by a segmentation mask.

[0139] The model training module is configured to train the image super-resolution model based on the training data set, and guide the attention map output by the spatial attention module through the second image to obtain the trained image super-resolution model.

[0140] Further, the model training module comprises a variance value determination submodule and a loss calculation submodule, wherein:

[0141] The variance value determination submodule is configured to determine a numerical set in a same category region of the attention map output by the spatial attention module corresponding to the segmentation mask, and determine a variance value in the numerical set.

[0142] The loss calculation submodule is configured to perform loss calculation on the attention map based on the variance value, so that the variance value in the same category region is lower than a variance threshold.

[0143] Further, the model training module comprises a model training submodule, wherein:

[0144] The model training submodule is configured to train the image super-resolution model based on the training data set, and guide the attention map output by the spatial attention module through the second image when the image super-resolution model meets a preset training condition to obtain the trained image super-resolution model.

[0145] Further, the model training submodule comprises a first determination unit, a second determination unit, a third determination unit and a fourth determination unit, wherein:

[0146] The first determination unit is configured to determine that the image super-resolution model meets the preset training condition when the number of training times reaches a number threshold.

[0147] The second determination unit is configured to determine that the image super-resolution model meets the preset training condition when the training duration reaches a duration threshold.

[0148] The third determination unit is configured to determine that the image super-resolution model meets the preset training condition when the number of remaining training data in the training data set is less than a number threshold.

[0149] The fourth determination unit is configured to determine that the image super-resolution model satisfies the preset training condition when the confidence of the output result of the image super-resolution model reaches a confidence threshold.

[0150] Further, the image super-resolution device 200 further includes a target video obtaining module.

[0151] The target video obtaining module is configured to perform fusion processing on the target image and the to-be-processed video to obtain a target video.

[0152] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and module can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0153] In several embodiments provided in the present application, the coupling between the modules can be electrical, mechanical or other forms of coupling.

[0154] In addition, each functional module in each embodiment of the present application can be integrated in one processing module, or each module can exist physically independently, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module.

[0155] Please refer to Figure 13 which shows a structural block diagram of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 can be a smart phone, a tablet computer, an electronic book or the like, which can run an application program. The electronic device 100 in the present application can include one or more of the following components: a processor 110, a memory 120 and one or more application programs, wherein the one or more application programs can be stored in the memory 120 and configured to be executed by the one or more processors 110, and the one or more programs are configured to perform the method as described in the foregoing method embodiments.

[0156] The processor 110 can include one or more processing cores. The processor 110 connects various parts within the entire electronic device 100 by various interfaces and lines, performs various functions of the electronic device 100 and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Alternatively, the processor 110 can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), a programmable logic array (PLA). The processor 110 can integrate a combination of one or several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes an operating system, a user interface, and an application program, etc.; the GPU is responsible for rendering and drawing of content to be displayed; and the modem is used for processing wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 110, but can be implemented by a separate communication chip.

[0157] The memory 120 can include a random access memory (RAM) and can also include a read-only memory (ROM). The memory 120 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for implementing functions (such as touch functions, sound playing functions, image playing functions, etc.), instructions for implementing various method embodiments described below, etc. The data storage area can also store data created by the electronic device 100 in use (such as a phone book, audio and video data, chat record data, etc.).

[0158] Please refer to Figure 14 which shows a structural block diagram of a computer readable storage medium provided by the embodiments of the present application. The computer readable medium 300 stores program codes therein, and the program codes can be called and executed by a processor to perform the methods described in the above method embodiments.

[0159] The computer-readable storage medium 300 can be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk or a ROM. Optionally, the computer-readable storage medium 300 comprises a non-transitory computer-readable medium. The computer-readable storage medium 300 has storage space for program code 310 for carrying out any of the method steps of the above-described methods. The program code can be read from or written to one or more computer program products. The program code 310 can be compressed, for example, in a suitable form.

[0160] To sum up, the image super-resolution method and device, electronic device and storage medium provided by the embodiments of the present application deploy an image super-resolution model on an electronic device, and design the image super-resolution model to adopt a UNet structure, the UNet structure comprising a plurality of skip connections, and each of the plurality of skip connections is configured with a spatial attention module. Since the skip connections are responsible for transmitting texture details in an image, the image super-resolution model can pay more attention to the details of the image region by configuring the spatial attention module on the skip connections. Based on this, a to-be-processed image is obtained, the to-be-processed image is input into the image super-resolution model, and a target image corresponding to the to-be-processed image output by the image super-resolution model is obtained, so that the image super-resolution effect is more stable, and the generated target image is more realistic and natural.

[0161] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not drive the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image super-resolution method, characterized in that, The application is applied to an electronic device which is deployed with an image super-resolution model adopting a semantic segmentation network UNet structure, wherein a plurality of skip connections are included in the UNet structure, and a spatial attention module is configured on each of the plurality of skip connections, and the method comprises: obtaining a training data set, wherein the training data set comprises a plurality of first images and second images corresponding to the plurality of first images respectively, the resolution of the second image is higher than that of the first image with a corresponding relationship, and the second image is divided into a plurality of categories by a segmentation mask; training the image super-resolution model based on the training data set, and guiding the attention map output by the spatial attention module through the second image to obtain the trained image super-resolution model; wherein the guiding the attention map output by the spatial attention module through the second image comprises: determining a numerical set in the same category region of the segmentation mask corresponding to the attention map output by the spatial attention module, and determining a variance value in the numerical set; based on the variance value, the variance value in the same category region is lower than the variance threshold; obtaining a to-be-processed image; inputting the to-be-processed image into the image super-resolution model to obtain a target image corresponding to the to-be-processed image output by the image super-resolution model, wherein the resolution of the target image is higher than that of the to-be-processed image.

2. The method of claim 1, wherein, The training of the image super-resolution model based on the training data set and the guiding of the attention map output by the spatial attention module through the second image to obtain the trained image super-resolution model, comprising: training the image super-resolution model based on the training data set, and guiding the attention map output by the spatial attention module through the second image to obtain the trained image super-resolution model.

3. The method of claim 2, wherein, wherein: in the case that the number of training times reaches a number threshold, it is determined that the image super-resolution model meets the preset training condition; and / or in the case that the training duration reaches a duration threshold, it is determined that the image super-resolution model meets the preset training condition; and / or in the case that the number of remaining training data in the training data set is less than a number threshold, it is determined that the image super-resolution model meets the preset training condition; and / or in the case that the credibility of the output result of the image super-resolution model reaches a credibility threshold, it is determined that the image super-resolution model meets the preset training condition.

4. The method according to any one of claims 1 to 3, characterized in that, The electronic device comprises a camera, and the obtaining of the to-be-processed image comprises: obtaining a first to-be-confirmed image collected by the camera, and obtaining the resolution of the first to-be-confirmed image; if the resolution of the first to-be-confirmed image is lower than a resolution threshold, the first to-be-confirmed image is determined as the to-be-processed image.

5. The method according to any one of claims 1 to 3, characterized in that, The electronic device comprises a camera, and the obtaining of the to-be-processed image comprises: acquire a second to-be-verified image collected by the camera, and acquire image content included in the second to-be-verified image; if the image content includes a human face, determine the second to-be-verified image as the to-be-processed image.

6. The method of claim 5, wherein, The inputting the to-be-processed image into the image super-resolution model to obtain a target image corresponding to the to-be-processed image output by the image super-resolution model comprises: intercepting a to-be-processed sub-image from the to-be-processed image based on a human face in the to-be-processed image, wherein the to-be-processed sub-image includes the human face; inputting the to-be-processed sub-image into the image super-resolution model to obtain a target sub-image corresponding to the to-be-processed sub-image output by the image super-resolution model; obtaining the target image based on the target sub-image and the to-be-processed image.

7. The method according to any one of claims 1 to 3, characterized in that, The acquiring a to-be-processed image comprises: acquiring a to-be-processed video and extracting a to-be-processed image with a resolution lower than a resolution threshold from the to-be-processed video; after the inputting the to-be-processed image into the image super-resolution model to obtain a target image corresponding to the to-be-processed image output by the image super-resolution model, the method further comprises: performing fusion processing on the target image and the to-be-processed video to obtain a target video.

8. An image super-resolution device, characterized by, The application is applied to an electronic device, the electronic device is deployed with an image super-resolution model, the image super-resolution model adopts a semantic segmentation network UNet structure, the UNet structure includes a plurality of skip connections, each skip connection in the plurality of skip connections is configured with a spatial attention module, and the device comprises: a training data set acquisition module configured to acquire a training data set, wherein the training data set includes a plurality of first images and second images corresponding to the plurality of first images respectively, the resolution of the second image is higher than the resolution of the first image with a corresponding relationship, and the second image is divided into a plurality of categories by a segmentation mask; a model training module configured to train the image super-resolution model based on the training data set, determine a numerical set in a same category region of a segmentation mask output by the spatial attention module of the image super-resolution model, determine a variance value in the numerical set, perform loss calculation on the attention map based on the variance value, make the variance value in the same category region lower than a variance threshold, and obtain the trained image super-resolution model; a to-be-processed image acquisition module configured to acquire a to-be-processed image; a target image obtaining module configured to input the to-be-processed image into the image super-resolution model to obtain a target image corresponding to the to-be-processed image output by the image super-resolution model, wherein the resolution of the target image is higher than the resolution of the to-be-processed image.

9. An electronic device, comprising: The memory stores instructions, and when the instructions are executed by the processor, the processor executes the method of any one of claims 1-7.

10. A computer readable storage medium, characterized in that, The computer readable storage medium stores program codes, and the program codes can be called and executed by the processor to execute the method of any one of claims 1-7.

Images