Image super-division method and device, electronic equipment and storage medium

Abstract

The invention discloses an image super-division method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence. The method is applied to the electronic equipment, the electronic equipment is provided with an image super-division model, the image super-division model adopts a UNet structure, the UNet structure comprises a plurality of jump connections, and each jump connection in the plurality of jump connections is provided with a space attention module. The method comprises the steps that a to-be-processed image is acquired, the to-be-processed image is input into an image super-division model, a target image which is output by the image super-division model and corresponds to the to-be-processed image is acquired, and the resolution of the target image is higher than that of the to-be-processed image. According to the image super-division model based on the UNet structure, the spatial attention mechanism is added on the global jump connection of the UNet structure, so that the image super-division effect is more stable, and the generated target image is more real and natural.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to an image super-resolution method, device, electronic device and storage medium. Background Art

[0002] Improving image clarity has always been a key issue in taking photos with electronic devices (such as smartphones). Image super-resolution refers to reconstructing low-resolution input images, that is, images with small size, relatively few pixels and poor visual effects, into clear images with relatively more pixels and good visual effects. However, current image super-resolution technology has the problem of incorrect reconstruction of details, resulting in unnatural images. Summary of the invention

[0003] In view of the above problems, the present application proposes an image super-resolution method, device, electronic device and storage medium to solve the above problems.

[0004] In a first aspect, an embodiment of the present application provides an image super-resolution method, which is applied to an electronic device, wherein the electronic device is deployed with an image super-resolution model, wherein the image super-resolution model adopts a semantic segmentation network UNet structure, wherein the UNet structure includes multiple jump connections, and each of the multiple jump connections is configured with a spatial attention module. The method includes: obtaining an image to be processed; inputting the image to be processed into the image super-resolution model, and obtaining a target image corresponding to the image to be processed output by the image super-resolution model, wherein the resolution of the target image is higher than the resolution of the image to be processed.

[0005] In a second aspect, an embodiment of the present application provides an image super-resolution device, which is applied to an electronic device, wherein the electronic device is deployed with an image super-resolution model, and the image super-resolution model adopts a semantic segmentation network UNet structure, wherein the UNet structure includes multiple jump connections, and each of the multiple jump connections is configured with a spatial attention module. The device includes: a to-be-processed image acquisition module, which is used to acquire the to-be-processed image; and a target image acquisition module, which is used to input the to-be-processed image into the image super-resolution model to obtain a target image corresponding to the to-be-processed image output by the image super-resolution model, wherein the resolution of the target image is higher than the resolution of the to-be-processed image.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor executes the above method.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored, and the program code can be called by a processor to execute the above method.

[0008] The image super-resolution method, device, electronic device and storage medium provided in the embodiments of the present application are designed by deploying an image super-resolution model on an electronic device, and the image super-resolution model is designed to adopt a UNet structure, wherein the UNet structure includes multiple jump connections, and each of the multiple jump connections is configured with a spatial attention module, wherein, since the jump connection is responsible for transmitting the texture details in the image, the image super-resolution model can more effectively focus on the image area details by configuring the spatial attention module on the jump connection. Based on this, an image to be processed is obtained, and the image to be processed is input into the image super-resolution model, and a target image corresponding to the image to be processed output by the image super-resolution model is obtained, which can make the image super-resolution effect more stable and the generated target image more realistic and natural. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 A schematic diagram of a process flow of an image super-resolution method provided by an embodiment of the present application is shown;

[0011] Figure 2 A first schematic diagram of the UNet structure provided in an embodiment of the present application is shown;

[0012] Figure 3 A second schematic diagram of the UNet structure provided in an embodiment of the present application is shown;

[0013] Figure 4 A schematic diagram of the structure of an image super-resolution model provided in an embodiment of the present application is shown;

[0014] Figure 5 A schematic diagram of a process flow of an image super-resolution method provided by an embodiment of the present application is shown;

[0015] Figure 6 A schematic diagram of loss calculation of the image super-resolution model provided in an embodiment of the present application is shown;

[0016] Figure 7 This application shows Figure 5 The schematic diagram of the process of step S220 of the image super-resolution method shown;

[0017] Figure 8 A schematic diagram of a process flow of an image super-resolution method provided by an embodiment of the present application is shown;

[0018] Fig. 9 A schematic diagram of a process flow of an image super-resolution method provided by an embodiment of the present application is shown;

[0019] Fig.10 This application shows Fig. 9 The flowchart of step S430 of the image super-resolution method shown in FIG.

[0020] Fig.11 A schematic diagram of a process flow of an image super-resolution method provided by an embodiment of the present application is shown;

[0021] Fig.12 A module block diagram of an image super-resolution device provided by an embodiment of the present application is shown;

[0022] Fig.13 A block diagram of an electronic device for executing an image super-resolution method according to an embodiment of the present application is shown;

[0023] Fig.14 A storage unit for storing or carrying program codes for implementing the image super-resolution method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0025] Among them, in the photography of electronic devices (such as smartphones), improving the clarity of images (such as portraits) has always been a key topic, among which one of the hot topics is the improvement of face clarity. With the widespread application of artificial intelligence (AI) deep learning technology, convolutional neural network (CNN), especially generative adversarial network (GAN) technology, has shined in the field of image restoration and face restoration (Face Super Resolution FaceSR), showing powerful face restoration and detail restoration capabilities.

[0026] Among them, in the photo taking of electronic devices (such as smartphones), blind face super-resolution (blind faceSR), which is easy to deploy and has controllable effects, is a single-image face super-resolution, and all face information only comes from the camera ISP output image of the electronic device.

[0027] One of the common problems of blind face super-resolution is that the restoration effect is not realistic enough. The restoration effect of severely distorted face images has serious smearing or pseudo-texture problems. The main reason is that the low-resolution output image, especially the severely distorted low-resolution image, loses a lot of effective detail information (texture information, contour lines, etc.). The model trained by GAN can supplement the texture based on the residual texture (texture enhancement), but when the texture information is completely lost, it is easy to learn pseudo-texture or regular texture (artifacts) and no texture (smearing) or noise and dirt are mistakenly recognized as texture and enhanced (artifacts).

[0028] Another problem of blind face super-resolution is the effect. In the face scene without severe blur (distortion), the input face still has (remaining parts) its real and effective texture details. It is necessary to use the CNN model to perform appropriate enhancement or generation to fill in the lost details on this basis. In practical applications, there is still the problem of excessive enhancement or generation of details in local areas of the face, which leads to the appearance of pseudo-texture and dirt in the face area.

[0029] The reason is that an important feature of CNN is the sharing of convolution kernel parameters, which reduces the number of parameters of the entire network exponentially to improve computing efficiency. However, weight sharing also results in the limited ability of the network to recognize different areas of the face in low-level pixel-level super-resolution tasks, and it is unable to make corresponding distinctions in different areas (corresponding to facial features, skin, hair, etc.) in space.

[0030] In view of the above problems, the inventors have discovered after long-term research and proposed the image super-resolution method, device, electronic device and storage medium provided in the embodiments of the present application. By proposing an image super-resolution model based on the UNet structure and adding a spatial attention mechanism to the global skip connection of the UNet structure, the image super-resolution effect can be made more stable and the generated target image can be more realistic and natural. The specific image super-resolution method will be described in detail in the subsequent embodiments.

[0031] See also Figure 1 , Figure 1 The flowchart of the image super-resolution method provided by an embodiment of the present application is shown. The method is used to make the image super-resolution effect more stable and the generated target image more realistic and natural by proposing an image super-resolution model based on the UNet structure and adding a spatial attention mechanism to the global skip connection of the UNet structure. In a specific embodiment, the image super-resolution method is applied to Fig.12 The image super-resolution device 200 and the electronic device 100 equipped with the image super-resolution device 200 are shown in FIG. Fig.13). The following will take an electronic device as an example to illustrate the specific process of this embodiment. Of course, it can be understood that the electronic device used in this embodiment may include a smart phone, a tablet computer, a wearable electronic device, etc., which is not limited here. Figure 1 The process shown is described in detail, and the method is applied to an electronic device, which is deployed with an image super-resolution model, and the image super-resolution model adopts a UNet structure, and the UNet structure includes multiple jump connections, and each of the multiple jump connections is configured with a spatial attention module. The image super-resolution method can specifically include the following steps:

[0032] Step S110: Acquire the image to be processed.

[0033] Among them, the inventors have found through research that the network suitable for deployment on electronic devices (such as smart phones) is implemented using the semantic segmentation network UNet structure. Based on this, the image super-resolution model deployed on electronic devices in this embodiment can also adopt the UNet network structure. Specifically, the image super-resolution model deployed on the electronic device side is limited by hardware resources, and its model volume is limited. Therefore, the image super-resolution model usually adopts a more efficient UNet network structure. Therefore, this embodiment chooses to improve the UNet network structure, so as to obtain a more robust effect improvement on the image and avoid the problem of local area dirt.

[0034] First, the inventors tried to assemble a model structure consisting of an encoder and a decoder in the UNet structure, which is equivalent to removing the skip-connection of the UNet structure, similar to the network structure of the autoencoder, such as Figure 2 As shown in the figure, under the GAN training framework, the image super-resolution (such as face super-resolution) effect is mainly manifested as follows: 1. The eye contour is distorted; 2. The original beard shape is lost, and a regular dot-shaped beard is generated; 3. The skin texture is lost, and a regular repetitive texture is generated. After that, compared with the same training conditions (training data, training parameters, loss, etc.), after restoring the original jump connection on the UNet network, Figure 3 As shown in the figure, the image super-resolution (such as face super-resolution) effect is significantly improved, that is, it can be determined that the texture details on the face can be retained and enhanced through skip connections. However, the ordinary UNet structure still causes regular textures to appear in local areas of the image, hair is abnormally enhanced, and there are local skin texture abnormalities (pseudo textures).

[0035] By comparison, it is found that the image texture detail information (high-frequency components) is transmitted through the jump connection between the encoder and decoder. At the same time, high-frequency classification is enhanced or generated between the encoder and decoder of UNet, but due to the parameter sharing of CNN convolution operation, the network's own ability to process spatial differentiation is weakened, resulting in unstable local processing effects. The network cannot perform pixel differentiation processing in different regions, so there will be misrecognition of image regions, resulting in incorrect enhancement of layout regions (for example, forehead shadows are misrecognized as hair, and extra hair strands are generated).

[0036] Among them, Figure 4 As shown, in order to deal with the problem of misidentification of image areas and incorrect enhancement of layout areas, the inventors thought of adding a "spatial attention mechanism" to the network to enable the image super-resolution model to have better spatial partition recognition and processing capabilities. At the same time, considering that when the Unet structure is used to implement image super-resolution, the jump connection between the encoder and the decoder is responsible for transmitting image texture details, and the current Unet is prone to effect problems also due to the misenhancement of local details or pseudo-texture problems, therefore, this embodiment places the spatial attention module in the jump connection. This structure can enable the image super-resolution model to pay more attention to the detailed performance of the image area: for example, in the eye area, this embodiment can consider the detailed differentiation performance of the eyeball and the eyeball area, consider the morphological performance of different hair areas of hair, eyebrows, and beards, and consider the differentiated performance of different skin areas, including the forehead, cheeks, nose, mouth, etc.

[0037] Based on the above description, this embodiment deploys an image super-resolution model in an electronic device. The image super-resolution model adopts a UNet structure. The UNet structure includes multiple jump connections (multiple jump connections between the encoder and the decoder), and each of the multiple jump connections is configured with a spatial attention module.

[0038] Among them, the UNet structure is a typical encoding-decoding structure. The encoder part uses the pooling layer to downsample step by step, and the decoder uses deconvolution to upsample step by step. The high-frequency information (image details) and low-frequency information (image contours) in the original input image will be gradually restored. As a result, the low-resolution feature map will eventually be mapped to a pixel-level segmentation result map. Specifically, the left side of the UNet structure is the encoder, which is responsible for feature extraction. Optionally, the encoder can be composed of four downsampling modules, each of which can be convolved by two 3*3 convolution kernels and a 2*2 pooling kernel for maximum pooling. The right side of the UNet structure is the decoder, which restores the original resolution. Optionally, the decoder can be composed of four upsampling modules, each of which fuses the feature map generated by upsampling with the feature map on the left and convolves with two 3*3 convolution kernels to generate a feature map. Based on the above structure, the fusion of deep features and shallow features can be achieved, so that the low-resolution image containing high-level abstract features can be turned into high-resolution while retaining the high-level abstract features, and then fused with the high-resolution image of the low-level surface features on the left.

[0039] At the same time, in order to further compensate for the information lost in the downsampling in the encoding stage, the UNet structure adds multiple skip-connects between the encoder and the decoder, which enables the UNet structure to fuse the feature maps of the corresponding positions of the encoder on the channel during each level of upsampling. By fusing the underlying features with the high-level features, the UNet structure can retain more high-resolution detail information contained in the high-level feature maps, thereby improving the image segmentation accuracy.

[0040] Among them, the spatial attention module is a commonly used module in deep learning, which can help the model learn important areas or features in images or videos. Optionally, the spatial attention module can be composed of a convolutional layer and a gating mechanism (Gate), the convolutional layer is used to extract the features of the input data, and the gating mechanism calculates the importance of each pixel based on these features. Among them, the gating mechanism can be a sigmoid function, a softmax function, or other custom functions, etc., which are not limited here. After calculating the importance of each pixel, the spatial attention module can perform weighted averaging on the input data to obtain important areas or features. This weighted averaging can be achieved through a convolutional layer or in other ways, which are not limited here. When the spatial attention module is embedded in the image super-resolution model, it can help the image super-resolution model automatically focus on important areas or features, thereby improving the performance and generalization ability of the image super-resolution model.

[0041] In this embodiment, the image to be processed can be obtained. Optionally, the image to be processed can be collected by a camera of an electronic device, can be obtained from a local photo album of the electronic device, can be obtained from a server communicating with the electronic device, or can be obtained from other electronic devices communicating with the electronic device, etc., which is not limited here.

[0042] Optionally, the number of images to be processed may be one or more, which is not limited here.

[0043] Step S120: inputting the image to be processed into the image super-resolution model to obtain a target image output by the image super-resolution model and corresponding to the image to be processed, wherein the resolution of the target image is higher than the resolution of the image to be processed.

[0044] In this embodiment, when the image to be processed is obtained, the image to be processed can be input into the image super-resolution model to obtain the target image corresponding to the image to be processed output by the image super-resolution model. It can be understood that the resolution of the target image processed by the image super-resolution model is higher than the resolution of the image to be processed.

[0045] As an implementable method, when an image to be processed is obtained, the image to be processed can be directly input into an image super-resolution model to obtain a target image corresponding to the image to be processed output by the image super-resolution model.

[0046] As another feasible method, when the image to be processed is obtained, the image to be processed can be stored in a target storage area of ​​the electronic device. Thereafter, the image to be processed can be read from the target storage area through the image super-resolution model, and the target image corresponding to the image to be processed output by the image super-resolution model can be obtained.

[0047] As an implementable manner, if the number of images to be processed is one, then when one image to be processed is obtained, the image to be processed can be input into the image super-resolution model to obtain a target image corresponding to the image to be processed output by the image super-resolution model.

[0048] As another feasible method, if there are multiple images to be processed, then when multiple images to be processed are obtained, the multiple images to be processed can be sequentially input into the image super-resolution model to obtain target images corresponding to the multiple images to be processed that are sequentially output by the image super-resolution model.

[0049] In some embodiments, when an image to be processed is obtained, the image to be processed can be preprocessed (such as noise reduction, filtering, etc.) to obtain a preprocessed image to be processed, and the preprocessed image to be processed is input into an image super-resolution model to obtain a target image corresponding to the image to be processed output by the image super-resolution model.

[0050] An image super-resolution method provided by an embodiment of the present application deploys an image super-resolution model on an electronic device, and designs the image super-resolution model to adopt a UNet structure, wherein the UNet structure includes multiple jump connections, and each of the multiple jump connections is configured with a spatial attention module, wherein, since the jump connection is responsible for transmitting the texture details in the image, the image super-resolution model can more effectively focus on the image area details by configuring the spatial attention module on the jump connection. Based on this, an image to be processed is obtained, and the image to be processed is input into the image super-resolution model, and a target image corresponding to the image to be processed output by the image super-resolution model is obtained, which can make the image super-resolution effect more stable and the generated target image more realistic and natural.

[0051] See also Figure 5 , Figure 5 The flowchart of the image super-resolution method provided by an embodiment of the present application is shown. The method is applied to an electronic device, which is equipped with an image super-resolution model. The image super-resolution model adopts a UNet structure, and the UNet structure includes multiple jump connections. Each of the multiple jump connections is configured with a spatial attention module. Figure 5 The process shown in FIG. 1 is described in detail, and the image super-resolution method may specifically include the following steps:

[0052] Step S210: Obtain a training data set, wherein the training data set includes multiple first images and second images corresponding to each of the multiple first images, the resolution of the second images is higher than the resolution of the first images with which there is a corresponding relationship, and the second images are divided into multiple categories through a segmentation mask.

[0053] In this embodiment, a training data set may be obtained, wherein the training data set may include a plurality of first images and second images corresponding to the plurality of first images, the resolution of the second images is higher than the resolution of the first images with which there is a corresponding relationship, and the second images are divided into multiple categories by a segmentation mask.

[0054] As an example, assume that the training data set includes 3 first images (first image 1, first image 2, and first image 3) and 3 second images (second image 1, second image 2, and second image 3), wherein the first image 1 corresponds to the second image 1, the first image 2 corresponds to the second image 2, and the third image 3 corresponds to the second image 3. Then, the resolution of the second image 1 is higher than the resolution of the first image 1, the resolution of the second image 2 is higher than the resolution of the first image 2, and the resolution of the second image 3 is higher than the resolution of the first image 3. In addition, the second image 1, the second image 2, and the second image 3 are divided into multiple categories by segmentation masks.

[0055] Among them, the mask in image segmentation refers to an image binary mask used to mark and separate the region of interest. In image segmentation, it is usually necessary to cut the original image (such as the second image in this embodiment) into different parts for further analysis and processing, and the mask is convenient for depicting the contour of the target area. When performing image segmentation, manual annotation or automatic recognition of electronic equipment is often used to generate a mask of the target area. These masks can be binary images or grayscale images, and usually black and white colors (or grayscale values) can be used to represent the background and target area. In the binary mask, the target area is marked as white, and the background is black. In the grayscale image, the pixels of the target area are usually set to a higher grayscale value, while the background is a lower grayscale value. Through the generation and application of the mask, the target part and the background area in the original image (such as the second image in this embodiment) can be separated for subsequent analysis, recognition, positioning and other operations.

[0056] As one method, if the second image is a face image, the forehead category, the chin category, the top of the head category, etc. can be divided from the second image through a mask, which is not limited here.

[0057] As another way, if the second image is a landscape image, the sky category, the sea category, the Saudi category, etc. can be divided from the second image through a mask, which is not limited here.

[0058] As another way, if the second image is a building image, the eaves category, wall category, column category, etc. can be divided from the second image through mask, which is not limited here.

[0059] In some embodiments, the training data set can be collected through a camera of the electronic device, can be obtained from a local photo album of the electronic device, can be obtained from a server that communicates with the electronic device, or can be obtained from other electronic devices that communicate with the electronic device, etc., without limitation herein.

[0060] As an implementable manner, when an electronic device acquires an image, it can determine the resolution of the image. If the resolution of the image is high (high-resolution image), the image can be determined as the second image, and the image is processed to obtain a low-resolution image by reducing the resolution, and the low-resolution image is determined as the corresponding first image. If the resolution of the image is low (low-resolution image), the image can be determined as the first image, and the image is processed to obtain a high-resolution image by increasing the resolution, and the high-resolution image is determined as the corresponding second image.

[0061] Step S220: training the image super-resolution model based on the training data set, and guiding the attention map output by the spatial attention module through the second image to obtain the trained image super-resolution model.

[0062] like Figure 6 As shown, in this embodiment, when a training data set is obtained, the image super-resolution model can be trained based on the training data set, and the attention map output by the spatial attention module is guided by the second image to obtain the trained image super-resolution model. Based on this, the image super-resolution task can be made more stable by adding the attention map guidance, and a more realistic and natural image restoration effect can be obtained.

[0063] In some embodiments, if the second image is a face image, the second image can be divided into multiple categories through a segmentation mask to obtain a segmentation map of the facial features. Based on this, in the process of training the image super-resolution model based on the training data set, the segmentation map of the facial features can be used to guide the attention map output by the spatial attention module to obtain a trained image super-resolution model.

[0064] In some embodiments, when a training data set is obtained, multiple first images can be used as input parameters of an image super-resolution model, and second images corresponding to each of the multiple first images can be used as output parameters of the image super-resolution model. The image super-resolution model is trained using a training strategy equivalent to GAN, and during the training process, the attention map output by the spatial attention module is guided by the second image to obtain a trained image super-resolution model.

[0065] Among them, since each pixel value in the attention map output by the spatial attention module represents the enhanced weight (attention level) of the feature map at its corresponding position. At the same time, since it is impossible to use label values ​​(artificially defined) to measure the numerical values ​​corresponding to each area in the attention map, the label values ​​of the segmentation map cannot be directly used to supervise the attention map learning. Based on this, in this embodiment, since the second image is divided into multiple categories through a segmentation mask, in the process of training the image super-resolution model based on the training data set, the attention map output by the spatial attention module can be guided by the second image to obtain a trained image super-resolution model. In doing so, on the one hand, the distance between classes can be widened, and on the other hand, the weight differences between different positions within the class (the same area, different positions, such as the forehead and cheeks of the skin class) can be retained.

[0066] In some embodiments, the image super-resolution model is trained based on a training data set, and the attention map output by the spatial attention module is guided by a second image to obtain a trained image super-resolution model, which may include: training the image super-resolution model based on the training data set, and when the image super-resolution model meets preset training conditions, guiding the attention map output by the spatial attention module by a second image to obtain a trained image super-resolution model.

[0067] Among them, when the image super-resolution model meets the preset training conditions, it can be considered that the image super-resolution model has reached the late stage of the training process. Attention map loss supervision can be added in the late stage of the training process. In the early stage of the training process, end-to-end supervised learning is used to update the parameters of the spatial attention module on the UNet main body and jump connection, so that the attention map values ​​of different regions fluctuate within a reasonable range of their categories.

[0068] As an implementable manner, when the number of training times reaches a times threshold, it can be determined that the image super-resolution model meets the preset training conditions. Optionally, the number of training times for training the image super-resolution model using the training data set can be counted, and the number of training times can be compared with the times threshold. When it is determined that the number of training times reaches the times threshold, it can be considered that the image super-resolution model has reached the late stage of the training process, and it can be determined that the image super-resolution model meets the preset training conditions.

[0069] As another feasible method, when the training time reaches the time threshold, it can be determined that the image super-resolution model meets the preset training conditions. Optionally, the training time of training the image super-resolution model using the training data set can be counted, and the training time can be compared with the time threshold. When it is determined that the training time reaches the time threshold, it can be considered that the image super-resolution model has reached the late stage of the training process, and it can be determined that the image super-resolution model meets the preset training conditions.

[0070] As another practicable manner, when the amount of remaining training data in the training data set is less than a quantity threshold, it is determined that the image super-resolution model meets the preset training condition. Optionally, the amount of remaining training data for training the image super-resolution model using the training data set can be counted, and the amount of the remaining training data can be compared with the quantity threshold. When it is determined that the amount of remaining training data is less than the quantity threshold, it can be considered that the image super-resolution model has reached the late stage of the training process, and it can be determined that the image super-resolution model meets the preset training condition.

[0071] As another practicable manner, when the credibility of the output result of the image super-resolution model reaches a credibility threshold, it is determined that the image super-resolution model meets the preset training condition. Optionally, during the process of training the image super-resolution model with the training data set, the credibility of the output result of the image super-resolution model can be detected, and the credibility can be compared with the credibility threshold. When it is determined that the credibility reaches the credibility threshold, it can be considered that the image super-resolution model has reached the late stage of the training process, and it can be determined that the image super-resolution model meets the preset training condition.

[0072] Of course, in this embodiment, the above-mentioned method can also be used to jointly determine whether the image super-resolution model meets the preset training conditions, or other more methods can be used to determine whether the image super-resolution model meets the preset training conditions, which will not be repeated here.

[0073] See also Figure 7 , Figure 7 This application shows Figure 5 The following is a flow chart of step S220 of the image super-resolution method. Figure 7 The process shown is described in detail, and the method may specifically include the following steps:

[0074] Step S221: Determine a set of values ​​within the same category area of ​​the segmentation mask corresponding to the attention map output by the spatial attention module, and determine the variance value within the set of values.

[0075] Among them, we can consider using segmentation mask to classify the attention-map: the value set in the same category area of ​​the attention-map corresponding to the segmentation mask, the variance statistics in the constrained set, the variance value participates in the loss calculation (attention map loss), and drives the variance value in a single area to be lower than a certain set value. Doing so can, on the one hand, increase the distance between classes, and on the other hand, preserve the weight difference of different positions within the class (the same area, different positions, such as the forehead and cheeks of the skin class).

[0076] Accordingly, in this embodiment, the attention map output by the spatial attention module corresponds to a set of values ​​in the same category area of ​​the segmentation mask, and the variance value in the set of values ​​can be determined. For example, the attention map output by the spatial attention module corresponds to a set of values ​​in the forehead area of ​​the segmentation mask, and the variance value in the set of values ​​can be determined; the attention map output by the spatial attention module corresponds to a set of values ​​in the cheek area of ​​the segmentation mask, and the variance value in the set of values ​​can be determined, and the like, which are not limited here.

[0077] Step S222: performing loss calculation on the attention map based on the variance value, so that the variance value within the same category area is lower than the variance threshold.

[0078] In this embodiment, when the variance value in the set of values ​​in the same category area is obtained, the loss calculation of the attention map can be performed based on the variance value to drive the variance value in the same category area (single category area) to be lower than the variance threshold.

[0079] Step S230: Acquire the image to be processed.

[0080] Step S240: inputting the image to be processed into the image super-resolution model to obtain a target image output by the image super-resolution model and corresponding to the image to be processed, wherein the resolution of the target image is higher than the resolution of the image to be processed.

[0081] For the detailed description of step S230 to step S240, please refer to step S110 to step S120, which will not be repeated here.

[0082] Compared with the image super-resolution method provided in an embodiment of the present application, Figure 1As shown, the image super-resolution method is obtained. In this embodiment, a training data set is obtained, wherein the training data set includes multiple first images and second images corresponding to each of the multiple first images, the resolution of the second image is higher than the resolution of the first image with which there is a corresponding relationship, and the second image is divided into multiple categories by a segmentation mask. The image super-resolution model is trained based on the training data set, and the attention map output by the spatial attention module is guided by the second image to obtain a trained image super-resolution model, thereby increasing the guidance of the attention map by the segmentation mask, which can, on the one hand, widen the distance between different classes, and on the other hand, retain the weight difference of different positions in the same class, thereby improving the super-resolution effect of the subsequently trained image super-resolution model for different categories.

[0083] See also Figure 8 , Figure 8 The flowchart of the image super-resolution method provided by an embodiment of the present application is shown. The method is applied to an electronic device, the electronic device includes a camera, and the electronic device is deployed with an image super-resolution model, the image super-resolution model adopts a UNet structure, the UNet structure includes multiple jump connections, and each of the multiple jump connections is configured with a spatial attention module. Figure 8 The process shown in FIG. 1 is described in detail, and the image super-resolution method may specifically include the following steps:

[0084] Step S310: Acquire a first image to be confirmed collected by the camera, and acquire a resolution of the first image to be confirmed.

[0085] In this embodiment, the electronic device includes a camera. Optionally, the camera may include a front camera, a rear camera, a rotating camera, a telescopic camera, etc., which is not limited here.

[0086] In some embodiments, an image may be captured by a camera, and the image captured by the camera may be obtained as the first image to be confirmed, and the resolution corresponding to the first image to be confirmed may be obtained. Optionally, when the first image to be confirmed captured by the camera is obtained, the attribute information of the first image to be confirmed may be obtained, and the resolution of the first image to be confirmed may be obtained based on the attribute information of the first image to be confirmed.

[0087] Step S320: If the resolution of the first image to be confirmed is lower than the resolution threshold, the first image to be confirmed is determined as the image to be processed.

[0088] In some embodiments, the electronic device may pre-set and store a resolution threshold, which is used as a basis for determining the resolution of an image. It is understood that, when the resolution of an image is lower than the resolution threshold, the image may be considered a low-resolution image, and the image super-resolution model may be used to perform image super-resolution on the image to improve the resolution of the image; when the resolution of an image is equal to or higher than the resolution threshold, the image may be considered a high-resolution image, and image super-resolution processing may not be performed on the image.

[0089] In this embodiment, when the resolution of the first image to be confirmed is obtained, the resolution of the first image to be confirmed can be compared with the resolution threshold to determine whether the resolution of the first image to be confirmed is lower than the resolution threshold. Optionally, if the resolution of the first image to be confirmed is lower than the resolution threshold, it can be determined that the first image to be confirmed needs to be subjected to image super-resolution processing, and the first image to be confirmed can be determined as the image to be processed; if the resolution of the first image to be confirmed is equal to or higher than the resolution threshold, it can be determined that the first image to be confirmed does not need to be subjected to image super-resolution processing, and the first image to be confirmed can be not determined as the image to be processed.

[0090] Step S330: inputting the image to be processed into the image super-resolution model to obtain a target image output by the image super-resolution model and corresponding to the image to be processed, wherein the resolution of the target image is higher than the resolution of the image to be processed.

[0091] For the detailed description of step S330, please refer to step S120, which will not be repeated here.

[0092] Compared with the image super-resolution method provided in an embodiment of the present application, Figure 1 The image super-resolution method shown in the figure, this embodiment also obtains a first image to be confirmed captured by a camera of an electronic device, and obtains the resolution of the first image to be confirmed. If the resolution of the first image to be confirmed is lower than a resolution threshold, the first image to be confirmed is determined as an image to be processed, so that super-resolution processing can be performed on the low-resolution image captured by the camera of the electronic device, which can improve the effect of the captured image.

[0093] See also Fig. 9 , Fig. 9 The flowchart of the image super-resolution method provided by an embodiment of the present application is shown. The method is applied to an electronic device, the electronic device includes a camera, and the electronic device is deployed with an image super-resolution model, the image super-resolution model adopts a UNet structure, the UNet structure includes multiple jump connections, and each of the multiple jump connections is configured with a spatial attention module. Fig. 9The process shown in FIG. 1 is described in detail, and the image super-resolution method may specifically include the following steps:

[0094] Step S410: Acquire a second image to be confirmed collected by the camera, and acquire image content included in the second image to be confirmed.

[0095] In this embodiment, the electronic device includes a camera. Optionally, the camera may include a front camera, a rear camera, a rotating camera, a telescopic camera, etc., which is not limited here.

[0096] In some embodiments, an image may be captured by a camera, and the image captured by the camera may be obtained as the second image to be confirmed, and the image content included in the second image to be confirmed may be obtained. Optionally, when the second image to be confirmed captured by the camera is obtained, image recognition may be performed on the image to be confirmed to obtain the image content included in the second image to be confirmed.

[0097] Step S420: If the image content includes a face, the second image to be confirmed is determined as the image to be processed.

[0098] In some embodiments, when the image content included in the second image to be confirmed is obtained, it can be determined whether the image content includes a face. If it is determined that the image content includes a face, the second image to be confirmed can be determined as the image to be processed, so that the second image to be confirmed can be processed by a subsequent image super-resolution model, so that the face super-resolution effect is more stable, the generated face is more real and natural, and the side effects such as pseudo-texture are resisted. If it is determined that the image content does not include a face, the second image to be confirmed may not be determined as the image to be processed.

[0099] Step S430: inputting the image to be processed into the image super-resolution model to obtain a target image output by the image super-resolution model and corresponding to the image to be processed, wherein the resolution of the target image is higher than the resolution of the image to be processed.

[0100] For the detailed description of step S430, please refer to step S120, which will not be repeated here.

[0101] See also Fig.10 , Fig.10 This application shows Fig. 9 The following is a flow chart of step S430 of the image super-resolution method. Fig.10 The process shown is described in detail, and the method may specifically include the following steps:

[0102] Step S431: extracting a sub-image to be processed from the image to be processed based on the face in the image to be processed, wherein the sub-image to be processed includes the face.

[0103] In some embodiments, if it is determined that the image content includes a face, the second image to be confirmed may be determined as the image to be processed, and based on the face in the image to be processed, a sub-image to be processed including the face may be intercepted from the image to be processed. Optionally, the size of the sub-image to be processed is smaller than the size of the image to be processed, and the sub-image to be processed may include only the face, or may include the face and a portion of other images, which is not limited here.

[0104] As an implementable method, if it is determined that the image content includes a face, the second image to be confirmed can be determined as the image to be processed, and the image to be processed can be expanded with the center of the face in the image to be processed as the center point until the expanded range happens to include the face, and the image to be processed is intercepted, and the image obtained by interception that happens to include the face is determined as the sub-image to be processed.

[0105] Step S432: input the sub-image to be processed into the image super-resolution model to obtain a target sub-image output by the image super-resolution model and corresponding to the sub-image to be processed.

[0106] In some embodiments, when a sub-image to be processed is obtained, the sub-image to be processed can be input into an image super-resolution model to obtain a target sub-image corresponding to the sub-image to be processed output by the image super-resolution model. It can be understood that since the size of the sub-image to be processed is smaller than the size of the image to be processed, super-resolution processing of the sub-image to be processed by the image super-resolution model can effectively reduce the data processing amount of the image super-resolution model and improve the processing speed of the image super-resolution compared to super-resolution processing of the image to be processed by the image super-resolution model.

[0107] Step S333: obtaining the target image based on the target sub-image and the image to be processed.

[0108] In some implementations, when a target sub-image is obtained, a target image may be obtained based on the target sub-image and the image to be processed.

[0109] As an implementable manner, when a target sub-image is obtained, the target sub-image and the image to be processed may be fused to obtain a target image.

[0110] As another feasible manner, when a target sub-image is obtained, the target sub-image may be used to replace an image of a corresponding area in the image to be processed to obtain the target image.

[0111] Compared with the image super-resolution method provided in an embodiment of the present application, Figure 1 The image super-resolution method shown in the figure, this embodiment also obtains a second image to be confirmed captured by a camera of the electronic device, and obtains the image content included in the second image to be confirmed. If the image content includes a face, the second image to be confirmed is determined as the image to be processed, so that the face captured by the camera of the electronic device can be super-resolution processed, which can improve the effect of the captured face image.

[0112] See also Fig.11 , Fig.11 The flowchart of the image super-resolution method provided by an embodiment of the present application is shown. The method is applied to an electronic device, which is equipped with an image super-resolution model. The image super-resolution model adopts a UNet structure, and the UNet structure includes multiple jump connections. Each of the multiple jump connections is configured with a spatial attention module. Fig.11 The process shown in FIG. 1 is described in detail, and the image super-resolution method may specifically include the following steps:

[0113] Step S510: obtaining a video to be processed, and extracting an image to be processed having a resolution lower than a resolution threshold from the video to be processed.

[0114] In this embodiment, a video to be processed may be obtained, and an image to be processed having a resolution lower than a resolution threshold may be extracted from the video to be processed. Optionally, the video to be processed may include a surveillance video, a vehicle-mounted video, a video recorded by a camera of an electronic device, etc., which is not limited here.

[0115] In some embodiments, when a video to be processed is obtained, multiple image frames included in the video to be processed can be obtained, and the resolutions corresponding to each of the multiple image frames can be obtained. Thereafter, the resolutions corresponding to each of the multiple image frames are compared with a resolution threshold to determine whether the resolutions corresponding to each of the multiple image frames are lower than the resolution threshold to obtain a comparison result. Finally, based on the comparison result, the image frames among the multiple image frames whose corresponding resolutions are lower than the resolution threshold are determined as images to be processed.

[0116] Step S520: inputting the image to be processed into the image super-resolution model to obtain a target image output by the image super-resolution model and corresponding to the image to be processed, wherein the resolution of the target image is higher than the resolution of the image to be processed.

[0117] For the detailed description of step S520, please refer to step S120, which will not be repeated here.

[0118] Step S530: performing fusion processing on the target image and the video to be processed to obtain a target video.

[0119] In some implementations, when the target image is obtained, the target image and the video to be processed are fused to obtain the target video. Based on this, the low-resolution image frames in the video to be processed can be super-resolved to increase the resolution, so that the image frames in the target video finally obtained are all high-resolution image frames, which can improve the effect of the obtained video.

[0120] As an implementable manner, when a target image is obtained, an image frame serving as the image to be processed in a video to be processed may be replaced by the target image to obtain the target video.

[0121] Compared with the image super-resolution method provided in an embodiment of the present application, Figure 1 The image super-resolution method shown in the figure, this embodiment also obtains a video to be processed, and extracts an image to be processed whose resolution is lower than a resolution threshold from the video to be processed, performs super-resolution processing on the image to be processed to obtain a target image, and performs fusion processing on the target image and the video to be processed to obtain a target video, thereby realizing super-resolution processing of low-resolution image frames in the video to improve the overall effect of the video.

[0122] See also Fig.12 , Fig.12 The module block diagram of an image super-resolution device provided by an embodiment of the present application is shown. The image super-resolution device 200 is applied to an electronic device, and the electronic device is equipped with an image super-resolution model. The image super-resolution model adopts a UNet structure, and the UNet structure includes multiple jump connections, and each of the multiple jump connections is configured with a spatial attention module. Fig.12 The block diagram shown in FIG. 2 is used to illustrate that the image super-resolution device 200 includes: a to-be-processed image acquisition module 210 and a target image acquisition module 220, wherein:

[0123] The to-be-processed image acquisition module 210 is used to acquire the to-be-processed image.

[0124] Further, the electronic device includes a camera, and the to-be-processed image acquisition module 210 includes: a first to-be-confirmed image acquisition submodule and a first to-be-processed image acquisition submodule, wherein:

[0125] The first image to be confirmed obtaining submodule is used to obtain the first image to be confirmed collected by the camera and obtain the resolution of the first image to be confirmed.

[0126] The first to-be-processed image acquisition submodule is configured to determine the first to-be-confirmed image as the to-be-processed image if the resolution of the first to-be-confirmed image is lower than a resolution threshold.

[0127] Furthermore, the electronic device includes a camera, and the to-be-processed image acquisition module 210 includes: a second to-be-confirmed image acquisition submodule and a second to-be-processed image acquisition submodule, wherein:

[0128] The second image to be confirmed obtaining submodule is used to obtain the second image to be confirmed collected by the camera, and obtain the image content included in the second image to be confirmed.

[0129] The second to-be-processed image acquisition submodule is configured to determine the second to-be-confirmed image as the to-be-processed image if the image content includes a face.

[0130] Furthermore, the to-be-processed image acquisition module 210 includes: a third to-be-processed image acquisition submodule, wherein:

[0131] The third to-be-processed image acquisition submodule is used to acquire a to-be-processed video and extract an to-be-processed image having a resolution lower than a resolution threshold from the to-be-processed video.

[0132] The target image acquisition module 220 is used to input the image to be processed into the image super-resolution model to obtain the target image output by the image super-resolution model and corresponding to the image to be processed, wherein the resolution of the target image is higher than the resolution of the image to be processed.

[0133] Furthermore, the target image acquisition module 220 includes: a to-be-processed sub-image acquisition sub-module, a target sub-image acquisition sub-module and a target image acquisition sub-module, wherein:

[0134] The sub-image to be processed acquisition sub-module is used to intercept the sub-image to be processed from the image to be processed based on the human face in the image to be processed, wherein the sub-image to be processed includes the human face.

[0135] The target sub-image acquisition sub-module is used to input the sub-image to be processed into the image super-resolution model to obtain the target sub-image output by the image super-resolution model and corresponding to the sub-image to be processed.

[0136] The target image acquisition submodule is used to obtain the target image based on the target sub-image and the image to be processed.

[0137] Furthermore, the image super-resolution device 200 further includes: a training data set acquisition module and a model training module, wherein:

[0138] A training data set acquisition module is used to acquire a training data set, wherein the training data set includes multiple first images and second images corresponding to each of the multiple first images, the resolution of the second images is higher than the resolution of the first images with which there is a corresponding relationship, and the second images are divided into multiple categories through a segmentation mask.

[0139] A model training module is used to train the image super-resolution model based on the training data set, and to guide the attention map output by the spatial attention module through the second image to obtain the trained image super-resolution model.

[0140] Furthermore, the model training module includes: a variance value determination submodule and a loss calculation submodule, wherein:

[0141] The variance value determination submodule is used to determine a set of values ​​within the same category area of ​​the segmentation mask corresponding to the attention map output by the spatial attention module, and to determine the variance value within the set of values.

[0142] The loss calculation submodule is used to perform loss calculation on the attention map based on the variance value so that the variance value within the same category area is lower than the variance threshold.

[0143] Furthermore, the model training module includes: a model training submodule, wherein:

[0144] The model training submodule is used to train the image super-resolution model based on the training data set, and when the image super-resolution model meets the preset training conditions, guide the attention map output by the spatial attention module through the second image to obtain the trained image super-resolution model.

[0145] Furthermore, the model training submodule includes: a first determination unit, a second determination unit, a third determination unit and a fourth determination unit, wherein:

[0146] The first determining unit is used to determine that the image super-resolution model meets the preset training condition when the number of training times reaches a threshold number of times.

[0147] The second determining unit is used to determine that the image super-resolution model meets the preset training condition when the training time reaches a time threshold.

[0148] The third determining unit is used to determine that the image super-resolution model meets the preset training condition when the amount of remaining training data in the training data set is less than the amount threshold.

[0149] The fourth determination unit is used to determine that the image super-resolution model meets the preset training condition when the credibility of the output result of the image super-resolution model reaches a credibility threshold.

[0150] Furthermore, the image super-resolution device 200 further includes: a target video acquisition module, wherein:

[0151] The target video acquisition module is used to perform fusion processing on the target image and the video to be processed to obtain the target video.

[0152] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0153] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0154] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0155] See also Fig.13 , which shows a structural block diagram of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may be an electronic device capable of running applications, such as a smart phone, a tablet computer, an e-book, etc. The electronic device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110, and the one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0156] Among them, the processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 110 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.

[0157] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing functions (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device 100 during use.

[0158] See also Fig.14 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 300 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0159] The computer readable storage medium 300 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 300 includes a non-transitory computer-readable storage medium. The computer readable storage medium 300 has storage space for program code 310 that performs any method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code 310 can be compressed, for example, in an appropriate form.

[0160] In summary, the image super-resolution method, device, electronic device and storage medium provided in the embodiments of the present application, by deploying an image super-resolution model on an electronic device, the image super-resolution model is designed to adopt a UNet structure, wherein the UNet structure includes multiple jump connections, and each of the multiple jump connections is configured with a spatial attention module, wherein, since the jump connection is responsible for transmitting the texture details in the image, the image super-resolution model can more effectively focus on the image area details by configuring the spatial attention module on the jump connection. Based on this, an image to be processed is obtained, the image to be processed is input into the image super-resolution model, and a target image corresponding to the image to be processed output by the image super-resolution model is obtained, which can make the image super-resolution effect more stable and the generated target image more realistic and natural.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image super-resolution method, characterized in that: Applied to an electronic device, the electronic device is deployed with an image super-resolution model, the image super-resolution model adopts a semantic segmentation network UNet structure, the UNet structure includes multiple jump connections, each of the multiple jump connections is configured with a spatial attention module, and the method includes: Get the image to be processed; The image to be processed is input into the image super-resolution model to obtain a target image corresponding to the image to be processed and output by the image super-resolution model, wherein the resolution of the target image is higher than the resolution of the image to be processed.

2. The method according to claim 1, characterized in that The method further comprises: Acquire a training data set, wherein the training data set includes a plurality of first images and second images corresponding to each of the plurality of first images, the resolution of the second images is higher than the resolution of the first images with which there is a corresponding relationship, and the second images are divided into a plurality of categories by a segmentation mask; The image super-resolution model is trained based on the training data set, and the attention map output by the spatial attention module is guided by the second image to obtain the trained image super-resolution model.

3. The method according to claim 2, characterized in that The guiding the attention map output by the spatial attention module through the second image includes: Determine a set of values ​​within the same category area of ​​the segmentation mask corresponding to the attention map output by the spatial attention module, and determine a variance value within the set of values; A loss is calculated for the attention map based on the variance value, so that the variance value within the same category area is lower than a variance threshold.

4. The method according to claim 2, characterized in that: The step of training the image super-resolution model based on the training data set and guiding the attention map output by the spatial attention module through the second image to obtain the trained image super-resolution model comprises: The image super-resolution model is trained based on the training data set, and when the image super-resolution model meets the preset training conditions, the attention map output by the spatial attention module is guided by the second image to obtain the trained image super-resolution model.

5. The method according to claim 4, characterized in that in: When the number of training times reaches a threshold number of times, determining that the image super-resolution model meets the preset training condition; and / or When the training time reaches the time threshold, determining that the image super-resolution model meets the preset training condition; and / or When the amount of remaining training data in the training data set is less than a quantity threshold, determining that the image super-resolution model meets the preset training condition; and / or When the credibility of the output result of the image super-resolution model reaches a credibility threshold, it is determined that the image super-resolution model meets the preset training condition.

6. The method according to any one of claims 1 to 5, characterized in that: The electronic device includes a camera, and the step of obtaining an image to be processed includes: Acquire a first image to be confirmed collected by the camera, and acquire a resolution of the first image to be confirmed; If the resolution of the first image to be confirmed is lower than the resolution threshold, the first image to be confirmed is determined as the image to be processed.

7. The method according to any one of claims 1 to 5, characterized in that: The electronic device includes a camera, and the step of acquiring an image to be processed includes: Acquire a second image to be confirmed collected by the camera, and acquire image content included in the second image to be confirmed; If the image content includes a human face, the second image to be confirmed is determined as the image to be processed.

8. The method according to claim 7, characterized in that The step of inputting the image to be processed into the image super-resolution model to obtain a target image output by the image super-resolution model and corresponding to the image to be processed includes: Based on the human face in the image to be processed, intercepting a sub-image to be processed from the image to be processed, wherein the sub-image to be processed includes the human face; Inputting the sub-image to be processed into the image super-resolution model to obtain a target sub-image output by the image super-resolution model and corresponding to the sub-image to be processed; The target image is obtained based on the target sub-image and the image to be processed.

9. The method according to any one of claims 1 to 5, characterized in that: The step of obtaining the image to be processed comprises: Acquire a video to be processed, and extract an image to be processed having a resolution lower than a resolution threshold from the video to be processed; After inputting the image to be processed into the image super-resolution model to obtain a target image output by the image super-resolution model and corresponding to the image to be processed, the method further includes: The target image and the video to be processed are fused to obtain a target video.

10. An image super-resolution device, characterized in that: Applied to an electronic device, the electronic device is deployed with an image super-resolution model, the image super-resolution model adopts a semantic segmentation network UNet structure, the UNet structure includes multiple jump connections, each of the multiple jump connections is configured with a spatial attention module, and the device includes: The module for acquiring the image to be processed is used to acquire the image to be processed; The target image acquisition module is used to input the image to be processed into the image super-resolution model to obtain the target image output by the image super-resolution model and corresponding to the image to be processed, wherein the resolution of the target image is higher than the resolution of the image to be processed.

11. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 9.

Images