Intelligent local image super-resolution processing method and system based on AI

Abstract

The invention discloses an AI-based intelligent local image super-resolution processing method and system, and the method comprises the steps: loading a to-be-processed original image, and determining a target region of the to-be-processed original image; cutting the target area, and outputting an image protruding part; inputting the highlighted part of the image into a pre-trained super-resolution model to obtain a high-resolution local image; and splicing the processed local image and the unprocessed original image, and displaying the spliced image on a preview interface. The technical scheme of the invention aims to improve the efficiency and quality of image processing and output.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an AI-based intelligent local image super-resolution processing method and system. Background Art

[0002] As the performance of mobile devices improves, applications based on image processing and computer vision are increasingly used on smart devices, especially the development of real-time image processing and target detection technology, which brings new experiences to the shooting and imaging functions of mobile devices. Traditional camera applications can usually only provide basic image previews and simple filter processing, which is difficult to meet users' needs for high-precision target detection, real-time processing and image enhancement. Summary of the invention

[0003] The main purpose of the present invention is to provide an AI-based intelligent local image super-resolution processing method and system, aiming to improve the efficiency and quality of image processing and output.

[0004] To achieve the above object, the present invention proposes the following technical solution.

[0005] First, an AI-based intelligent local image super-resolution processing method is proposed, including:

[0006] Loading the original image to be processed and determining the target area of ​​the original image to be processed;

[0007] Crop the target area and output the prominent part of the image;

[0008] Input the prominent part of the image into the pre-trained super-resolution model to obtain a high-resolution local image;

[0009] The processed local image is stitched with the unprocessed original image and displayed on the preview interface.

[0010] In a possible implementation, the super-resolution model adaptively adjusts parameters according to the type of the loaded original image to be processed.

[0011] In a possible implementation manner, the type of the original image includes a natural image, a medical image, a text or a document scan image.

[0012] In a possible implementation, the step of inputting the prominent part of the image into a pre-trained super-resolution model to obtain a high-resolution local image includes:

[0013] Global feature extraction, capturing the overall structure, shape, and color distribution of the image;

[0014] Global information encoding: After obtaining the global features, they are encoded into feature maps as a reference for local processing;

[0015] Local area extraction, automatically detecting local detail-rich areas in images;

[0016] Convolutional network detail processing,In the detected local area, detailed processing is performed through the convolutional network.

[0017] In a possible implementation manner, after the step of splicing the processed partial image with the unprocessed original image and displaying them on the preview interface, the method further includes:

[0018] Optimize the model through multiple loss functions.

[0019] In a possible implementation, the multiple loss functions include a pixel-level loss function, a perceptual loss function, an adversarial loss function, and a global-local balance loss function.

[0020] In a possible implementation manner, after the step of optimizing the model by using multiple loss functions, the method further includes:

[0021] Perform edge smoothing on the boundary between the local image and the original image.

[0022] In a second aspect, the present application proposes a system comprising: a processor and a memory, wherein the memory is coupled to the processor, the memory is used to store computer program code, the computer program code comprising computer instructions, and when the processor reads the computer instructions from the memory, the electronic device executes the AI-based intelligent local image super-resolution processing method as described in the above-mentioned first aspect or any possible implementation method of the first aspect.

[0023] The technical solution of the present invention can accurately identify the object and the corresponding area in the image by determining the target area of ​​the original image to be processed, cropping the target area, and outputting the prominent part of the image; for the non-target area, the blurring algorithm is applied to reduce background interference and make the image focus more prominent. For the target area, the image details are enhanced and the original proportion is restored through the super-resolution model, thereby further improving the visual effect and detail fidelity of the image. After blurring and super-resolution processing, the final merged image can achieve a balance between details and overall effect while ensuring visual quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0025] Figure 1 This is a flow chart of an embodiment of an AI-based intelligent local image super-resolution processing method of the present invention;

[0026] Figure 2 FIG. 4 is a flow chart of an embodiment of step S3.

[0027] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0029] Embodiment 1

[0030] The present invention proposes an AI-based intelligent local image super-resolution processing method, which can be run on a mobile device (such as an Android phone), a desktop device or a laptop computer. Figure 1 As shown, the following steps are included:

[0031] S1: Loading the original image to be processed and determining the target area of ​​the original image to be processed.

[0032] In this step, read the image from disk or data source, usually using image processing libraries such as OpenCV, PIL (Python Imaging Library) or other deep learning frameworks' native APIs (such as PyTorch, TensorFlow). Make sure the image is read correctly, including the correct color mode (such as RGB, grayscale). Convert the image to NumPy array or tensor format for subsequent operations. Next, determine the target area, that is, the part that needs to increase the resolution.

[0033] S2: Crop the target area and output the salient part of the image.

[0034] In this step, the target area needs to be cropped and adjusted to the input size required by the model. Usually, bilinear interpolation, bicubic interpolation, nearest neighbor interpolation and other methods are used to adjust the image size. For super-resolution tasks, it may be necessary to first reduce the high-resolution image to generate a low-resolution image as input, and then perform super-resolution restoration through the model. During the training data generation process, the high-resolution image can be downsampled to generate input-output pairs with different resolutions from the original image.

[0035] S3: Input the prominent part of the image into the pre-trained super-resolution model to obtain a high-resolution local image.

[0036] In this step, first, use object detection or image segmentation methods to identify and separate the prominent parts of the image that need super-resolution processing (such as people, objects, etc.). Deep learning models such as Mask R-CNN or U-Net can be used for image segmentation. Normalize the image pixel values ​​to a certain range (such as [0,1] or [-1,1]) to speed up the convergence of the model and avoid data values ​​that are too large or too small to affect the training effect. By diversifying the training data, increasing the amount of data, and improving the generalization ability of the model. Data enhancement is particularly important in image super-resolution tasks. The diversity of input images can be increased by flipping, rotating, and randomly cropping. Convert image data from array or image format to tensor for subsequent processing by the deep learning framework. In super-resolution tasks, it is usually necessary to downsample high-resolution images to generate low-resolution images as input to the model. In order to improve computational efficiency and utilize the parallel processing capabilities of the GPU, images need to be processed in batches after preprocessing. Usually, a group of images are packaged into a batch of a fixed size, and several images are concatenated into a tensor and sent to the model. The prepared low-resolution image tensor is input into the pre-trained super-resolution model to obtain a high-resolution output. It should be noted that the high-resolution result output by the model is in tensor format, which needs to be converted back to image format and denormalized before it can be displayed or saved.

[0037] In addition, after identifying and separating the salient parts that need to be processed in the image, the areas outside the salient parts can also be blurred, including background blur, object segmentation and post-processing. The main purpose is to make the focus of the image more prominent, while other parts become soft and blurred. After determining the salient parts, a mask needs to be generated. The mask is an image of the same size as the original image, marking which areas are to be kept clear and which areas are to be blurred. For example, the segmented image will generate a black and white mask image, in which the character part is white and the background part is black. The generated mask image can use the Gaussian blur function to blur the area marked as the background in the mask.

[0038] S4: splicing the processed local image with the unprocessed original image and displaying them on a preview interface.

[0039] In this step, first read the local high-resolution image and the original unprocessed image after super-resolution processing. In order to stitch them together, make sure that the processed local image has the same size as the original image. If the sizes of the two images are inconsistent, the processed local image may need to be adjusted. The high-resolution highlight part after super-resolution processing is combined with the original background image to generate the final high-resolution image. This step can be used to merge the images again through masks.

[0040] In a possible implementation, the super-resolution model adaptively adjusts parameters according to the type of the loaded original image to be processed.

[0041] In a possible implementation manner, the type of the original image includes a natural image, a medical image, a text or a document scan image.

[0042] Choosing a suitable super-resolution model depends on multiple factors, such as the type of image to be processed, the target application, and performance requirements. Image types include natural images (such as landscapes, photos of people, etc.). For this type of image, the model is usually required to enhance texture details and sharpness. You can use ESRGAN (Enhanced SRGAN), which is based on the Generative Adversarial Network (GAN) and can generate very realistic details visually and performs well in many image enhancement tasks. For medical images, the accuracy of details and noise suppression are very important. GAN models may introduce subtle artifacts, so more robust models are needed. You can use VDSR (Very Deep Super-Resolution), which is good at handling noise and maintaining image details. Documents or document scanned images require super-resolution models that can restore text and edges well. You can use ESPCN (Efficient Sub-Pixel Convolutional Neural Network), which is more suitable for real-time applications and fast, and can enhance edge details.

[0043] When the super-resolution model processes a local image, there may be inconsistencies between the local and global images. The locally processed area may appear unnatural or not match the style of the original image. This is because the super-resolution model may only over-enhance the resolution of local details, while ignoring the coordination with the overall style of the whole image.

[0044] In view of this, in a possible implementation manner, in combination with reference Figure 2 As shown, the step of inputting the prominent part of the image into the pre-trained super-resolution model to obtain a high-resolution local image includes:

[0045] S301, global feature extraction, capturing the overall structure, shape, and color distribution of the image;

[0046] S302, global information encoding, after obtaining the global features, encoding them into feature maps as a reference for local processing;

[0047] S303, local area extraction, automatically detecting local detail-rich areas in the image;

[0048] S304: Convolutional network detail processing: performing detailed processing in the detected local area through the convolutional network.

[0049] The above steps can process the global and local information of the image at the same time to obtain a more refined and natural super-resolution effect. Traditional super-resolution models usually process at the whole image level, which may not be accurate enough for local areas with rich details; while models that process local details alone may ignore the overall visual consistency. The above steps are intended to resolve the contradiction between the two, combining global structure and local detail information to ensure that the enlarged image has both clear details and global consistency. Global information usually refers to the overall structure of the image, the shape and general outline of the object, etc. This type of information is crucial to maintaining the consistency and naturalness of the image. If only local areas are super-enlarged, the global structure of the image may be destroyed, so the model needs to consider the global features of the entire image. Local details such as textures, edges, small objects, etc. are usually blurred in low-resolution images. The local super-resolution model can focus on these details and restore high-frequency information after enlargement. However, if the local super-resolution model is separated from the global context, the enhanced local area may be inconsistent with the style or content of the original image. By combining global features with local detail enhancement, the model can maintain the overall perceptual consistency of the image while carefully processing the high-frequency information of the local area. The key to this model is how to coordinate the global structure and local details so that the image has both macroscopic visual coherence and fine texture expression.

[0050] Specifically, a convolutional neural network (CNN) or visual transformer is used to extract features of the global image. At this stage, the focus is on capturing low-frequency information such as the overall structure, shape, and color distribution of the image. After obtaining the global features, they are encoded into feature maps, which will be used as references for local processing. Thus, complete global features are generated to guide subsequent local detail enhancement and ensure the consistency of the whole image. Then, local detail-rich areas in the image (such as edges, textures, small objects, etc.) are automatically detected through attention mechanisms, object detection, or semantic segmentation techniques, which usually require key enhancement. In the detected local areas, detailed processing is performed through convolutional networks. In order to enhance the details, higher-resolution convolutional layers are often used to extract local high-frequency information. According to the characteristics of the local area, the model can dynamically adjust the degree of enhancement to avoid over-enhancement or loss of details. In this way, the high-frequency details of the local area can be restored and enhanced to ensure the clarity and naturalness of the details. Furthermore, the feature maps output by the global and local branches are spliced ​​at a certain layer to form a joint feature map. This fusion process is usually completed in the convolutional layer to ensure that global features and local details are combined. In some models, local and global features are weighted and fused. By introducing weight coefficients, the model can dynamically adjust the contribution of global and local features to ensure that the processed image has both global consistency and local detail expression. For multi-scale models, global and local features are gradually fused at different scales to ensure that details and global information at different levels are well processed. Combining global and local features ensures that the enhanced image has both global consistency and natural expression of local details.

[0051] In a possible implementation manner, after the step of splicing the processed partial image with the unprocessed original image and displaying them on the preview interface, the method further includes:

[0052] Optimize the model through multiple loss functions.

[0053] In this step, multiple loss function optimization models refer to the use of multiple different loss functions to simultaneously optimize different objectives during the model training process, thereby improving the overall performance of the model. For super-resolution tasks, especially global-local joint super-resolution models, multiple loss functions can help the model maintain the consistency of the global structure while enhancing the clarity of local details when generating high-resolution images.

[0054] In a possible implementation, the multiple loss functions include a pixel-level loss function, a perceptual loss function, an adversarial loss function, and a global-local balance loss function.

[0055] Specifically, the pixel-level loss function is the L1 / L2 loss, where the L1 loss is the absolute error between the generated high-resolution image and the true high-resolution image at the pixel level, and the L2 loss is the square error. The commonly used expression is:

[0056]

[0057] Among them, x i is the generated pixel value, y i is the true pixel value and N is the total number of pixels in the image.

[0058] Ensure that the generated high-resolution image is as close to the real image as possible at the pixel level. L1 loss can reduce over-smoothing, and L2 loss is more commonly used to ensure the similarity of the overall structure.

[0059] Perceptual loss measures the high-level semantic similarity between generated images and real images by comparing their feature representations at different levels in deep neural networks (such as VGG). The perceptual loss function is expressed as:

[0060]

[0061] Among them, φl represents the features extracted in the lth layer of the pre-trained network (such as VGG), and Nl represents the number of features in this layer.

[0062] Perceptual loss focuses more on the visual quality and semantic consistency of the image, and can capture more advanced image features, not just pixel-level similarities. For example, it can help the model better restore details such as the edges and textures of objects. Perceptual loss can improve the restoration of local details, making the generated image visually closer to the real image.

[0063] Adversarial loss comes from the Generative Adversarial Network (GAN), which trains the generator and the discriminator to make the generated images more realistic. The goal of the generator is to deceive the discriminator so that it cannot distinguish between the generated images and the real images. The form of adversarial loss is usually:

[0064]

[0065] Among them, G is the generator, D is the discriminator, y is the real image, and G(x) is the generated image.

[0066] By training the generator through GAN, the generated images look more realistic and natural. Adversarial loss is particularly suitable for enhancing local details, because GAN can focus on generating high-frequency information and textures, making the image look clearer and more realistic. Adversarial loss can generate more high-frequency details, which helps to enhance local details and make the super-resolution image more visually impactful.

[0067] Edge loss is used to enhance the clarity of the edge of the image. It optimizes the details by calculating the edge difference between the generated image and the real image, usually using the Sobel operator or Canny edge detection to extract the edge:

[0068]

[0069] in, represents the edge gradient of the generated image, Represents the edge gradient of the real image.

[0070] It can improve the clarity of the edge area of ​​the image, reduce blur, and make the outline of the object more obvious. It can effectively enhance the edges of fine structures or objects in the image and is suitable for processing local details.

[0071] Texture loss is to restore the texture details of the image. By measuring the similarity between the generated image and the real image in texture, the generated local high-frequency details are optimized.

[0072] The similarity of image textures can be measured by the Gram matrix:

[0073] Texture loss = ||G(φ l (x))-G(φ l (y))|| 2

[0074] Where G represents the Gram matrix of the feature, and φl represents the feature extracted from a certain layer in the VGG network.

[0075] It can restore complex texture information in the image, such as hair, grass, wall texture, etc., making the generated image more visually rich. It can also enhance local complex details and make the super-resolution image more realistic.

[0076] In order to ensure a balance between global structure and local details, an adaptive weight mechanism is designed to dynamically adjust the weight of the loss according to the importance of the global area and local details. For example:

[0077] Total loss = α·global loss + β·local loss

[0078] Wherein, α and β are adaptive weights, α+β=1.

[0079] In this way, the model can automatically adjust the global and local optimization goals at different training stages to avoid excessive enhancement of local details or destruction of the global structure. It ensures the natural transition of global consistency and local details without over-optimization of a certain part.

[0080] In the training process of the global-local joint super-resolution model, the above-mentioned multiple loss functions can be weighted and combined to form the final total loss function. For example:

[0081] Total loss = λ1·Pixel-level loss + λ2·Perceptual loss + λ3·Adversarial loss + λ4·Edge loss + λ5·Texture loss

[0082] Among them, λ1,λ2,λ3,λ4,λ5 are weight coefficients used to balance the contribution of different losses to model optimization. During the training process, these weights can be adjusted through experiments to find the optimal combination.

[0083] In summary, by optimizing pixel-level loss and perceptual loss at the same time, the model can maintain global structure and color consistency while enhancing the detail restoration effect in local areas. Perceptual loss, adversarial loss, and edge loss can improve the visual perception quality of images, ensuring that the enlarged image is not only pixel-accurate but also has better sensory effects. Texture loss and edge loss can optimize specific areas (such as textures and edges), making the details of local areas more realistic and natural.

[0084] In a possible implementation manner, after the step of optimizing the model by using multiple loss functions, the method further includes:

[0085] Perform edge smoothing on the boundary between the local image and the original image.

[0086] In this step, possible stitching problems are dealt with, especially at the junction of the local area and the global area, and the abruptness of the transition area is reduced by edge smoothing.

[0087] Embodiment 2

[0088] The present application proposes a system, comprising: a processor and a memory, wherein the memory is coupled to the processor, the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor reads the computer instructions from the memory, the electronic device executes the AI-based intelligent local image super-resolution processing method as described in the above-mentioned first aspect or any possible implementation method of the first aspect.

[0089] In some embodiments, the computer readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or various devices including one or any combination of the above memories. The computer may be various computing devices including intelligent terminals and servers.

[0090] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0091] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0092] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0093] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An AI-based intelligent local image super-resolution processing method, characterized in that: include: Loading the original image to be processed and determining the target area of ​​the original image to be processed; Crop the target area and output the prominent part of the image; Input the prominent part of the image into the pre-trained super-resolution model to obtain a high-resolution local image; The processed local image is stitched with the unprocessed original image and displayed on the preview interface.

2. The AI-based intelligent local image super-resolution processing method according to claim 1, characterized in that: The super-resolution model adaptively adjusts parameters according to the type of the loaded original image to be processed.

3. The AI-based intelligent local image super-resolution processing method according to claim 2, characterized in that: The types of the original images include natural images, medical images, text or document scan images.

4. The AI-based intelligent local image super-resolution processing method according to any one of claims 1 to 3, characterized in that: The step of inputting the prominent part of the image into the pre-trained super-resolution model to obtain a high-resolution local image includes: Global feature extraction, capturing the overall structure, shape, and color distribution of the image; Global information encoding: After obtaining the global features, they are encoded into feature maps as a reference for local processing; Local area extraction, automatically detecting local detail-rich areas in images; Convolutional network detail processing,In the detected local area, detailed processing is performed through the convolutional network.

5. The AI-based intelligent local image super-resolution processing method according to any one of claims 1 to 3, characterized in that: After the step of splicing the processed partial image with the unprocessed original image and displaying them on the preview interface, the method further includes: Optimize the model through multiple loss functions.

6. The AI-based intelligent local image super-resolution processing method according to claim 5, characterized in that: The multiple loss functions include pixel-level loss function, perceptual loss function, adversarial loss function, and global-local balance loss function.

7. The AI-based intelligent local image super-resolution processing method according to claim 5, characterized in that: After the step of optimizing the model by multiple loss functions, the method further includes: Perform edge smoothing on the boundary between the local image and the original image.

8. A system, characterized in that: include: A processor and a memory, wherein the memory is coupled to the processor, the memory is used to store computer program code, the computer program code includes computer instructions, and when the processor reads the computer instructions from the memory, the electronic device executes the AI-based intelligent local image super-resolution processing method as described in any one of claims 1-7.

Images