Image processing method, electronic device, storage medium and product

Abstract

The application provides an image processing method, an electronic device, a storage medium and a product, relates to the technical field of image processing, and the image processing method is processed by inputting image blocks with different complexities into corresponding complexity image processing models. For simple image blocks with less edge information, a lower complexity (i.e. fewer feature dimensions) image processing model is used for processing, unnecessary performance overhead can be reduced, and the processing speed can be accelerated. For complex image blocks with more edge information, a higher complexity (i.e. more feature dimensions) image processing model is used for processing, and the image processing effect is ensured.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more particularly to an image processing method, electronic device, storage medium, and product. Background Technology

[0002] After an electronic device acquires an image, it needs to process the image to present it to the user in order to meet the user's visual requirements.

[0003] To facilitate image processing on electronic devices with limited computing resources, images are typically segmented first. A model then processes these segmented image blocks sequentially, and the processed blocks are stitched together to obtain the complete image. This method requires less memory and computing resources from the electronic device, making it suitable for resource-constrained environments such as mobile devices. However, this method still incurs unnecessary performance overhead, resulting in slower image processing speeds. Summary of the Invention

[0004] This application provides an image processing method, electronic device, storage medium, and product that can reduce unnecessary performance overhead and accelerate image processing.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] In a first aspect, an image processing method is provided for use in an electronic device, the method comprising:

[0007] An image to be processed is acquired and divided into multiple image blocks, and the position information of each image block is recorded. An edge score is determined for each image block, whereby the edge score characterizes the richness of edge information within the image block. Based on the different preset ranges within which the edge scores of each image block fall, each image block is assigned to an image processing model of different complexity corresponding to the preset range for processing, resulting in processed image blocks. The image processing models of different complexities have different numbers of feature dimensions, and the larger the value within the preset range, the more feature dimensions the image processing model corresponding to the preset range has. The processed image blocks are then stitched together according to the position information to obtain a complete processed image.

[0008] In the embodiments of this application, the edge score of each image block is calculated to measure the richness of edge information of each image block, that is, the complexity of each image block. According to the different preset ranges in which the edge scores of each image block fall, image blocks of different complexities are assigned to image processing models of different complexities. If the edge score of an image block is small, the richness of edge information in the image block is low, that is, the image block is low in complexity. In this case, the image block with low complexity is input into an image processing model with low complexity (that is, fewer feature dimensions). If the edge score of an image block is large, the richness of edge information in the image block is high, that is, the image block is high in complexity. In this case, the image block with high complexity is input into an image processing model with high complexity (that is, more feature dimensions). Compared to inputting all image patches into a model of the same complexity, this embodiment processes image patches of different complexities into image processing models of corresponding complexities. Simpler image patches with less edge information are processed using a lower-complexity model (i.e., fewer feature dimensions), reducing unnecessary performance overhead and accelerating processing. Conversely, more complex image patches with more edge information are processed using a higher-complexity model (i.e., more feature dimensions), ensuring optimal image processing results. Therefore, this embodiment reduces unnecessary performance overhead and accelerates image processing speed while maintaining optimal image processing performance.

[0009] In one possible implementation of the first aspect, determining the edge score of each of the plurality of image blocks includes: obtaining a binarized edge map of each of the plurality of image blocks; and calculating the edge score of each of the image blocks based on the binarized edge map. In this embodiment, calculating the edge score based on the binarized edge map corresponding to the image block can further accelerate the processing speed compared to directly calculating the edge score from grayscale or color image blocks.

[0010] In one possible implementation of the first aspect, obtaining the binarized edge map of each of the plurality of image blocks includes: inputting each of the plurality of image blocks into a pre-trained encoder to obtain the corresponding binarized edge map of each image block. In this embodiment, a pre-trained encoder can be used to generate the binarized edge map, further accelerating the processing speed and providing good adaptability to image blocks of different sizes.

[0011] In one possible implementation of the first aspect, the image to be processed is a noisy image, and the image processing model is a denoising model. The training process of the pre-trained encoder includes the following steps: acquiring a first clear image sample; adding noise to the first clear image sample to generate a first noisy image sample; downsampling the first clear image sample by a factor of s, where s is the downsampling factor of the encoder; performing edge detection on the downsampled first clear image sample to generate a target binarized edge map; and training the encoder using the first noisy image sample as input and the target binarized edge map as a supervision label. In this embodiment, by training the model to extract edge maps from noisy images, the accuracy of edge extraction is improved, and the noise in image blocks is avoided from affecting edge extraction.

[0012] In one possible implementation of the first aspect, calculating the edge score of each image block based on the binarized edge map of each image block includes: performing global pooling processing on the binarized edge map of each image block to obtain the edge score of each image block. This embodiment uses a global pooling method to calculate the edge score, which can automatically adapt to input image blocks of any size and reduce manual operation steps.

[0013] In a second aspect, an electronic device is provided, comprising: a memory and a processor, the memory for storing a computer program executable on the processor, the processor for performing steps of the first aspect or any of the image processing methods in the first aspect.

[0014] Thirdly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the steps of the first aspect or any of the image processing methods in the first aspect.

[0015] Fourthly, a computer program product is provided, the computer program product comprising: computer program code, which, when executed by an electronic device, causes the electronic device to perform the steps of the first aspect or any of the methods in the first aspect of the image processing method. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating an image processing method according to an embodiment of this application;

[0017] Figure 2 This is a flowchart illustrating another image processing method in an embodiment of this application;

[0018] Figure 3 This is an embodiment of the present application. Figure 2 A schematic diagram of the framework of the image processing method shown;

[0019] Figure 4 This is a schematic diagram of three image blocks with different complexities in the embodiments of this application;

[0020] Figure 5 This is a flowchart illustrating another image processing method in an embodiment of this application;

[0021] Figure 6 This is an embodiment of the present application. Figure 5 A schematic diagram of the framework of the image processing method shown;

[0022] Figure 7 This is a schematic diagram of the structure of the image processing device in the embodiments of this application;

[0023] Figure 8 This is a schematic diagram of the structure of the electronic device in the embodiments of this application. Detailed Implementation

[0024] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0025] In the description of the embodiments of this application, unless otherwise stated, " / " means "or", for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0026] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0027] First, some terms in the embodiments of this application will be explained to facilitate understanding by those skilled in the art.

[0028] 1. Image noise

[0029] Image noise is a random variation in brightness or color information within an image (not present in the object being photographed), typically manifested as electronic noise. Image denoising is the process of removing unwanted noise from an image while preserving as much useful information as possible.

[0030] 2. Edge information and edge images

[0031] Edges are points in an image where features such as brightness, color, or texture change abruptly. These changes typically represent the boundaries of different objects within the image.

[0032] Edge information refers to the geometric and structural features of significant areas such as object outlines and texture changes in an image. Its complexity is positively correlated with the difficulty of denoising. For example, areas with dense textures, sharp edges, or severe noise interference require more complex network processing during denoising.

[0033] An edge image is an image obtained by extracting the edges of the original image.

[0034] 3. s-fold bilinear downsampling

[0035] "s times" usually refers to reducing the image size to one-sth of its original width and height, thus reducing the total number of pixels to 1 / s of the original. 2 For example, if an image is 1000x1000 pixels, after a four-fold downsampling, the size will become 250x250 pixels.

[0036] Bilinear downsampling refers to a method of calculating pixel values ​​using a bilinear interpolation algorithm when downscaling an image. Its core principle is to calculate the target pixel value by weighted averaging of four adjacent pixels in the original image, thereby reducing jagged edges or distortion during image downscaling.

[0037] 4. Canny edge detection operator

[0038] The Canny edge detection operator is a classic image processing algorithm designed to efficiently and accurately detect edges in images. Its core idea is to identify true edges while suppressing noise through multi-step optimization.

[0039] The Canny edge detection operator takes a color or grayscale image as input and outputs a binary image, where white pixels represent detected edges and black pixels represent the background.

[0040] The original sharp image may contain some minor degradation factors (such as slight blurring or compression artifacts), which may be mistaken for important edge information at higher resolutions. By downsampling, these minor degradation factors are smoothed or removed, while the truly significant edges are still preserved after downsampling due to their high contrast, thus making Canny edge detection more robust.

[0041] 5. Global pooling processing

[0042] Global pooling refers to compressing the spatial dimensions of the entire edge map, converting it into a scalar representing global features (such as edge scores). Global pooling methods include:

[0043] (1) Global Average Pooling (GAP): Calculates the average value of all pixels in the edge map.

[0044]

[0045] Where E(i,j) is the edge map pixel value, and H and W are the image height and width.

[0046] (2) Global Max Pooling (GMP): Takes the maximum value of the pixel value in the edge map.

[0047] e = maxE(i,j)

[0048] The specific process of global pooling (GAP) includes: inputting an edge map (such as a continuous value matrix generated by an encoder, or a binary map using the Canny operator); if it is a continuous value edge map (such as gradient magnitude), GAP calculates the average intensity, and GMP captures the most salient edges; if it is a binary edge map (0 / 1 represents edge presence), GAP is equivalent to the edge pixel ratio (density), and GMP is always 1 (when edges exist). The output is a scalar value (edge ​​score), used to quantify the overall edge complexity of the image patch. A higher edge score indicates a greater edge density or intensity (such as dense textures, sharp contours, or noisy regions), requiring a more complex denoising model.

[0049] 6. Peak Signal-to-Noise Ratio

[0050] Peak Signal-to-Noise Ratio (PSNR) is a widely used objective metric for measuring the quality of signals such as images and videos. It quantifies the restoration effect by calculating the difference between the original signal and the distorted signal. In image denoising tasks, PSNR is used to evaluate how closely the denoised image patch resembles the original sharp image patch; a higher value indicates a better denoising effect.

[0051] The calculation process of PSNR:

[0052] (1) Calculate the mean square error

[0053] Input: The original clear image patch Iclear and the denoised image patch Idenoised (both are matrices of the same size).

[0054] formula:

[0055]

[0056] Where H and W are the height and width of the image patch. The average of the squared differences of each pixel reflects the overall error intensity.

[0057] (2) Calculate PSNR (in dB)

[0058]

[0059] Where MAX is the maximum possible value of the image pixel (e.g., MAX = 255 in an 8-bit image).

[0060] Specifically, the smaller the MSE (the closer the denoised image is to the original image), the higher the PSNR value. If the MSE is 0 (perfect restoration), the PSNR approaches infinity.

[0061] 7. Encoder

[0062] An encoder is essentially a model or module that transforms input data into a specific representation. Its goal is to generate a more efficient and abstract representation by extracting key features or compressing information. In deep learning, encoders are typically used to transform high-dimensional, complex input data (such as images, text, or audio) into low-dimensional, compact representations that can be used for subsequent tasks such as classification, generation, or reconstruction.

[0063] In convolutional neural networks (CNNs), the encoder typically consists of multiple convolutional and pooling layers. Its core function is to extract local features from the input data through convolutional operations and gradually reduce the spatial dimensionality of the data through pooling operations while preserving important information. The following are the essence and characteristics of a CNN encoder:

[0064] (1) Feature extraction

[0065] Convolutional layers extract local features (such as edges and textures) by sliding convolutional kernels across the input data. As the network depth increases, convolutional layers can extract higher-level features (such as shape and object parts).

[0066] (2) Dimensionality reduction and information compression

[0067] Pooling layers (such as max pooling or average pooling) reduce the spatial size of feature maps through downsampling operations, thereby reducing computational complexity and preventing overfitting.

[0068] The output of an encoder is typically a low-dimensional representation of the input data, retaining key information while removing redundancy.

[0069] (3) Hierarchical representation

[0070] Convolutional neural network encoders generate hierarchical feature representations through multiple layers of convolution and pooling operations. Shallow layers extract low-level features, while deeper layers extract high-level semantic features. This hierarchical representation gives convolutional neural networks powerful expressive capabilities when processing structured data such as images.

[0071] The above is a brief introduction to the terms used in the embodiments of this application, and will not be repeated below.

[0072] The technical solutions of the embodiments of this application are described below:

[0073] After an electronic device acquires an image, it needs to process the image to present it to the user in order to meet the user's visual requirements.

[0074] To facilitate image processing on electronic devices with limited computing resources, images are typically segmented first. A model then processes these segmented image blocks sequentially, and the processed blocks are stitched together to obtain the complete image. This method requires less memory and computing resources from the electronic device, making it suitable for resource-constrained environments such as mobile devices. However, this method still incurs unnecessary performance overhead, resulting in slower image processing speeds.

[0075] In view of this, embodiments of this application provide an image processing method applied to an electronic device. The method includes: acquiring an image to be processed and dividing the image to be processed into multiple image blocks, recording the position information of each image block; determining the edge score of each image block in the multiple image blocks, wherein the edge score characterizes the richness of edge information in the image block; assigning each image block to an image processing model of different complexity corresponding to the preset range for processing according to the different preset ranges in which the edge scores of each image block fall, to obtain processed image blocks, wherein the number of feature dimensions of the image processing models of different complexities is different, and the larger the value in the preset range, the more feature dimensions of the image processing model corresponding to the preset range; and stitching the processed image blocks together according to the position information to obtain a processed complete image.

[0076] In the embodiments of this application, the edge score of each image block is calculated to measure the richness of edge information of each image block, that is, the complexity of each image block. According to the different preset ranges in which the edge scores of each image block fall, each image block with different complexities is assigned to an image processing model with different complexities. If the edge score of an image block is small, the richness of edge information in the image block is low, that is, the image block is low in complexity. In this case, the image block with low complexity is input into an image processing model with low complexity (that is, fewer feature dimensions). If the edge score of an image block is large, the richness of edge information in the image block is high, that is, the image block is high in complexity. In this case, the image block with high complexity is input into an image processing model with high complexity (that is, more feature dimensions). Compared to inputting all image patches into a model of the same complexity, this embodiment processes image patches of different complexities into image processing models of corresponding complexities. Simpler image patches with less edge information are processed using a lower-complexity model (i.e., fewer feature dimensions), reducing unnecessary performance overhead and accelerating processing. Conversely, more complex image patches with more edge information are processed using a higher-complexity model (i.e., more feature dimensions), ensuring optimal image processing results. Therefore, this embodiment reduces unnecessary performance overhead and accelerates image processing speed while maintaining optimal image processing results.

[0077] The image processing method provided in this application can be applied to various electronic devices, such as SLR cameras, point-and-shoot cameras, mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc., or other devices or apparatuses capable of image processing. This application does not impose any restrictions on the specific type of electronic device.

[0078] Figure 1 This is a flowchart illustrating the first image processing method provided in this application embodiment, including the following steps:

[0079] It is worth noting that the purpose of the image processing method and image processing model in this embodiment is to obtain an image that meets the user's visual requirements.

[0080] Step S101: Obtain the image to be processed, divide the image to be processed into multiple image blocks, and record the position information of each image block.

[0081] The image to be processed can be an image captured by the camera of an electronic device. After capturing the image to be processed, the electronic device uses the image processing method in this embodiment to process the image to obtain an image that meets the user's visual requirements. The image to be processed can also be an image transmitted to the electronic device from other devices, or an image downloaded by the electronic device from the network or cloud. All of these can be processed using the image processing method in this embodiment to obtain an image that meets the user's visual requirements.

[0082] In this embodiment, to facilitate image processing on electronic devices with limited computing resources, the image to be processed is first segmented, and each image block is processed sequentially. In this way, compared to processing the entire image to be processed directly, processing each image block sequentially can significantly reduce the consumption of computing resources during image processing and reduce the performance overhead of the electronic device.

[0083] In one implementation, a uniform grid segmentation strategy is used to segment the image to be processed. For example, a fixed size w×h (e.g., 256×256 pixels) is set, and the image is divided equally by rows and columns. When the size of the image to be processed is W×H, the number of image blocks after segmentation = (W / w)×(H / h). In another implementation, an unequal segmentation strategy can also be used. For example, segmentation is performed based on the amount of edge information in each region of the image to be processed, grouping adjacent regions with similar edge information richness into the same image block. In practical applications, other image segmentation methods can also be used to segment the image to be processed, which will not be elaborated in this embodiment.

[0084] It is worth noting that the size of each image block after segmentation can be the same or different, and the shape of each image block after segmentation can be rectangular or other shapes. This embodiment does not impose any restrictions.

[0085] After segmenting the image to be processed into image blocks, the position information of each image block is recorded. This facilitates subsequent stitching of the processed image blocks based on this position information. If the shape of each segmented image block is rectangular, it is only necessary to record the position coordinates of one vertex of each image block in the image to be processed (for example, the coordinates of the top-left vertex of each image block can be recorded). In the subsequent image stitching process, the image blocks can be stitched together according to the position coordinates of the vertices to complete the image stitching. If the shape of each segmented image block is not rectangular, it is necessary to record the position coordinates of at least three vertices of each image block in the image to be processed, thereby determining the position coordinates of the image block in the image to be processed, which facilitates subsequent image stitching.

[0086] It should be noted that the image to be processed in this embodiment can be a color image or a grayscale image, and the image blocks obtained after segmentation can also be color images or grayscale images.

[0087] Step S102: Determine the edge score of each image block in the multiple image blocks.

[0088] In this embodiment, the edge score characterizes the richness of edge information in an image patch. A higher edge score indicates richer edge information in the image patch, and richer edge information in an image patch indicates higher processing complexity. Therefore, the edge score can also be used to measure the processing complexity of each image patch; a higher edge score indicates richer edge information in the image patch, and correspondingly, higher processing complexity.

[0089] In one implementation, edge scores are directly calculated for each image block. If the image block is a grayscale image, gradient operators such as Sobel can be used to directly calculate the pixel gradient magnitude, and the mean or variance of the gradient magnitudes across the entire image is used as the edge score. If the image block is a color image, the gradient magnitudes of each color channel (e.g., R, G, and B channels) of each pixel need to be calculated separately, and the weighted sum of the gradient magnitudes of each channel is taken as the gradient magnitude of that pixel. Then, the mean or variance of the gradient magnitudes across the entire image is used as the edge score.

[0090] In another implementation, instead of directly calculating the edge score for each image block, the edge score for each image block is calculated based on the binarized edge map of each image block.

[0091] In this embodiment, a binary edge map of 0 / 1 is first generated using algorithms such as Canny. 0 represents non-edge regions, i.e., relatively smooth or inconspicuous parts of the image patch, while 1 represents detected edges. These regions are typically where there are significant changes in color, brightness, or texture, such as object outlines or texture boundaries. Since the image contains less information after binarization, calculating edge scores based on the corresponding binary edge map of the image patch can further accelerate the processing speed compared to directly calculating edge scores from grayscale or color image patches.

[0092] In addition to using algorithms such as Canny to generate binary edge maps, pre-trained models can also be used to generate binary edge maps, further accelerating the processing speed and providing good adaptability to image blocks of different sizes.

[0093] This is feasible, and the pre-trained model can be a pre-trained encoder. In this embodiment, each image block from multiple image patches is input into the pre-trained encoder to obtain the corresponding binary edge map of each image patch. Compared to directly calculating edge scores from grayscale or color image patches, using a pre-trained encoder to extract the binary edge map of each image patch can further accelerate the processing speed. In practical applications, other pre-trained models can also be used to extract the binary edge map of each image patch, which will not be described in detail in this embodiment.

[0094] After obtaining the binarized edge map of each image block, the edge density of each image block can be calculated as the edge score. Edge density = total number of edge pixels / total number of pixels in the image × 100%. The higher the edge density of the image block, the more edge pixels there are in the image block, and the richer the edge information of the image block.

[0095] In addition to using edge density as the edge score, global pooling can also be performed on the binarized edge maps of each image block to obtain the edge score for each image block. In this embodiment, global pooling can automatically adapt to input image blocks of any size, while the edge density index method requires manual adjustment of the pixel counting logic according to the size of the input image block. Therefore, this embodiment uses global pooling to calculate the edge score, which can automatically adapt to input image blocks of any size and reduce manual operation steps.

[0096] Specifically, the edge scores of the binarized edge map can be calculated using the Global Average Pooling (GAP) method. The formula for global average pooling is as follows (1):

[0097]

[0098] Where e represents the edge score, H and W are the height and width of the image patch, and E(i,j) is the edge map pixel value in the i-th row and j-th column of the image patch.

[0099] In practical applications, other methods can also be used to calculate the edge score of the binarized edge map, which will not be described in detail in this embodiment.

[0100] Step S103: Based on the different preset ranges in which the edge scores of each image block fall, each image block is assigned to an image processing model of different complexity corresponding to the preset range for processing, and the processed image block is obtained.

[0101] To ensure the quality of all processed image patches, using the same high-complexity model for all patches would incur unnecessary computational overhead and prolong processing time. Therefore, this embodiment inputs image patches into image processing models of corresponding complexity based on their processing complexity. The processing complexity of an image patch is represented by its edge score. A higher edge score indicates higher processing complexity, and a more complex image processing model (i.e., more feature dimensions) is used. Conversely, a lower edge score indicates lower processing complexity, and a less complex image processing model (i.e., fewer feature dimensions) is used. Compared to using the same high-complexity model for all image patches, this reduces unnecessary computational overhead and speeds up image processing.

[0102] In this embodiment, a pre-defined correspondence is established between different preset ranges and image processing models of different complexities. The number of feature dimensions varies among image processing models of different complexities, and the larger the value in the preset range, the more feature dimensions the corresponding image processing model has. The more feature dimensions an image processing model has, the more complex the model, the greater the computational overhead for processing image blocks, and the longer the processing time.

[0103] The two preset threshold ranges indicate that, beyond these thresholds, models with higher complexity perform better than models with lower complexity.

[0104] One possible implementation, step S103 includes: if the edge score of an image block falls within a first preset range, the image block is assigned to a first image processing model for processing; if the edge score of an image block falls within a second preset range, the image block is assigned to a second image processing model for processing; if the edge score of an image block falls within a third preset range, the image block is assigned to a third image processing model for processing; wherein, any value in the first preset range is less than any value in the second preset range, any value in the second preset range is less than any value in the third preset range, the number of feature dimensions of the first image processing model is less than the number of feature dimensions of the second image processing model, and the number of feature dimensions of the second image processing model is less than the number of feature dimensions of the third image processing model.

[0105] Assume that the edge score is denoted as e, the first preset range is e≤T1, the second preset range is T1<e≤T2, the third preset range is e≥T2, and T1>T2, that is, any value in the third preset range is greater than any value in the second preset range, and any value in the second preset range is greater than any value in the first preset range. The first image processing model corresponding to the first preset range is denoted as F1, the second image processing model corresponding to the second preset range is denoted as F2, the third image processing model corresponding to the third preset range is denoted as F3, the number of feature dimensions of F3 is greater than the number of feature dimensions of F2, and the number of feature dimensions of F2 is greater than the number of feature dimensions of F1, that is, the complexity of the third image processing model is greater than the complexity of the second image processing model, and the complexity of the second image processing model is greater than the complexity of the first image processing model.

[0106] In this way, when the edge score e of an image block satisfies e≤T1, that is, falls within the first preset range, the first image processing model F1 is used to process the image block; when the edge score of the image block satisfies T1<e≤T2, that is, falls within the second preset range, the second image processing model F2 is used to process the image block; when the edge score e of the image block satisfies e≥T2, that is, falls within the third preset range, the third image processing model F3 is used to process the image block. Therefore, a simple image block with less edge information is processed by an image processing model with lower complexity (that is, fewer feature dimensions), which can reduce unnecessary performance overhead and speed up the processing speed; a complex image block with more edge information is processed by an image processing model with higher complexity (that is, more feature dimensions), so as to ensure the image processing effect. As described above, on the basis of ensuring the image processing effect in this embodiment, unnecessary performance overhead is reduced, and the image processing speed is accelerated.

[0107] Step S104: Splice the processed image blocks according to the position information to obtain a processed complete image.

[0108] After the image blocks of different complexities are input into the image processing models of corresponding complexities for processing, the processed image blocks are spliced according to the position information in step 101, so as to obtain a processed complete image that meets the user's visual requirements.

[0109] Figure 2 This is a schematic flowchart of a second image processing method provided in the embodiments of the present application. In this embodiment, the to-be-processed image is a noisy image, and the image processing model is a denoising model.

[0110] Noisy images can be caused by a variety of factors. For example, due to the influence of factors such as the imaging environment and transmission channel, Gaussian noise is always mixed into the image during acquisition, compression and transmission, causing blurring and loss of image information; or due to weather factors, raindrop noise is introduced into the obtained image; or due to external interference such as high-frequency interference, high-frequency interference stripes, also known as moiré patterns, appear on the photosensitive elements of digital cameras, scanners and other devices, which introduce moiré pattern noise into the obtained image.

[0111] These noises can severely impact subsequent tasks such as video processing, image analysis, and target tracking; therefore, image denoising technology is crucial in modern image processing systems. In this embodiment, the pre-trained denoising model can remove Gaussian noise, raindrop noise, or moiré noise, among others.

[0112] If the same high-complexity denoising model is used to process all image patches of the entire image in order to ensure the denoising quality of the processed image patches, it will bring unnecessary computational overhead and prolong the processing time.

[0113] To address this, this embodiment proposes another image processing method, which will be described below in conjunction with the appendix. Figure 3 The schematic diagram of the image processing flow illustrates the image processing method of this embodiment. Specifically, it includes the following steps:

[0114] Step S201: Acquire a noisy image, divide the noisy image into multiple image blocks, and record the position information of each image block.

[0115] Step S201 in this embodiment is largely the same as step 101 in the above embodiment. The difference is that the image to be processed in this embodiment is a noisy image, which can be an image with Gaussian noise, moiré noise, or raindrop noise, etc. To avoid repetition, the similarities will not be described again here.

[0116] Step S202: Determine the edge score of each image block in the multiple image blocks.

[0117] Step S202 in this embodiment is largely the same as step 102 in the above embodiment. The difference is that, in this embodiment, since the image to be processed is a noisy image, and noise can affect the extraction of edges in image blocks, a pre-trained model can be used to obtain the binarized edge map of each image block. By training the model to extract edge maps in noisy images, the accuracy of edge extraction is improved, and the noise in the image blocks is avoided from affecting edge extraction. Similarities will not be repeated here to avoid repetition.

[0118] This is feasible. In this embodiment, a pre-trained encoder can be used to obtain the binarized edge map of each image block. The training process of the pre-trained encoder includes the following steps: First, a first clear image sample is obtained, and noise is added to the first clear image sample to generate a first noisy image sample; then, the first clear image sample is downsampled by a factor of s, where s is the downsampling factor of the encoder; after that, edge detection is performed on the downsampled first clear image sample to generate the target binarized edge map; finally, the encoder is trained using the first noisy image sample as input and the target binarized edge map as the supervision label.

[0119] Since the goal of training the encoder is to enable it to accurately extract edge maps from noisy image patches, a first clear image sample is obtained when selecting samples. Each image in the first clear image sample is free of noise. A first noisy image sample is obtained by adding noise to the first clear image sample. The type of noise added is determined based on the type of noise in the noisy image. If the noise in the noisy image is Gaussian noise, the noise added to the first clear image sample is Gaussian noise; if the noise in the noisy image is raindrop noise, the noise added to the first clear image sample is raindrop noise, and so on.

[0120] Because the encoder performs downsampling by a factor of 's' during image patch processing, the size of the binary edge map output by the encoder is 1 / 's' times the size of the input image patch. Therefore, to ensure that the target binary edge map and the encoder's output image have the same size during training, facilitating the calculation of the loss function, the first sharp image sample needs to be downsampled by a factor of 's'. Then, edge detection is performed on the downsampled first sharp image sample to generate the target binary edge map. In this way, the resulting target binary edge map has the same size as the output image obtained by inputting the first sharp image sample into the encoder—both are 1 / 's' times the size of the first sharp image sample—facilitating the calculation of the loss function.

[0121] It should be noted that since the first clear image sample is free of noise, edge extraction operators can be used to perform edge detection on the downsampled first clear image sample, such as the Canny edge operator, which will not be elaborated here.

[0122] After obtaining the first noisy image sample and the target binarized edge map, the encoder is trained using the first noisy image sample as input and the target binarized edge map as the supervision label. During training, the encoder is input with the first noisy image sample to obtain the output image. The loss function is calculated based on the output image and the corresponding target binarized edge map of the first noisy image sample. This process is iterated until the loss function converges, at which point the encoder training is complete.

[0123] It is worth noting that this embodiment uses the encoder as an example to illustrate the training process of the model. If other models are used, their training process is roughly the same as that of the encoder, and will not be repeated here.

[0124] After obtaining the binarized edge maps of each image block, in this embodiment, the edge density of each image block can be calculated as the edge score, or global pooling can be performed on the binarized edge maps of each image block to obtain the edge score of each image block. The specific calculation process is the same as the calculation process in step 102 of the above embodiment, and will not be repeated here.

[0125] Step S203: Based on the different preset ranges in which the edge scores of each image block fall, each image block is assigned to a denoising model of different complexity corresponding to the preset range for processing, and the processed image block is obtained.

[0126] Step S203 in this embodiment is largely the same as step 103 in the above embodiment. The difference is that, since the image to be processed in this embodiment is a noisy image, a denoising model is used for image processing. Depending on the type of noise in the noisy image, the type of noise removed by the denoising model is also different. If the noise in the noisy image is Gaussian noise, the type of noise that the denoising model can remove is also Gaussian noise; if the noise in the noisy image is raindrop noise, the type of noise that the denoising model can remove is also raindrop noise, and so on. Similarities will not be repeated here to avoid repetition.

[0127] Before step S203, a pre-defined correspondence between different preset ranges and denoising models of different complexities is established. Therefore, denoising models of different complexities need to be trained first. Then, based on the trained denoising models of different complexities, the boundary thresholds between each denoising model are calculated, thereby determining the different preset ranges corresponding to denoising models of different complexities based on the boundary thresholds.

[0128] First, the training process of denoising models with different complexities will be explained.

[0129] The number of models should be determined according to needs. For example, if you want to obtain three denoising models with different complexities, you need to provide three initial models with different numbers of feature dimensions. For example, you can provide small, medium, and large initial models. For example, the number of feature dimensions of the small denoising model F1 is 8-16-32, the number of feature dimensions of the medium denoising model F2 is 16-32-48, and the number of feature dimensions of the large denoising model F3 is 32-48-64. This is just an example. You can choose the number of models and the number of feature dimensions in each model according to your needs, as long as the feature requirements are met.

[0130] After obtaining three initial models (small, medium, and large), three types of clear image samples were acquired. The number of edge features in the three types of clear image samples differs. The first type of clear image sample has no edge features or very few edge features, such as... Figure 4 As shown in (a); the second type of clear image sample contains edge features, but the number of edge features is small or the pattern is simple, such as... Figure 4 As shown in (b), the second type of clear image sample has a large number of edge features or complex edge feature patterns, such as... Figure 4 As shown in (c), noise was added to the three clear image samples to generate three noisy image samples.

[0131] The type of noise added here is determined based on the type of noise in the noisy image. If the type of noise in the noisy image is Gaussian noise, the noise added to the third clear image sample is Gaussian noise; if the type of noise in the noisy image is raindrop noise, the noise added to the third clear image sample is raindrop noise, and so on.

[0132] The first noisy image sample is used as the training sample, and the first clear image sample is used as the supervision label to train a small denoising model F1.

[0133] The second type of noisy image sample is used as the training sample, and the second type of clear image sample is used as the supervision label to train the medium-sized denoising model F2.

[0134] The third type of noisy image sample is used as the training sample, and the third type of clear image sample is used as the supervision label to train the large-scale denoising model F3.

[0135] In this way, we can obtain three denoising models with different levels of complexity (different numbers of feature dimensions): small, medium, and large.

[0136] Secondly, how to calculate the boundary threshold between the three denoising models with different levels of complexity (different number of feature dimensions) based on the trained small, medium and large denoising models.

[0137] A second clear image sample is obtained, and noise is added to the second clear image sample to generate a second noisy image sample; the edge score of the second clear image sample is calculated; the second noisy image sample is input into at least two denoising models of different complexities to obtain the corresponding denoised images; the similarity between the denoised images obtained by each denoising model and the second clear image sample is calculated; the second clear image samples with similarity differences between different denoising models less than a preset value are selected, and the maximum edge score among them is taken as the boundary threshold; the preset range corresponding to the denoising models of different complexities is determined according to the boundary threshold.

[0138] The following explanation uses at least two denoising models of different complexities as examples to illustrate three denoising models F1, F2, and F3 with different complexity (different numbers of feature dimensions).

[0139] First, obtain the second clear image sample {y i |i=1,2,3,4,...,n}, where n represents the total number of second clear image samples, y i Let {x} represent the i-th second clear image; add noise to the second clear image sample to generate a second noisy image sample. i |i=1,2,3,4,...,n}, where n represents the total number of second noise image samples, x i Let represent the i-th second noisy image. The type of noise added here can be determined based on the type of noise in the noisy image. If the type of noise in the noisy image is Gaussian noise, the noise added to the third clear image sample is Gaussian noise; if the type of noise in the noisy image is raindrop noise, the noise added to the third clear image sample is raindrop noise, and so on.

[0140] Secondly, the second clear image sample {y} can be calculated using the same method as in step 202. i Marginal fractions of |i=1,2,3,4,...,n}, denoted as {e i |i=1,2,3,4,...,n}.

[0141] Specifically, calculating the edge score of the second clear image sample includes: inputting the second noisy image sample corresponding to the second clear image sample into a pre-trained encoder to obtain the binarized edge map of the corresponding second clear image sample; and calculating the edge score of the second clear image sample based on the binarized edge map of the corresponding second clear image sample.

[0142] In this embodiment, the edge score of the second clear image sample is calculated using the same method as in step 202, ensuring the consistency of the calculation method. As a result, the subsequent determination of the boundary threshold for distinguishing each denoising model based on the edge score of the second clear image sample is more accurate.

[0143] A second clear image sample can be obtained using a pre-trained encoder. i The binarized edge graph {E} of |i=1,2,3,4,...,n} i |i=1,2,3,4,...,n}, then, for the binarized edge map {E i |i=1,2,3,4,...,n} is subjected to global pooling to calculate the edge score {e i |i=1,2,3,4,...,n}.

[0144] Next, the effectiveness of different denoising models in restoring image patches was evaluated. The second noisy image sample {x} was used. i The values ​​of |i=1,2,3,4,...,n} are fed into a small denoising model F1 to obtain the first denoised image. The second noisy image sample {x i The values ​​of |i=1,2,3,4,...,n} are fed into a medium-sized denoising model F2 to obtain the second denoised image. The second noisy image sample {x i The values ​​of |i=1,2,3,4,...,n} are fed into a large-scale denoising model F3 to obtain the third denoised image.

[0145] Then, the denoised image of each denoising model and the second clear image sample {y} are calculated. i The similarity of |i=1,2,3,4,...,n}.

[0146] It is feasible to calculate the similarity between the denoised image obtained by each denoising model and the second clear image sample, including: calculating the peak signal-to-noise ratio (PSNR) of the denoised image obtained by each denoising model and the second clear image sample. The similarity here can be measured using PSNR. First denoised image Compared with the second clear image sample {y i The peak signal-to-noise ratio (PSNR) of |i=1,2,3,4,...,n} can be expressed as: Second denoised image Compared with the second clear image sample {y i The peak signal-to-noise ratio (PSNR) of |i=1,2,3,4,...,n} can be expressed as: Third denoised image Compared with the second clear image sample {y i The peak signal-to-noise ratio (PSNR) of |i=1,2,3,4,...,n} can be expressed as:

[0147] Finally, based on the peak signal-to-noise ratio and edge scores of each denoising model, two boundary thresholds T1 and T2 are determined for the three denoising models F1, F2 and F3. T1 is the boundary threshold between the small denoising model F1 and the medium denoising model F2, and T2 is the boundary threshold between the medium denoising model F2 and the large denoising model F3.

[0148] To determine the demarcation threshold T1 between the small denoising model F1 and the medium denoising model F2, a preset value C is set. When the absolute value of the difference between the peak signal-to-noise ratio results of the two denoising models is less than the preset value C, it indicates that the denoising effects of the two denoising models on this noisy image are relatively close. By screening out second noisy image samples where the similarity difference between the small denoising model F1 and the medium denoising model F2 is less than the preset value C, the maximum edge score therein is determined. Below this maximum edge score, the denoising effects of the two models are substantially the same. At this time, to save computational overhead, the model with lower complexity is preferred to process the noisy image; above this maximum edge score, the difference in denoising effects between the two models is relatively large. Therefore, the maximum edge score is used as the demarcation threshold T1. At this time, any noisy image with an edge score less than the demarcation threshold T1 can be processed by the small denoising model F1. The demarcation threshold T1 between the small denoising model F1 and the medium denoising model F2 can be expressed by the following formula (1):

[0149]

[0150] To determine the demarcation threshold T2 between the medium denoising model F2 and the large denoising model F3, a preset value C is also set. When the absolute value of the difference between the peak signal-to-noise ratio results of the two denoising models is less than the preset value C, it indicates that the denoising effects of the two denoising models on this noisy image are relatively close. By screening out second noisy image samples where the similarity difference between the medium denoising model F2 and the large denoising model F3 is less than the preset value C, the maximum edge score therein is determined. Below this maximum edge score, the denoising effects of the two models are substantially the same. At this time, to save computational overhead, the model with lower complexity is preferred to process the noisy image; above this maximum edge score, the difference in denoising effects between the two models is relatively large. Therefore, the maximum edge score is used as the demarcation threshold T2. At this time, any noisy image with an edge score less than the demarcation threshold T2 can be processed by the medium denoising model F2. The demarcation threshold T2 between the medium denoising model F2 and the large denoising model F3 can be expressed by the following formula (2):

[0151]

[0152] Finally, after obtaining the demarcation threshold T1 between the small denoising model F1 and the medium denoising model F2, and the demarcation threshold T2 between the medium denoising model F2 and the large denoising model F3, the preset range corresponding to the small denoising model can be determined as e≤T1, the preset range corresponding to the medium denoising model F2 as T1<e≤T2, and the preset range corresponding to the large denoising model F3 as e≥T2.

[0153] Through the above process, the respective preset ranges corresponding to the small denoising model F1, the medium denoising model F2, and the large denoising model F3 can be obtained.

[0154] It is worth noting that the preset value C can be set according to actual needs. Generally speaking, if the difference between the peak signal-to-noise ratio results of the two denoising models is less than 0.1, the denoising effect is considered to be relatively close. Therefore, C can be set to 0.1. Of course, it can also be set to 0.2, 0.15, 0.05, etc. as needed.

[0155] Thus, in this embodiment, the noisy image is segmented into multiple image blocks, and the edge score of each image block is calculated. A preset range within which the edge score of each image block falls is determined, and a denoising model corresponding to that preset range is then used to process the image block. For simple image blocks with smaller edge scores (i.e., fewer edge information), a denoising model with lower complexity (i.e., fewer feature dimensions) is used, reducing unnecessary performance overhead and speeding up processing. For complex image blocks with smaller edge scores (i.e., more edge information), a denoising model with higher complexity (i.e., more feature dimensions) is used to ensure image processing quality. Using a denoising model of the same complexity to process all image blocks uniformly may not adequately adapt to the noise complexity of different image blocks. For example, simple image blocks may be overprocessed, while complex image blocks may be underprocessed, resulting in an unbalanced overall denoising effect. In this embodiment, image blocks are input into denoising models of corresponding complexity based on their complexity. This allows for more precise processing of different image blocks. Simple image blocks use lightweight, simple denoising models to avoid overprocessing, reduce unnecessary computational overhead, save computing resources, and speed up processing. Complex image blocks use complex denoising models to ensure that image block details are preserved. This allows for concentrated computing resources to process complex image blocks, resulting in higher overall computational efficiency. This is especially true when noise distribution in the image is uneven and the image complexity varies greatly, making resource allocation more reasonable. At the same time, the overall effect of the processed image is more natural, with a better balance between detail preservation and noise removal.

[0156] In summary, this embodiment reduces unnecessary performance overhead and speeds up image processing while ensuring image processing quality. It has low hardware requirements and is suitable for resource-constrained environments such as mobile devices.

[0157] Step S204: Stitch the processed image blocks together according to their position information to obtain the complete processed image.

[0158] Step S204 in this embodiment is largely the same as step 104 in the above embodiment. To avoid repetition, it will not be described again in this embodiment.

[0159] Figure 5 This is a flowchart illustrating the third image processing method provided in this application embodiment. In this embodiment, the image to be processed is a low-resolution image, and the image processing model is a super-resolution model.

[0160] Image super-resolution is an image processing technique that aims to reconstruct a high-resolution image from a low-resolution image using software algorithms. This technique is crucial for improving image quality and increasing detail clarity, and it has wide applications in fields such as image processing, satellite imaging, medical imaging, and video enhancement.

[0161] Currently, image super-resolution is usually achieved using deep learning techniques, most commonly super-resolution models. Super-resolution models learn a large number of low-resolution image-to-high-resolution image pairs and are able to predict the missing high-frequency details in low-resolution images.

[0162] If the same high-complexity super-resolution model is used to process all image patches of the entire image in order to ensure the resolution of the super-resolution image patches, it will bring unnecessary computational overhead and prolong the processing time.

[0163] To address this, this embodiment proposes an image processing method, which is described below in conjunction with the appendix. Figure 6 The schematic diagram of the image processing flow illustrates the image processing method of this embodiment. Specifically, it includes the following steps:

[0164] Step S301: Acquire a low-resolution image, divide the low-resolution image into multiple image blocks, and record the position information of each image block.

[0165] Step S301 in this embodiment is largely the same as step 101 in the above embodiment, except that the image to be processed in this embodiment is a low-resolution image. To avoid repetition, the similarities will not be described again here.

[0166] Step S302: Determine the edge score of each image block in the multiple image blocks.

[0167] In this embodiment, step S302 is largely the same as step 102 in the above embodiment. The difference is that, in this embodiment, since the image to be processed is a low-resolution image, and low-resolution images do not contain noise, the binarized edge maps of each image block can be obtained directly using an edge extraction algorithm, or a pre-trained model can be used to obtain the binarized edge maps of each image block. To avoid repetition, the similarities will not be repeated here.

[0168] After obtaining the binarized edge maps of each image block, in this embodiment, the edge density of each image block can be calculated as the edge score, or global pooling can be performed on the binarized edge maps of each image block to obtain the edge score of each image block. The specific calculation process is the same as the calculation process in step 102 of the above embodiment, and will not be repeated here.

[0169] Step S303: Based on the different preset ranges in which the edge scores of each image block fall, each image block is assigned to a super-resolution model of different complexity corresponding to the preset range for processing, and the processed image block is obtained.

[0170] In this embodiment, step S303 is largely the same as step 103 in the above embodiment. The difference is that, in this embodiment, since the image to be processed is a low-resolution image, the image processing model adopts a super-resolution model.

[0171] Before step S303, a pre-defined correspondence between different preset ranges and super-resolution models of different complexities is established. Therefore, super-resolution models of different complexities need to be trained first. Then, based on the trained super-resolution models of different complexities, the boundary thresholds between each super-resolution model are calculated, thereby determining the different preset ranges corresponding to super-resolution models of different complexities based on the boundary thresholds.

[0172] First, the training process of super-resolution models with different complexities will be explained.

[0173] The number of models should be determined according to needs. For example, if you want to obtain three super-resolution models with different complexities, you need to provide three initial models with different numbers of feature dimensions. For example, you can provide small, medium, and large initial models. For example, the number of feature dimensions of the small super-resolution model K1 is 8-16-32, the number of feature dimensions of the medium super-resolution model K2 is 16-32-48, and the number of feature dimensions of the large super-resolution model K3 is 32-48-64. This is just an example. You can choose the number of models and the number of feature dimensions in each model according to your needs, as long as the feature requirements are met.

[0174] After obtaining three initial models (small, medium, and large), three types of high-resolution image samples were acquired. The number of edge features in the three high-resolution image samples differs: the first type of high-resolution image sample has no edge features or very few edge features; the second type of high-resolution image sample has edge features but few or few edge features; and the third type of high-resolution image sample has many edge features or complex edge feature patterns. The resolution of these three high-resolution image samples was then reduced to generate three low-resolution image samples.

[0175] The first low-resolution image sample is used as the training sample, and the first high-resolution image sample is used as the supervision label to train a small super-resolution model K1.

[0176] The second type of low-resolution image sample is used as the training sample, and the second type of high-resolution image sample is used as the supervision label to train the medium-sized super-resolution model K2.

[0177] The third type of low-resolution image sample is used as the training sample, and the third type of high-resolution image sample is used as the supervision label to train the large super-resolution model K3.

[0178] In this way, we can obtain three super-resolution models K1, K2, and K3 with different levels of complexity (different numbers of feature dimensions): small, medium, and large.

[0179] Secondly, how to calculate the boundary threshold between the super-resolution models K1, K2, and K3, which are of different complexity (different number of feature dimensions) and trained small, medium, and large.

[0180] Acquire high-resolution image samples, reduce the resolution of the high-resolution image samples to generate low-resolution image samples; calculate the edge scores of the low-resolution image samples; input the low-resolution image samples into at least two super-resolution models with different complexities to obtain the corresponding super-resolution images; calculate the similarity between the super-resolution images obtained by each super-resolution model and the high-resolution image samples; select high-resolution image samples whose similarity difference between different super-resolution models is less than a preset value, and take the maximum edge score among them as the boundary threshold; determine the preset range corresponding to the super-resolution models with different complexities based on the boundary threshold.

[0181] The following explanation uses at least two super-resolution models with different complexities as examples: three super-resolution models K1, K2, and K3 with different complexities (different numbers of feature dimensions).

[0182] First, obtain high-resolution image samples {h i |i=1,2,3,4,...,n}, where n represents the total number of high-resolution image samples, h i Represents the i-th high-resolution image; low-resolution image samples are generated by reducing the resolution of the high-resolution image samples {l}. i |i=1,2,3,4,...,n}, where n represents the total number of low-resolution image samples, l i This represents the i-th low-resolution image.

[0183] Secondly, the same method as in step 302 can be used to calculate the low-resolution image samples {l i Marginal fractions {e} of |i=1,2,3,4,...,n} i |i=1,2,3,4,...,n}.

[0184] Next, the super-resolution performance of different super-resolution models on image patches was evaluated.

[0185] Low-resolution image samples {l i The first super-resolution image is obtained by feeding |i=1,2,3,4,...,n} into a small super-resolution model K1. Low-resolution image samples {l i The values ​​of |i=1,2,3,4,...,n} are fed into a medium-sized super-resolution model K2 to obtain the second super-resolution image. Low-resolution image samples {l i The sequence |i=1,2,3,4,...,n} is fed into the large super-resolution model K3 to obtain the third super-resolution image.

[0186] Then, the super-resolution image and the high-resolution image sample {h} of each super-resolution model are calculated. i The similarity of |i=1,2,3,4,...,n}.

[0187] It is feasible to calculate the similarity between the super-resolution images obtained by each super-resolution model and the high-resolution image samples, including: calculating the structural similarity (SSIM) between the super-resolution images obtained by each super-resolution model and the high-resolution image samples. The similarity here can be measured using structural similarity (SSIM). First super-resolution image. With high-resolution image samples {h i The structural similarity of |i=1,2,3,4,...,n} can be expressed as: Second super-resolution image With high-resolution image samples {h i The structural similarity of |i=1,2,3,4,...,n} can be expressed as: Third denoised image With high-resolution image samples {h i The structural similarity of |i=1,2,3,4,...,n} can be expressed as:

[0188] Finally, based on the structural similarity and edge scores of each super-resolution model, two boundary thresholds Q1 and Q2 are determined for the three super-resolution models K1, K2 and K3. Q1 is the boundary threshold between the small super-resolution model K1 and the medium super-resolution model K2, and Q2 is the boundary threshold between the medium super-resolution model K2 and the large super-resolution model K3.

[0189] To determine the demarcation threshold Q1 between the small super-resolution model K1 and the medium super-resolution model K2, a preset value U is set. When the absolute value of the difference in structural similarity between the two super-resolution models is less than the preset value U, it indicates that the super-resolution effects of the two super-resolution models on the same low-resolution image are relatively close. By screening low-resolution image samples whose similarity difference between the small super-resolution model K1 and the medium super-resolution model K2 is less than the preset value U, the maximum edge score among them is determined. Below this maximum edge score, the super-resolution effects of the two models are basically the same. At this time, to save computational overhead, the model with lower complexity is preferred to process the low-resolution image; above this maximum edge score, the difference in super-resolution effects between the two models is relatively large. Therefore, the maximum edge score is used as the demarcation threshold Q1. At this time, all low-resolution images with edge scores less than the demarcation threshold Q1 can be processed by the small super-resolution model K1. The demarcation threshold Q1 between the small super-resolution model K1 and the medium super-resolution model K2 can be expressed by the following formula (3):

[0190]

[0191] To determine the demarcation threshold Q2 between the medium super-resolution model K2 and the large super-resolution model K3, a preset value U is also set. When the absolute value of the difference in structural similarity between the two super-resolution models is less than the preset value U, it indicates that the super-resolution effects of the two super-resolution models on this low-resolution image are relatively close. By screening low-resolution image samples whose similarity difference between the medium super-resolution model K2 and the large super-resolution model K3 is less than the preset value U, the maximum edge score among them is determined. Below this maximum edge score, the super-resolution effects of the two models are basically the same. At this time, to save computational overhead, the model with lower complexity is preferred to process the low-resolution image; above this maximum edge score, the difference in super-resolution effects between the two models is relatively large. Therefore, the maximum edge score is used as the demarcation threshold Q2. At this time, all low-resolution images with edge scores less than the demarcation threshold Q2 can be processed by the medium super-resolution model K2. The demarcation threshold Q2 between the medium super-resolution model K2 and the large super-resolution model K3 can be expressed by the following formula (4):

[0192]

[0193] Finally, after obtaining the demarcation threshold Q1 between the small super-resolution model K1 and the medium super-resolution model K2, and the demarcation threshold Q2 between the medium super-resolution model K2 and the large super-resolution model K3, it can be determined that the preset range corresponding to the small super-resolution model K1 is e≤Q1, the preset range corresponding to the medium super-resolution model K2 is Q1<e≤Q2, and the preset range corresponding to the large super-resolution model K3 is e≥Q2.

[0194] Through the above process, the respective preset ranges corresponding to the small super-resolution model K1, the medium super-resolution model K2, and the large super-resolution model K3 can be obtained.

[0195] It is worth noting that the preset value U can be set according to actual needs. Generally speaking, if the difference in structural similarity between two denoising models is less than 0.01, the denoising effect is considered to be relatively close. Therefore, U can be set to 0.01. Of course, it can also be set to 0.02, 0.03, 0.04, etc. as needed.

[0196] Thus, in this embodiment, the low-resolution image is segmented into multiple image blocks, and the edge score of each image block is calculated. A preset range within which the edge score of each image block falls is determined, and a super-resolution model corresponding to that preset range is used to process the image blocks. For simple image blocks with smaller edge scores (i.e., less edge information), a super-resolution model with lower complexity (i.e., fewer feature dimensions) is used, reducing unnecessary performance overhead and accelerating processing. For complex image blocks with smaller edge scores (i.e., more edge information), a super-resolution model with higher complexity (i.e., more feature dimensions) is used to ensure image processing quality. Using a model of the same complexity to process all image blocks uniformly may not adequately adapt to the different edge complexities of the image blocks. For example, simple image blocks may be over-processed, while complex image blocks may be under-processed, resulting in an unbalanced overall super-resolution effect. In this embodiment, image blocks are input into super-resolution models of corresponding complexity based on their edge complexity. This allows for more precise processing of different image blocks. Simple image blocks use lightweight, simple super-resolution models to avoid overprocessing, reduce unnecessary computational overhead, save computational resources, and speed up processing. Complex image blocks use complex super-resolution models to ensure detail preservation. This allows for concentrated computational resources to process complex image blocks, resulting in higher overall computational efficiency. This is especially true when the image has uneven edge distribution and significant differences in image complexity, leading to more reasonable resource allocation. Furthermore, the processed image has a more natural overall effect, with a better balance between detail preservation and super-resolution performance.

[0197] In summary, this embodiment reduces unnecessary performance overhead and speeds up image processing while ensuring image processing quality. It has low hardware requirements and is suitable for resource-constrained environments such as mobile devices.

[0198] Step S304: The processed image blocks are stitched together according to their position information to obtain the complete processed image.

[0199] Step S304 in this embodiment is largely the same as step 104 in the above embodiment. To avoid repetition, it will not be described again in this embodiment.

[0200] It should be understood that the above examples are provided to help those skilled in the art understand the embodiments of this application, and are not intended to limit the embodiments of this application to the specific values ​​or scenarios illustrated. Those skilled in the art can obviously make various equivalent modifications or changes based on the above examples, and such modifications or changes also fall within the scope of the embodiments of this application.

[0201] Secondly, embodiments of this application also provide an image processing apparatus, such as... Figure 7 As shown, the image processing device 700 includes: an image segmentation module 710, used to acquire an image to be processed, segment the image to be processed into multiple image blocks, and record the position information of each image block;

[0202] The edge score determination modulus 720 is used to determine the edge score of each image patch in multiple image patches. The edge score characterizes the richness of edge information in the image patch.

[0203] The edge discrimination module 730 is used to assign each image block to an image processing model of different complexity corresponding to the preset range based on the different preset ranges in which the edge scores of each image block fall, so as to obtain the processed image block. The number of feature dimensions of the image processing models of different complexities is different, and the larger the value in the preset range, the more feature dimensions of the image processing model corresponding to the preset range.

[0204] The image aggregation module 740 is used to stitch together the processed image blocks according to their position information to obtain the processed complete image.

[0205] In this embodiment, the image processing device calculates the edge score of each image block to measure the richness of edge information in each image block, i.e., the complexity of each image block. Based on the different preset ranges in which the edge scores of each image block fall, the image blocks of different complexities are assigned to image processing models of different complexities. If the edge score of an image block is small, the richness of edge information in the image block is low, i.e., the image block is low in complexity. In this case, the image block with low complexity is input into an image processing model with low complexity (i.e., fewer feature dimensions). If the edge score of an image block is large, the richness of edge information in the image block is high, i.e., the image block is high in complexity. In this case, the image block with high complexity is input into an image processing model with high complexity (i.e., more feature dimensions). Compared to inputting all image patches into a model of the same complexity, this embodiment processes image patches of different complexities into image processing models of corresponding complexities. Simpler image patches with less edge information are processed using a lower-complexity model (i.e., fewer feature dimensions), reducing unnecessary performance overhead and accelerating processing. Conversely, more complex image patches with more edge information are processed using a higher-complexity model (i.e., more feature dimensions), ensuring optimal image processing results. Therefore, this embodiment reduces unnecessary performance overhead and accelerates image processing speed while maintaining optimal image processing performance.

[0206] It should be understood that the image processing apparatus in this embodiment is the apparatus claim corresponding to the image processing method in the above embodiments. The image processing method in any of the above embodiments can be implemented by the image processing apparatus in this embodiment. To avoid repetition, it will not be described in detail here.

[0207] Thirdly, embodiments of this application also provide an electronic device, such as... Figure 8 The diagram shown is a schematic structural block diagram of an electronic device 800, which includes a processor 810, a memory 820, a communication interface 830, and a bus 840.

[0208] The processor 810 can be connected to the memory 820. The memory 820 can be used to store the program code and data. Therefore, the memory 820 can be a storage unit inside the processor 810, an external storage unit independent of the processor 810, or a component that includes both the storage unit inside the processor 810 and the external storage unit independent of the processor 810.

[0209] Optionally, the electronic device 800 may further include a bus 840. The memory 820 and communication interface 830 can be connected to the processor 810 via the bus 840. The bus 840 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 840 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The symbol is represented by only one line, but this does not mean that there is only one bus or one type of bus.

[0210] It should be understood that in the embodiments of this application, the processor 810 may be a central processing unit (CPU). The processor 810 may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. Alternatively, the processor 810 may employ one or more integrated circuits to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0211] The memory 820 may include read-only memory and random access memory, and provides instructions and data to the processor 810. A portion of the processor 810 may also include non-volatile random access memory. For example, the processor 810 may also store device type information.

[0212] When the electronic device 800 is running, the processor 810 executes computer execution instructions in the memory 820 to perform the operation steps of the image processing method in any of the above embodiments using the hardware resources in the electronic device 800.

[0213] It should be understood that the above and other operations and / or functions of the various modules in the electronic device 800 of this application embodiment are respectively for implementing Figure 1 , Figure 2 and Figure 5 The corresponding process of the method in the document will not be elaborated here for the sake of brevity.

[0214] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer instructions; when the computer-readable storage medium is operated on an image processing apparatus, the image processing apparatus performs the method as described above.

[0215] The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or can include one or more data storage devices such as servers or data centers that can be integrated with media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media, or semiconductor media (e.g., solid-state disks (SSDs)).

[0216] Fifthly, this application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed, implement the method for determining a charging / discharging strategy provided in the embodiments of this application.

[0217] Sixthly, this application also provides a computer program product, which includes: computer program code, which, when run on a computer, causes the computer to execute the method for determining a charge / discharge strategy provided in the embodiments of this application.

[0218] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive (SSD).

[0219] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the embodiments of this application.

[0220] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0221] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0222] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0223] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0224] If the aforementioned function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application embodiment, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a memory (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0225] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be included within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.

Claims

1. An image processing method, characterized in that, Applied to electronic devices, the method includes: Acquire the image to be processed, divide the image to be processed into multiple image blocks, and record the position information of each image block; Determine the edge score of each of the plurality of image blocks, wherein the edge score characterizes the richness of edge information in the image block; Based on the different preset ranges in which the edge scores of each image block fall, each image block is assigned to an image processing model of different complexity corresponding to the preset range for processing, resulting in processed image blocks. The number of feature dimensions of the image processing models of different complexities is different, and the larger the value in the preset range, the more feature dimensions the image processing model corresponding to the preset range has. The processed image blocks are stitched together according to the location information to obtain the complete processed image.

2. The method according to claim 1, characterized in that, The step of assigning each image block to an image processing model of different complexity corresponding to the preset range based on the different edge scores of each image block, to obtain processed image blocks, includes: If the edge score of the image block falls within the first preset range, the image block is assigned to the first image processing model for processing; If the edge score of the image block falls within the second preset range, the image block is assigned to the second image processing model for processing; If the edge score of the image block falls within the third preset range, the image block will be assigned to the third image processing model for processing. Wherein, any value in the first preset range is less than any value in the second preset range, any value in the second preset range is less than any value in the third preset range, the number of feature dimensions of the first image processing model is less than the number of feature dimensions of the second image processing model, and the number of feature dimensions of the second image processing model is less than the number of feature dimensions of the third image processing model.

3. The method according to claim 1 or 2, characterized in that, Determining the edge score of each of the plurality of image blocks includes: Binary edge maps of each of the multiple image blocks are obtained; The edge score of each image block is calculated based on the binarized edge map of each image block.

4. The method according to claim 3, characterized in that, The step of obtaining the binarized edge map of each of the plurality of image patches includes: Each of the plurality of image blocks is input into a pre-trained encoder to obtain a binarized edge map corresponding to each image block.

5. The method according to claim 4, characterized in that, The image to be processed is a noisy image, and the image processing model is a denoising model; The training process of the pre-trained encoder includes the following steps: Obtain a first clear image sample, and add noise to the first clear image sample to generate a first noisy image sample; The first clear image sample is downsampled by a factor of s, where s is the downsampling factor of the encoder; Edge detection is performed on the first clear image sample after downsampling to generate a target binarized edge map; The encoder is trained using the first noisy image sample as input and the target binarized edge map as the supervision label.

6. The method according to any one of claims 3 to 5, characterized in that, The step of calculating the edge score of each image patch based on the binarized edge map of each image patch includes: Global pooling is performed on the binarized edge maps of each image block to obtain the edge scores of each image block.

7. The method according to any one of claims 1 to 4, characterized in that, The image to be processed is a noisy image, and the image processing model is a denoising model; Before assigning each image block to an image processing model of different complexity corresponding to the preset range based on the different edge scores of each image block, and obtaining the processed image block, the method further includes: Obtain a second clear image sample, and add noise to the second clear image sample to generate a second noisy image sample; Calculate the edge score of the second clear image sample; The second noisy image sample is input into at least two denoising models of different complexities to obtain the corresponding denoised image; Calculate the similarity between the denoised image obtained by each of the denoising models and the second clear image sample; The second clear image sample with a similarity difference of less than a preset value among different denoising models is selected, and the maximum edge score among them is taken as the boundary threshold. The preset range corresponding to the denoising models of different complexities is determined based on the boundary threshold.

8. The method according to claim 7, characterized in that, The calculation of the edge score of the second clear image sample includes: The second noisy image sample corresponding to the second clear image sample is input into the pre-trained encoder to obtain the binarized edge map corresponding to the second clear image sample; The edge score of the second clear image sample is calculated based on the binarized edge map corresponding to the second clear image sample.

9. The method according to claim 7, characterized in that, The calculation of the similarity between the denoised image obtained by each of the denoising models and the second clear image sample includes: Calculate the peak signal-to-noise ratio (PSNR) of the denoised image obtained by each of the denoising models and the second clear image sample.

10. The method according to any one of claims 1 to 4, characterized in that, The image to be processed is a low-resolution image, and the image processing model is a super-resolution model; Before assigning each image block to an image processing model of different complexity corresponding to the preset range based on the different edge scores of each image block, and obtaining the processed image block, the method further includes: Acquire high-resolution image samples, and reduce the resolution of the high-resolution image samples to generate low-resolution image samples; Calculate the edge scores of the low-resolution image samples; The low-resolution image samples are input into at least two super-resolution models of different complexities to obtain the corresponding super-resolution images; Calculate the similarity between the super-resolution image obtained by each super-resolution model and the high-resolution image sample; High-resolution image samples with similarity differences between different super-resolution models less than a preset value are selected, and the maximum edge score among them is taken as the boundary threshold. The preset range corresponding to the super-resolution models of different complexities is determined based on the boundary threshold.

11. An electronic device, characterized in that, include: A memory and a processor, the memory being used to store a computer program executable on the processor, the processor being used to perform the steps of the image processing method as described in any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the steps of the image processing method as described in any one of claims 1 to 10.

13. A computer program product, characterized in that, The computer program product includes: computer program code, which, when executed by an electronic device, causes the electronic device to perform the steps of the image processing method as described in any one of claims 1 to 10.

Images