Image processing method and device, medium and product

Abstract

The invention provides an image processing method which comprises the steps that hierarchical wavelet transformation is carried out on a to-be-processed image, multiple layers of to-be-processed sub-images are obtained, and each layer of to-be-processed sub-image comprises a high-frequency sub-band image and a low-frequency sub-band image; inputting the high-frequency sub-band image in each layer of to-be-processed sub-image into a first image processing model for moire removal, and obtaining a high-frequency sub-band repair image of the layer; inputting the low-frequency sub-band image in each layer of to-be-processed sub-image into a second image processing model for color feature restoration to obtain a low-frequency sub-band restoration image of the layer; and performing wavelet inverse transformation on the high-frequency sub-band repair image and the low-frequency sub-band repair image layer by layer to obtain a target image. The application also provides an electronic device, a computer readable medium, and a computer program product.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, electronic device, computer-readable medium, and computer program product. Background Technology

[0002] Moiré patterns are interference stripes that appear in an image, usually caused by the proximity of the spatial frequency of the image's photosensitive element to the stripes in the image.

[0003] Typically, methods for removing moiré patterns work in the spatial domain. First, the moiré patterns in the image are removed, and then the colors in the image are updated. This process is complex and cannot completely remove the moiré patterns, reducing image processing efficiency and failing to obtain high-resolution images. Summary of the Invention

[0004] This application provides an image processing method, an electronic device, a computer-readable medium, and a computer program product.

[0005] In a first aspect, embodiments of this application provide an image processing method, comprising: performing hierarchical wavelet transform on the image to be processed to obtain multiple layers of sub-images to be processed, each layer of sub-images to be processed including a high-frequency sub-band image and a low-frequency sub-band image; inputting the high-frequency sub-band image of each layer of sub-images to be processed into a first image processing model to remove moiré patterns, thereby obtaining a high-frequency sub-band repair image for that layer; inputting the low-frequency sub-band image of each layer of sub-images to be processed into a second image processing model to repair color features, thereby obtaining a low-frequency sub-band repair image for that layer; and performing inverse wavelet transform on the high-frequency sub-band repair image and the low-frequency sub-band repair image layer by layer to obtain a target image.

[0006] Secondly, embodiments of this application provide an electronic device, including: one or more processors; and a memory storing one or more programs thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the image processing methods in the embodiments of this application.

[0007] Thirdly, embodiments of this application provide a readable storage medium storing a computer program that, when executed by a processor, implements any of the image processing methods described in the embodiments of this application.

[0008] Fourthly, embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, implements any of the image processing methods described in the embodiments of this application.

[0009] The image processing method in this embodiment is based on wavelet transform and performs different levels of processing on the image to be processed. Each level can represent information at different scales. Each sub-image to be processed includes a high-frequency sub-band image and a low-frequency sub-band image, enabling more refined processing of sub-band images in each frequency band and improving image processing accuracy. Furthermore, by using a first image processing model to remove moiré patterns from the high-frequency sub-band images in each sub-image to be processed, the moiré patterns in the high-frequency band can be removed more thoroughly, achieving better results. Since color features are mainly low-frequency information with less variation, a second image processing model is used to repair the color of the low-frequency sub-band images in each sub-image to be processed, resulting in better repair of color features in the low-frequency band. Further, by performing inverse wavelet transform on the repaired high-frequency and low-frequency sub-band repaired images layer by layer, the moiré patterns at each scale can be completely removed, and the color features of the image can be restored, reconstructing a clearer target image. Attached Figure Description

[0010] In the accompanying drawings of the embodiments of this application:

[0011] Figure 1 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0012] Figure 2 This is a schematic diagram of the structure of a first image processing model provided in an embodiment of this application;

[0013] Figure 3 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0014] Figure 4 This is a schematic diagram of the structure of an initial repair unit in a first image processing model provided in an embodiment of this application;

[0015] Figure 5 This is a schematic diagram of the structure of a densely connected module provided in an embodiment of this application;

[0016] Figure 6 This is a schematic diagram of the structure of a detail enhancement unit in a first image processing model provided in an embodiment of this application;

[0017] Figure 7 This is a schematic diagram of the structure of a second image processing model provided in an embodiment of this application;

[0018] Figure 8 A block diagram of an image processing apparatus provided in an embodiment of this application;

[0019] Figure 9 This is a block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0021] The present application will be described more fully below with reference to the accompanying drawings; however, the embodiments shown may be embodied in different forms, and the present application should not be construed as limited to the embodiments set forth below. Rather, these embodiments are provided so that this application will be thorough and complete, and will enable those skilled in the art to fully understand the scope of the application.

[0022] The accompanying drawings of the embodiments of this application are used to provide a further understanding of the embodiments of this application and constitute a part of the specification. They are used together with the detailed embodiments to explain this application and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the description of the detailed embodiments with reference to the accompanying drawings.

[0023] This application can be described with reference to plan views and / or cross-sectional views using the ideal schematic diagram of this application. Therefore, the example illustrations can be modified according to manufacturing techniques and / or tolerances.

[0024] Where there is no conflict, the various embodiments of this application and the features thereof may be combined with each other.

[0025] The terminology used in this application is for describing specific embodiments only and is not intended to limit the application. The term "and / or" as used herein includes any and all combinations of one or more of the associated enumerated entries. The singular forms "a" and "the" as used herein are also intended to include the plural forms unless the context clearly indicates otherwise. The terms "comprising," "made of," etc., as used herein, specify the presence of the stated feature, integral, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof.

[0026] Unless otherwise specified, all terms used in this application (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this application, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined in this application.

[0027] This application is not limited to the embodiments shown in the accompanying drawings, but includes modifications to the configuration based on the manufacturing process. Therefore, the areas illustrated in the drawings are schematic, and the shapes of the areas shown in the drawings illustrate the specific shapes of the areas of the element, but are not intended to be limiting.

[0028] In some related technologies, moiré patterns appearing in images are usually separated and removed directly in the spatial domain, but this process is complex and cannot completely eliminate moiré patterns.

[0029] In other related techniques, the image needs to be converted to the frequency domain first, and then the moiré pattern removal and color updates are performed sequentially. While this method simplifies the processing to some extent, it leads to a loss of image texture and reduces the sharpness of the final image.

[0030] To address the aforementioned problems, this application provides an image processing method, an electronic device, a computer-readable medium, and a computer program product.

[0031] In a first aspect, embodiments of this application provide an image processing method.

[0032] Figure 1 This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. This image processing method can be applied to an image processing device. Figure 1 As shown, the image processing method includes the following steps.

[0033] Step S101: Perform hierarchical wavelet transform on the image to be processed to obtain multiple sub-images to be processed.

[0034] Wavelet Transform (WT) is a time-frequency analysis technique that can convert a raw image (e.g., the image to be processed) into information at different scales and frequencies.

[0035] By employing wavelet transform, the image to be processed is decomposed, enabling hierarchical processing to obtain multiple levels of sub-images. Each level of sub-image corresponds to a different scale. Furthermore, each sub-image includes both high-frequency and low-frequency sub-band images, facilitating corresponding processing of information in different frequency bands. This helps extract local features within each frequency band, providing convenience for subsequent image processing.

[0036] The high-frequency sub-band map is a sub-map representing image features within a high-frequency range (e.g., a high-frequency range determined by a first preset frequency band threshold). Each layer of the sub-map to be processed may include multiple high-frequency sub-band maps, with different high-frequency sub-band maps corresponding to different frequency band information within the high-frequency range, thus enabling frequency subdivision within the high-frequency range. The low-frequency sub-band map is a sub-map representing image features within a low-frequency range (e.g., a low-frequency range determined by a second preset frequency band threshold), enabling frequency subdivision within the low-frequency range. The first preset frequency band threshold and the second preset frequency band threshold may be the same or different (e.g., the first preset frequency band threshold is greater than the second preset frequency band threshold).

[0037] It should be noted that the specific implementation process of wavelet transform will vary depending on the characteristics of different images and the selected wavelet function. Therefore, an appropriate wavelet basis function can be selected to transform the image based on its specific features.

[0038] Step S102: Input the high-frequency sub-band image of each layer to be processed into the first image processing model to remove moiré patterns and obtain the high-frequency sub-band repair image of that layer; input the low-frequency sub-band image of each layer to be processed into the second image processing model to repair color features and obtain the low-frequency sub-band repair image of that layer.

[0039] The first image processing model is used to refine the removal of moiré patterns in the high-frequency subband image, extracting edge details and brightness information to the maximum extent possible, and removing moiré patterns to improve the clarity of the restored high-frequency subband image. The second image processing model is used to repair the color features in the low-frequency subband image, so that the restored low-frequency subband image can improve the brightness, contrast, and clarity of the sub-image.

[0040] Moiré patterns are interference stripes that appear in images, usually caused by the proximity of the spatial frequency of the image's sensor to the stripes in the image. The presence of moiré patterns in an image can affect its sharpness; therefore, image processing models are needed to remove them.

[0041] By processing the high-frequency sub-band image in each input sub-image layer using the first image processing model, the moiré patterns in each sub-band image can be effectively reduced, achieving refined moiré pattern removal, thereby resulting in a higher clarity in the output high-frequency sub-band restoration image of that layer.

[0042] It should be noted that the first image processing model and the second image processing model run in parallel. Therefore, they can process both the high-frequency sub-band image and the low-frequency sub-band image of each layer of the image to be processed at the same time, thereby speeding up the image processing.

[0043] Step S103: Perform wavelet inverse transform on the high-frequency subband repair map and the low-frequency subband repair map layer by layer to obtain the target image.

[0044] The inverse wavelet transform is the reverse process of the wavelet transform. It is used to reverse the result of the wavelet transform to recover the original image. For example, the inverse transform matrix of the wavelet transform can be used to perform matrix multiplication on the result of the wavelet transform to recover the original image.

[0045] In this embodiment, the inverse wavelet transform is used to recombine the high-frequency subband repair images and low-frequency subband repair images of each layer, and perform image restoration layer by layer to obtain the reconstructed target image, thereby making the obtained target image clearer.

[0046] For example, for each sub-image to be processed, the high-frequency sub-band repair image and the low-frequency sub-band repair image corresponding to each sub-image to be processed are fused (for example, based on frequency domain features, the high-frequency sub-band repair image and the low-frequency sub-band repair image are processed according to preset rules, or the high-frequency sub-band repair image and the low-frequency sub-band repair image are spliced ​​together, etc.), so as to obtain the repair sub-image corresponding to the sub-image to be processed in that layer, so that the repair sub-image in that layer can have more accurate and richer information than any sub-band repair image, thereby improving the repair effect of the repair sub-image.

[0047] It should be noted that the success of the inverse wavelet transform is closely related to factors such as the selected wavelet basis function, the number of decomposition levels, and the quantization method. Therefore, it is necessary to select appropriate wavelet functions and parameters for the inverse transform. These wavelet functions should match the wavelet basis functions used in the wavelet transform to ensure that the inverse wavelet transform can accurately reconstruct a target image that is clearer than the image to be processed, and that the target image does not contain moiré patterns.

[0048] The image processing method in this embodiment is based on wavelet transform and performs different levels of processing on the image to be processed. Each level can represent information at different scales. Each sub-image to be processed includes a high-frequency sub-band image and a low-frequency sub-band image, enabling more refined processing of sub-band images in each frequency band and improving image processing accuracy. Furthermore, by using a first image processing model to remove moiré patterns from the high-frequency sub-band images in each sub-image to be processed, the moiré patterns in the high-frequency band can be removed more thoroughly, achieving better results. Since color features are mainly low-frequency information with less variation, a second image processing model is used to repair the color of the low-frequency sub-band images in each sub-image to be processed, resulting in better repair of color features in the low-frequency band. Further, by performing inverse wavelet transform on the repaired high-frequency and low-frequency sub-band repaired images layer by layer, the moiré patterns at each scale can be completely removed, and the color features of the image can be restored, reconstructing a clearer target image.

[0049] In some exemplary embodiments, the hierarchical wavelet transform of the image to be processed in step S101 to obtain multiple sub-images to be processed can be implemented in the following way:

[0050] The image to be processed is decomposed for the first time according to the preset wavelet transform algorithm to obtain the high-frequency sub-band image and low-frequency sub-band image of the first layer; when the current decomposition number is less than or equal to n, the low-frequency sub-band image of the (i-1)th layer is decomposed according to the preset wavelet transform algorithm to obtain the high-frequency sub-band image and low-frequency sub-band image of the i-th layer.

[0051] Where i is the current decomposition count, 1≤i≤n, and n is the preset decomposition count.

[0052] For example, when n is 3, using a budget wavelet transform algorithm (such as the Haar wavelet transform, Meyer wavelet transform, Morlet wavelet transform, etc.) to perform a first decomposition on the image to be processed yields the first low-frequency sub-band image and the first high-frequency sub-band image of the first layer. Then, performing a second decomposition on the first low-frequency sub-band image yields the second low-frequency sub-band image and the second high-frequency sub-band image of the second layer. Similarly, performing a third decomposition on the second low-frequency sub-band image yields the third low-frequency sub-band image and the third high-frequency sub-band image of the third layer. Through the above decomposition, multiple low-frequency sub-band images and multiple high-frequency sub-band images can be obtained.

[0053] It should be noted that since moiré patterns are typically high-frequency irregular stripes, while the slow-changing color regions in an image mostly appear in the low-frequency region, the above decomposition allows for the decomposition of moiré pattern features into the high-frequency sub-band image and the decomposition of color features into the low-frequency sub-band image. This facilitates subsequent processing of the corresponding features in the high- and low-frequency sub-band images, enabling parallel image processing. Specifically, moiré patterns in the high-frequency sub-band image are removed while color features in the low-frequency sub-band image are repaired, thus improving image processing efficiency.

[0054] In some embodiments, each subgraph to be processed includes multiple high-frequency subband graphs and one low-frequency subband graph.

[0055] For example, if each sub-image to be processed includes one low-frequency sub-band image and three high-frequency sub-band images, then after decomposing the image to be processed three times, three low-frequency sub-band images of different scales and nine high-frequency sub-band images can be obtained. Furthermore, the high-frequency sub-band images in each sub-image to be processed correspond to the same scale.

[0056] By decomposing the image to be processed multiple times, multiple high- and low-frequency sub-band maps of different scales can be obtained. Sub-band maps of each frequency band can be processed at different scales, thereby improving the processing accuracy of the image.

[0057] In some exemplary embodiments, step S102, which involves inputting the high-frequency subband image of each layer to be processed into the first image processing model for moiré removal to obtain the high-frequency subband restoration image of that layer, can be implemented as follows: Based on multiple downsampling and upsampling methods, the high-frequency subband image of each layer to be processed is analyzed to obtain the high-frequency feature set corresponding to the high-frequency subband image; the moiré weight of each high-frequency feature in the high-frequency feature set is determined respectively; an initial restoration image of the high-frequency subband image is generated according to the high-frequency feature set corresponding to the high-frequency subband image and the moiré weight of each high-frequency feature in the high-frequency feature set; and detail enhancement is performed on the initial restoration image of the high-frequency subband image to obtain the high-frequency subband restoration image of the layer.

[0058] The high-frequency feature set includes multiple high-frequency features at different scales. The moiré weight of each high-frequency feature is negatively correlated with the intensity of the moiré patterns it contains.

[0059] In some embodiments, the moiré weights of high-frequency features in the high-frequency feature set are determined by inputting multiple high-frequency features of different scales into a global average pooling layer and a fully connected layer for processing, thereby obtaining the moiré weights of each high-frequency feature.

[0060] After obtaining the moiré weights of each high-frequency feature, it is necessary to perform dot product processing on each high-frequency feature and its corresponding moiré weight to obtain the dot product feature corresponding to each high-frequency feature. Then, the dot product features corresponding to multiple high-frequency features are concatenated to obtain the initial repair map of each high-frequency sub-band map.

[0061] For example, if a high-frequency subband image is decomposed into a set of high-frequency features comprising three high-frequency features, then each of the three high-frequency features in the set needs to be multiplied by its corresponding three moiré weights. The resulting three multiplied features are then concatenated to obtain an initial restoration image of the high-frequency subband image. This allows the different moiré weights of the high-frequency features to highlight their respective proportions within the high-frequency subband image. Furthermore, since the moiré weight of each high-frequency feature is negatively correlated with the intensity of its moiré patterns—in other words, a larger moiré weight indicates a smaller intensity of the moiré patterns within that feature—multiplying each moiré weight by its corresponding high-frequency feature can reduce the intensity of the moiré patterns in the multiplied feature. This allows the initial restoration image to eliminate moiré patterns and preliminarily restore the correct image features in the high-frequency subband image.

[0062] Furthermore, detail enhancement is performed on the initial repair images of each high-frequency subband image to further remove moiré patterns in the detail features of the initial repair images. This allows the high-frequency subband repair images corresponding to the obtained subband images to be processed to eliminate the interference of moiré patterns and improve the accuracy of the image features of the high-frequency subband repair images.

[0063] In some exemplary embodiments, based on multiple downsampling and upsampling methods, the high-frequency sub-band maps in each layer of the sub-graph to be processed are analyzed to obtain the high-frequency feature set corresponding to the high-frequency sub-band map, including:

[0064] The high-frequency sub-band image is downsampled multiple times using the nearest neighbor interpolation algorithm to obtain the first feature set corresponding to the high-frequency sub-band image; the receptive field of each first feature is expanded to obtain the second feature set corresponding to the high-frequency sub-band image; and the second features in the second feature set corresponding to the high-frequency sub-band image are upsampled according to the nearest neighbor interpolation algorithm and the downsampling factor corresponding to each second feature to obtain the high-frequency feature set corresponding to the high-frequency sub-band image.

[0065] The first feature set includes multiple first features with different sampling multiples, and the second feature set includes multiple second features.

[0066] Nearest neighbor interpolation is an algorithm for scaling and resolution conversion of images. The core of this algorithm is to find the nearest pixel in the original image for each pixel in the high-frequency sub-band image and assign the value of that nearest pixel to the pixel in the high-frequency sub-band image. It is simple to implement, computationally efficient, and requires relatively few computing resources, enabling rapid processing of various high-frequency sub-band images.

[0067] Using the aforementioned nearest neighbor interpolation algorithm, multiple downsampling operations are performed on a high-frequency sub-band image (e.g., simultaneous 1x downsampling, 2x downsampling, and 4x downsampling). This yields a first feature set corresponding to the high-frequency sub-band image, which includes multiple first features with different sampling multiples.

[0068] Before performing multiple downsampling on each high-frequency sub-band image, it is also necessary to perform dimensionality enhancement on each high-frequency sub-band image separately (for example, using 1*1 convolutional blocks to process the high-frequency sub-band image to give it more dimensional features) to obtain the multi-dimensional features corresponding to each high-frequency sub-band image, which is convenient for subsequent analysis.

[0069] In some embodiments, expanding the receptive field of each first feature can be achieved by inputting each first feature into a dense connection module for processing to obtain a second feature set corresponding to each high-frequency subband map. This second feature set includes multiple second features.

[0070] The dense connection module is a network defined by k dilated convolutions and their corresponding activation functions. Each dilated convolution has a different dilation rate, and k is an integer greater than or equal to 5. For example, when k equals 5, the dense connection module includes 5 dilated convolutions, and the dilation rates of each dilated convolution are 1, 2, 3, 2, and 1, respectively.

[0071] It should be noted that when the downsampling factor is M, the upsampling factor for the second feature corresponding to that downsampling factor is also M, where M is an integer greater than or equal to 1. In other words, the high-frequency features in the high-frequency feature set corresponding to the final high-frequency subband map are all features of the same size.

[0072] The nearest neighbor interpolation algorithm is used to downsample each high-frequency sub-band image by multiple factors to obtain the first feature set corresponding to each high-frequency sub-band image. The receptive field of each first feature is then expanded to obtain more dimensional features, which is convenient for subsequent analysis. Then, the nearest neighbor interpolation algorithm is used to upsample the second features in the second feature set corresponding to each high-frequency sub-band image based on the downsampling factor corresponding to each second feature. This allows for feature restoration of each second feature at different downsampling factors, so that all high-frequency features in the high-frequency feature set corresponding to the high-frequency sub-band image are features of the same size, and the high-frequency features corresponding to the high-frequency sub-band image are enriched.

[0073] In some exemplary embodiments, detail enhancement is performed on the initial repair map of the high-frequency subband map to obtain the high-frequency subband repair map of the layer, including: extracting positional attention features from the initial repair map of the high-frequency subband map to obtain a first branch feature, and expanding the receptive field of the initial repair map of the high-frequency subband map to obtain a second branch feature; connecting the first branch feature and the second branch feature to obtain a third feature; and fusing the third feature with the initial repair map to obtain the high-frequency subband repair map of the layer.

[0074] The location attention feature refers to the spatial dependency between any two locations in the initial restoration map. For a specific feature in the initial restoration map, the first branch feature can be obtained by weighting and updating the specific feature using features from all locations in the initial restoration map. The weighting values ​​used are determined based on the feature similarity between the corresponding two locations.

[0075] The method for expanding the receptive field of the initial repair map of the high-frequency subband map can be any of the following:

[0076] 1) Overlay more layers on the initial repair map of each high-frequency subband map, so that the obtained second branch features are the features obtained by expanding the depth of the initial repair map;

[0077] 2) The initial repair map of each high-frequency sub-band map is sampled twice, and the sampled features are merged to obtain the second branch features;

[0078] 3) A filter dilation method is used to expand the receptive field of the initial repair map of each high-frequency subband map, thereby obtaining the second branch features.

[0079] Furthermore, the first branch feature and the second branch feature are connected to obtain the third feature; and the third feature is fused with the initial repair map to obtain the final repair map of each high-frequency subband map.

[0080] When a layer of subgraphs to be processed contains only one high-frequency subband image, the final repaired image of that high-frequency subband image is the high-frequency subband repaired image of that layer. When a layer of subgraphs to be processed contains multiple high-frequency subband images, the final repaired images of the multiple high-frequency subband images need to be stitched together to obtain the high-frequency subband repaired image of that layer.

[0081] In some related technologies, image separation is used in the spatial domain to remove moiré patterns from images. While this can remove some moiré patterns, it cannot completely remove moiré patterns from the details of the image, resulting in incomplete removal.

[0082] As one embodiment of this application, by extracting the positional attention features from the initial restoration image of each high-frequency sub-band image, that is, extracting the spatial dependency features between any two positions in the initial restoration image, the positional attention features between each position can be more clearly defined; and by connecting them with the second branch features (i.e., features after expanding the receptive field), the moiré patterns in the high-frequency sub-band image can be removed more accurately, thus achieving precise removal of moiré patterns; thereby, the high-frequency sub-band restoration image corresponding to the final sub-image to be processed is free of moiré patterns, improving the image clarity.

[0083] In some embodiments, the first image processing model includes an initial inpainting unit and a detail enhancement unit. For example, Figure 2 This is a schematic diagram of the structure of a first image processing model provided in an embodiment of this application. Figure 2 As shown, the first image processing model 200 includes an initial repair unit 201 and a detail enhancement unit 202.

[0084] The initial repair unit 201 is used to perform preliminary removal of moiré patterns on the high-frequency subband map of each input sub-map to be processed, so as to obtain the initial repair map of the high-frequency subband map and thereby reduce the moiré patterns in the high-frequency subband map.

[0085] The detail enhancement unit 202 is used to perform detail restoration on the initial restoration map of the high-frequency subband map input by the initial restoration unit 201, so as to further remove moiré patterns in the high-frequency features and make the final restoration map of the high-frequency subband map of each layer clearer.

[0086] By inputting each high-frequency sub-band image in the sub-image to be processed into the initial repair unit 201 in the first image processing model for preliminary removal of moiré patterns, an initial repair image of each high-frequency sub-band image is obtained. Then, the initial repair image of the high-frequency sub-band image is input into the detail enhancement unit 202 for further repair, which can improve the removal of moiré patterns and make the final repair image of the high-frequency sub-band image clearer.

[0087] In some exemplary embodiments, step S102, which involves inputting the low-frequency sub-band image of each layer to be processed into the second image processing model for color feature restoration to obtain the low-frequency sub-band restoration image of that layer, can be implemented in the following way:

[0088] Feature extraction is performed on the low-frequency subband map, and the receptive field of the extracted low-frequency features is expanded to obtain multiple color features to be repaired; each color feature to be repaired is multiplied by its corresponding preset color weight to obtain the low-frequency subband repair map of the layer.

[0089] In this process, color features can be extracted from the low-frequency sub-band image through a dense connection module to obtain low-frequency features; then, any of the extended feature receptive field methods in the embodiments of this application are used to expand the receptive field of the extracted low-frequency features, thereby obtaining multiple color features to be repaired.

[0090] The dense connection module is a network defined by p dilated convolutions and their corresponding activation functions. Each dilated convolution has a different dilation rate, where p is an integer greater than or equal to 3. For example, when p equals 3, the dense connection module includes 3 dilated convolutions, with dilation rates of 1, 2, and 1 respectively.

[0091] In some embodiments, a multilayer perceptron network can be used to perform a dot product operation on each color feature to be repaired and its corresponding preset color weight, so that the color features in the low-frequency sub-band repair map corresponding to the final obtained sub-map to be processed are repaired.

[0092] In some exemplary embodiments, the step S103 of performing inverse wavelet transform on the high-frequency subband repair map and the low-frequency subband repair map layer by layer to obtain the target image can be implemented in the following way:

[0093] Perform inverse wavelet transform on the high-frequency subband restoration map and the low-frequency subband restoration map of the nth layer respectively, and then fuse the obtained transformed high-frequency subband restoration map and the transformed low-frequency subband restoration map of the nth layer to obtain the low-frequency subband restoration map of the (n-1)th layer, where n is the preset number of decompositions;

[0094] If the level corresponding to the obtained low-frequency subband repair map is greater than 1, perform inverse wavelet transform on the high-frequency subband repair map of the (j+1)th level to obtain the transformed high-frequency subband repair map of the (j+1)th level, and fuse the low-frequency subband repair map of the (j+1)th level and the transformed high-frequency subband repair map of the (j+1)th level to obtain the low-frequency subband repair map of the jth level, where j≤n-2.

[0095] Given the low-frequency subband restoration map of layer 1, the target image is generated based on the low-frequency subband restoration map and the high-frequency subband restoration map of layer 1.

[0096] Inverse wavelet transform refers to performing a reverse transformation on the result of the wavelet transform, so that the recovered target image is the same as the image to be processed, but does not contain moiré patterns.

[0097] In some embodiments, the Haar wavelet inverse transform algorithm can be used to perform an inverse wavelet transform on the high-frequency subband restoration map of the nth layer to obtain the transformed high-frequency subband restoration map of the nth layer, and the Haar wavelet inverse transform algorithm can be used to perform an inverse wavelet transform on the low-frequency subband restoration map of the nth layer to obtain the transformed low-frequency subband restoration map of the nth layer; then, the transformed high-frequency subband restoration map of the nth layer and the transformed low-frequency subband restoration map of the nth layer are fused for the first time (for example, the features of the two subband restoration maps are extracted and matched, and then fused according to the matching results) to obtain the low-frequency subband restoration map of the (n-1)th layer.

[0098] Then, the high-frequency subband restoration map in the restoration submap of the (n-1)th layer is processed again using the Haar wavelet inverse transform algorithm to obtain the transformed high-frequency subband restoration map of the (n-1)th layer. Then, the low-frequency subband restoration map of the (n-1)th layer and the transformed high-frequency subband restoration map of the (n-1)th layer are fused for the second time to obtain the low-frequency subband restoration map of the (n-2)th layer.

[0099] The high-frequency subband restoration image of layer (n-2) is then processed using the Haar wavelet inverse transform algorithm to obtain the transformed high-frequency subband restoration image of layer (n-2). Then, the low-frequency subband restoration image of layer (n-2) and the transformed high-frequency subband restoration image of layer (n-2) are fused for the third time to obtain the low-frequency subband restoration image of layer (n-3); ...; and so on, until the layer corresponding to the obtained low-frequency subband restoration image is equal to 1. At this point, the low-frequency subband restoration image of layer 1 is obtained. Then, the low-frequency subband restoration image of layer 1 and the high-frequency subband restoration image of layer 1 are stitched together to generate the target image.

[0100] Through the above operations, it is possible to fuse multi-level repair sub-images, so that the final target image can completely retain the features of each repair sub-image, so that the target image not only has no moiré patterns, but also improves the brightness of the target image, retains the color texture in the image to be processed, and improves the clarity of the target image.

[0101] In some exemplary embodiments, before performing hierarchical wavelet transform on the image to be processed in step S101 to obtain multiple sub-images to be processed, the method further includes:

[0102] The acquired sample images are subjected to hierarchical wavelet transforms to obtain multi-layer sample sub-images. The high-frequency sample sub-band images from each layer are input into a first image processing model for moiré pattern removal, resulting in a high-frequency sub-band restoration image for that layer. The low-frequency sub-band images from each layer are input into a second image processing model for color feature restoration, resulting in a low-frequency sub-band restoration image for that layer. Inverse wavelet transforms are performed layer by layer on both the high-frequency and low-frequency sample sub-band restoration images to obtain the inverse transform result image. Based on the loss information between the inverse transform result image and a preset inverse transform standard image, an image loss function is determined. The parameters of the first and second image processing models are updated according to the image loss function to obtain the updated image processing model.

[0103] Each layer of sample subplots includes high-frequency sample subplots and low-frequency sample subplots.

[0104] In some embodiments, the sample images include multiple images, and each sample image contains moiré patterns. Simultaneously with acquiring the sample images, it is also necessary to acquire an inverse transform standard image identical to the sample image but without moiré patterns, so that the trained image processing model can be accurately used to remove moiré patterns from the sample images.

[0105] After processing the sample image in the above manner, it is necessary to compare the inverse transform result image with the inverse transform standard image to determine the difference (i.e., loss information) between the inverse transform result image processed by the image processing model and the inverse transform standard image, so as to determine the image loss function based on the loss information.

[0106] In some embodiments, the image loss function includes one of the following functions: L1 norm (L1 Loss) loss function, mean squared error loss function, cross-entropy loss function, binary cross-entropy loss function, and smoothed L1 loss function.

[0107] By employing different image loss functions, the degree of difference between the inverse transform result image and the inverse transform standard image can be accurately characterized. Based on this degree of difference, the parameters of the first image processing model and the second image processing model are updated in reverse, so that the updated first and second image processing models can be used to process other sample images in the future. After repeating this process multiple times, the first image processing model can accurately remove moiré patterns from the sample images, and the second image processing model can accurately repair the color features in the sample images.

[0108] In some exemplary embodiments, the preset number of decompositions is n, and n is an integer greater than 1; the inverse transform standard image includes n preset sub-images; the image loss function is determined based on the loss information between the inverse transform result image and the preset inverse transform standard image, including: comparing the inverse transform result image corresponding to each sample sub-image with the preset sub-image of that layer to obtain the sub-image loss function; and determining the image loss function based on the n sub-image loss functions.

[0109] The inverse transform standard image is the image corresponding to the sample image without moiré patterns. That is, the only difference between the inverse transform standard image and the sample image is that the inverse transform standard image has no moiré patterns, while the sample image has moiré patterns.

[0110] The subgraph loss function is determined by comparing the inverse wavelet transform subgraph corresponding to the repaired sample subgraph at one level with its corresponding preset subgraph at the same level. The loss function is a comprehensive summary of the losses from all n subgraph loss functions.

[0111] For example, if n is set to 3, the inverse transform standard image can be decomposed into 3 preset sub-images. When comparing the inverse transform result image corresponding to each sample sub-image with the preset sub-image of that layer, the L1 loss function can be used as the sub-image loss function for that layer, and the sub-image loss value of that layer can be calculated. Furthermore, the 3 sub-image loss functions can be comprehensively evaluated to calculate the image loss function.

[0112] For example, the sub-image loss values ​​corresponding to the three preset sub-images are 0.1, 0.2, and 0.1 respectively. Then the final determined image loss function corresponds to an image loss value of 0.1 + 0.2 + 0.1 = 0.4.

[0113] By calculating the subgraph loss value of each layer using the subgraph loss function layer by layer as described above, the repair status of each subgraph can be clearly identified. Then, the image loss function is determined comprehensively based on the subgraph loss values ​​of each layer, which can more accurately measure the repair status of the sample image. Thus, the parameters of the first image processing model and the second image processing model are adjusted in reverse based on the image loss value, making the first image processing model and the second image processing model more accurate.

[0114] The following is combined with Figures 3-7 The image processing method in this application will be described in detail.

[0115] Figure 3 This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. This image processing method can be applied to an image processing device.

[0116] like Figure 3As shown, firstly, the image processing device performs a hierarchical wavelet transform on the image to be processed (e.g., three wavelet transforms) to obtain a first low-frequency sub-band image 311 and a first high-frequency sub-band image 312 in the first layer of the image to be processed, a second low-frequency sub-band image 321 and a second high-frequency sub-band image 322 in the second layer of the image to be processed, and a third low-frequency sub-band image 331 and a third high-frequency sub-band image 332 in the third layer of the image to be processed.

[0117] The second layer of sub-images to be processed is obtained by performing wavelet transform on the first low-frequency sub-band image 311, and the corresponding third layer of sub-images to be processed is obtained by performing wavelet transform on the second low-frequency sub-band image 321.

[0118] Then, the image processing device inputs the first low-frequency sub-band image 311, the second low-frequency sub-band image 321 and the third low-frequency sub-band image 331 into the second image processing model 320 for processing, so as to repair the color features in each low-frequency sub-band image.

[0119] Correspondingly, the image processing device inputs the first high-frequency subband image 312, the second high-frequency subband image 322 and the third high-frequency subband image 332 into the first image processing model 310 for processing, so as to remove moiré patterns in each high-frequency subband image.

[0120] Further, the image processing device performs inverse wavelet transform on the third low-frequency subband repair image 341 corresponding to the third low-frequency subband image 331 output by the second image processing model 320 and the third high-frequency subband repair image 342 corresponding to the third high-frequency subband image 332 output by the first image processing model 310, respectively. The inversely transformed third low-frequency subband repair image 341 and the inversely transformed third high-frequency subband repair image 342 are then input into the first fusion module 330 for fusion to obtain the second low-frequency subband repair image 351.

[0121] Similarly, the image processing device performs an inverse wavelet transform on the second high-frequency subband repair image 352, which corresponds to the second high-frequency subband image 322, output by the first image processing model 310, to obtain the inversely transformed second high-frequency subband repair image 352. Then, the second low-frequency subband repair image 351, the inversely transformed second high-frequency subband repair image 352, and the repair result corresponding to the first low-frequency subband image 311 output by the second image processing model 320 are input into the second fusion module 340 for fusion to obtain the first low-frequency subband repair image 361.

[0122] Finally, the image processing device stitches together the first high-frequency subband repair image 362, which corresponds to the first high-frequency subband image 312, output by the first image processing model 310, with the first low-frequency subband repair image 361 to obtain the target image.

[0123] The image processing device processes the image to be processed in the manner described above, which can make the final target image clearer and reduce the impact of moiré patterns on the image.

[0124] In some embodiments, the second image processing model 320 includes an initial repair unit (not shown in the figure) for initially removing moiré patterns in the high-frequency subband image to obtain an initial repaired image of the high-frequency subband image, thereby reducing the moiré patterns in the high-frequency subband image.

[0125] For example, Figure 4 This is a schematic diagram of the structure of an initial repair unit in a first image processing model provided in an embodiment of this application.

[0126] like Figure 4 As shown, the initial repair unit includes the following modules: a 1×1 convolutional block, a first activation function 321, multiple densely connected modules (e.g., densely connected module A, densely connected module B, and densely connected module C), multiple global average pooling layers (e.g., global average pooling layer A, global average pooling layer B, and global average pooling layer C), multiple first fully connected layers (e.g., first fully connected layer A, first fully connected layer B, and first fully connected layer C), multiple second fully connected layers (e.g., second fully connected layer A, second fully connected layer B, and second fully connected layer C), multiple first activation functions (e.g., first activation function A, first activation function B, and first activation function C), multiple second activation functions (e.g., second activation function A, second activation function B, and second activation function C), and a splicing module 322.

[0127] A 1×1 convolutional block and a first activation function 321 are used to increase the dimensionality of each high-frequency subband map of the input to make the processed high-frequency features richer.

[0128] In some embodiments, the first activation function 321, the first activation function A, the first activation function B, and the first activation function C can all be implemented using the ReLU function (or the LeakyReLU function); the second activation function A, the second activation function B, and the second activation function C can all be implemented using the sigmoid function.

[0129] The number of dense connection modules, global average pooling layers, first fully connected layers, second fully connected layers, first activation functions, and second activation functions is determined based on the parameters for downsampling the high-frequency features corresponding to a high-frequency sub-band image output by the first activation function 321 (or, based on the number of wavelet transform layers applied to the image to be processed). These downsampling parameters include the number of downsampling operations and the downsampling factor.

[0130] For example, Figure 4 The algorithm employs nearest-neighbor interpolation to downsample the high-frequency features corresponding to a high-frequency sub-band image three times: by a factor of 1, 2, and 4. This yields the corresponding high-frequency feature g (one-fold downsampling) input to dense connection module A, high-frequency feature g2 (two-fold downsampling) input to the second dense connection module B, and high-frequency feature g4 (four-fold downsampling) input to dense connection module C. Therefore, the number of dense connection modules, global average pooling layers, first fully connected layers, second fully connected layers, first activation functions, and second activation functions are all 3.

[0131] The number of the aforementioned densely connected modules, global average pooling layer, first fully connected layer, second fully connected layer, first activation function, and second activation function can also be 4, 5, etc. This application does not impose any restrictions on this, and will not elaborate further here.

[0132] refer to Figure 4 The dense connection module A processes the high-frequency feature g to obtain the corresponding high-frequency feature g. 1 The densely connected module B processes the twice-high-frequency feature g2 to obtain the corresponding high-frequency features. The densely connected module C processes the 4x high-frequency feature g4 to obtain the corresponding high-frequency features.

[0133] The dense connection module not only expands the receptive field of the input high-frequency features but also performs dimensionality enhancement on the processed features. For example, dense connection module B uses its dimensionality enhancement sub-module to perform a 2x upsampling of the high-frequency features after expanding the receptive field using a nearest neighbor interpolation algorithm to obtain the high-frequency features.

[0134] The densely connected module C uses its included dimension-enhancing sub-modules to perform a 4-fold upsampling of the high-frequency features after expanding the receptive field using a nearest-neighbor interpolation algorithm to obtain the high-frequency features. This ensures that the high-frequency features output by each densely connected module are features of the same size as the high-frequency feature g.

[0135] Furthermore, the high-frequency features g of the input are sequentially processed by a global average pooling layer A, a first fully connected layer A, a first activation function A, a second fully connected layer A, and a second activation function A.1 After processing, high-frequency feature g can be obtained. 1 The corresponding first Moiré ripple weights; similarly, the high-frequency features of the input are sequentially processed through a global average pooling layer B, a first fully connected layer B, a first activation function B, a second fully connected layer B, and a second activation function B. By processing, high-frequency features can be obtained. The corresponding second Moiré weights; the high-frequency features of the input are sequentially processed through a global average pooling layer C, a first fully connected layer C, a first activation function C, a second fully connected layer C, and a second activation function C. By processing, high-frequency features can be obtained. The corresponding third moiré weight.

[0136] Then, the high-frequency feature g is respectively 1 The high-frequency feature g is obtained by multiplying it by the corresponding first moiré weight. 2 ; High-frequency features The high-frequency features are obtained by multiplying the corresponding second moiré weights. High frequency characteristics The high-frequency features are obtained by multiplying the corresponding third moiré weight. Furthermore, the splicing module 322 is used to splice the high-frequency feature g. 2 High-frequency characteristics and high frequency characteristics By splicing the images, we can obtain the initial repaired image G of the high-frequency subband map.

[0137] The aforementioned densely connected modules are used to expand the receptive field of their respective input high-frequency features in order to prepare for subsequent multi-scale feature fusion.

[0138] For example, Figure 5 This is a schematic diagram of a densely connected module provided in an embodiment of this application. Figure 5 As shown, the dense connection module includes N sets of feature processing modules (e.g., feature processing module 510). Each set of feature processing modules includes a 3×3 conv, a third activation function, and a connection module, where N is an integer greater than or equal to 1.

[0139] refer to Figure 5 The 3×3conv, third activation function 5101, and connection module 5102 in feature processing module 5100 process the first feature; ..., the 3×3conv, third activation function 51N1, and connection module 51N2 in feature processing module 51N0 process the processing result output by the previous feature processing module; then, the output processing result is input into 1×1conv for dimensionality enhancement, and then activated by the fourth activation function 520 to obtain the output second feature.

[0140] In some embodiments, the third activation function 5101, ..., the third activation function 51N1, and the fourth activation function 520 can all be implemented using ReLU functions (or LeakyReLU functions).

[0141] In some embodiments, if N is set to 5, the inflation rate corresponding to each feature processing module can be set to [1, 2, 3, 2, 1].

[0142] The dense connection module expands the receptive field of the input first feature by using multiple feature processing modules in sequence. Then, it uses 1×1 conv to increase the dimensionality of the feature with expanded receptive field, so that the final second feature can reflect more detailed high-frequency features, which helps to remove moiré patterns in the high-frequency features in the subsequent process.

[0143] In some embodiments, the second image processing model 320 further includes a detail enhancement unit (not shown in the figure), which is used to perform detail restoration on the initial repaired image G of the high-frequency subband image to further remove moiré patterns in the high-frequency features, so that the final repaired image of the high-frequency subband image can be clearer.

[0144] For example, Figure 6 This is a schematic diagram of the structure of a detail enhancement unit in a first image processing model provided in an embodiment of this application.

[0145] like Figure 6 As shown, the detail enhancement unit includes a 1×1 conv, a first branch module 601, a second branch module 602, a connection module 603, a second activation function, a 1×1 conv, and a fusion module 604 connected in sequence.

[0146] The first branch module 601 is used to extract the positional attention features in the initial repair map G of the high-frequency subband map to obtain the first branch features.

[0147] The second branch module 602 is used to expand the receptive field of the initial repair map G of the high-frequency subband map to obtain the second branch features.

[0148] First, the detail enhancement unit uses a 1×1 convolution to increase the dimensionality of the initial restored image G of the high-frequency subband map, thereby obtaining multi-dimensional features of the initial restored image G. Then, the first branch module 601 extracts the positional attention features from the multi-dimensional features of the initial restored image G, obtaining the first branch feature G1; simultaneously, the second branch module 602 performs a 3×3 convolution on the multi-dimensional features of the initial restored image G again to expand the receptive field of the features, obtaining the second branch feature G2.

[0149] The first branch module 601 includes a 1x1 convolution, a first function, and two 3x3 convolutions. For example, the first function can be implemented using the sigmoid function.

[0150] Furthermore, the first branch feature G1 and the second branch feature G2 are input into the connection module 603 for connection to obtain the third feature; then, the third feature is activated by the second activation function, and the dimension of the third feature is reduced by using 1x1conv.

[0151] Finally, the fusion module 604 is used to fuse the third feature after dimensionality reduction and the initial repair map G to obtain the final repair map of the high-frequency subband map.

[0152] In some embodiments, Figure 7 This is a schematic diagram of the structure of a second image processing model provided in an embodiment of this application. Figure 7 As shown, the second image processing model is used to repair the color features of the input low-frequency subband image to obtain the low-frequency subband repair image.

[0153] refer to Figure 7 The second image processing model includes two 1x1 convolutions, two first activation functions, a dense connection module 701, and a multilayer perceptron module 720.

[0154] Among them, the dense connection module 710 is used to expand the receptive field of color features in the low-frequency subband map and obtain multiple color features to be repaired.

[0155] The multilayer perceptron module 720 is used to perform dot multiplication of each color feature to be repaired with its corresponding preset color weight to obtain the low-frequency sub-band repair map corresponding to the sub-image to be processed.

[0156] First, the color features in the input low-frequency subband image are dimension-enhanced using 1x1conv, and after activation by the first activation function, they are input into the dense connection module 710 for processing to extract color features in the extended low-frequency subband image and expand the receptive field of the color features to obtain multiple color features to be repaired.

[0157] In some embodiments, the dense connection module 710 may be implemented using three 3x3 dilated convolutions and a ReLU function (or a LeakyReLU function).

[0158] Then, the multiple color features to be repaired output by the dense connection module 710 are input into the multilayer perceptron module 720 for processing to obtain the preset color weights of the channels corresponding to each color feature to be repaired.

[0159] Next, the color feature to be repaired on each channel is multiplied by its corresponding preset color weight, and the multiplication result of each channel is input into 1x1conv for dimensionality enhancement. After passing through the first activation function, the low-frequency subband repair map can be obtained.

[0160] refer to Figure 7 The multilayer perceptron module 720 includes m 3x3conv and m activation functions (e.g., first activation function 721, ..., first activation function 72(m-1) and second activation function 72m), where m is an integer greater than or equal to 2.

[0161] In some embodiments, the first activation function 721, ..., the first activation function 72(m-1) can be implemented using the ReLU function (or the LeakyReLU function); the second activation function 72m can be implemented using the sigmoid function.

[0162] It should be noted that after wavelet transform, the image to be processed can obtain three low-frequency sub-band images at different scales. For each scale of the low-frequency sub-band image, a second image processing model is needed to repair the color features in the low-frequency sub-band image at each scale to obtain the repaired low-frequency sub-band images at each level.

[0163] Secondly, embodiments of this application provide an image processing apparatus.

[0164] Figure 8 This is a block diagram of an image processing apparatus provided in an embodiment of this application. The image processing apparatus can be a device involved in digital image processing (e.g., a server or terminal device), and it can support basic image operations (e.g., image reading operations, convolution operations, and image display).

[0165] like Figure 8 As shown, the image processing device 800 includes, but is not limited to, the following modules.

[0166] The first transformation module 801 is configured to perform hierarchical wavelet transform on the image to be processed to obtain multiple layers of sub-images to be processed, each layer of sub-images to be processed including a high-frequency sub-band image and a low-frequency sub-band image.

[0167] The first processing module 802 is configured to input the high-frequency subband image of each layer to be processed into the first image processing model for moiré pattern removal, thereby obtaining the high-frequency subband restoration image of that layer.

[0168] The second processing module 803 is configured to input the low-frequency sub-band image of each layer to be processed into the second image processing model for color feature restoration, thereby obtaining the low-frequency sub-band restored image of that layer.

[0169] The second transformation module 804 is configured to perform wavelet inverse transform on the high-frequency subband repair map and the low-frequency subband repair map layer by layer to obtain the target image.

[0170] It should be noted that the image processing device 800 in this embodiment can implement any of the image processing methods applied in this application embodiment.

[0171] According to the image processing apparatus of this application embodiment, the first transformation module performs wavelet transform-based processing on the image to be processed at different levels, which can represent information at different scales based on different levels. Each sub-image to be processed includes a high-frequency sub-band image and a low-frequency sub-band image, which can process the sub-band images in each frequency band more finely and extract the image processing accuracy. Furthermore, the first processing module uses a first image processing model to remove moiré patterns from the high-frequency sub-band images in each sub-image to be processed, which can remove moiré patterns in the high-frequency band more thoroughly and obtain better results. Since color features are mainly low-frequency information with less variation, the second processing module uses a second image processing model to repair the color of the low-frequency sub-band images in each sub-image to be processed, which can improve the repair effect of color features in the low-frequency band. Further, the second transformation module performs inverse wavelet transform on the repaired high-frequency sub-band repaired images and low-frequency sub-band repaired images layer by layer, which can completely remove moiré patterns at each scale and restore the color features of the image, reconstructing a clearer target image.

[0172] It should be clarified that this application is not limited to the specific configurations and processes described in the above embodiments and shown in the figures. For the sake of convenience and brevity, detailed descriptions of known methods are omitted here, and the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0173] Thirdly, embodiments of this application provide an electronic device, a computer-readable medium, and a computer program product.

[0174] Figure 9 This is a block diagram of an electronic device provided in an embodiment of this application.

[0175] like Figure 9 As shown, the electronic device includes at least one processor 901, at least one memory 902, and one or more I / O interfaces 903. The processor 901, memory 902, and I / O interfaces 903 are interconnected via a bus 904. The memory 902 stores one or more computer programs, which are executed by the at least one processor 901 to enable the at least one processor 901 to implement any of the image processing methods described in the above embodiments.

[0176] Among them, the processor is a device with data processing capabilities, including but not limited to the central processing unit (CPU); the memory is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read-write interface) is connected between the processor and the memory, enabling information exchange between the memory and the processor, including but not limited to the data bus (Bus).

[0177] The modules in the aforementioned electronic devices can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0178] This application also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements any of the image processing methods described in the above embodiments. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0179] This application also provides a computer program product, including a computer program that implements the above-described image processing method when executed by a processor.

[0180] Those skilled in the art will understand that all or some of the steps, systems, and devices disclosed above, as functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0181] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be executed by several physical components working together.

[0182] Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit (CPU), digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technique for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH) or other disk storage; read-only optical disc (CD-ROM), digital versatile disc (DVD) or other optical disc storage; magnetic cartridges, magnetic tapes, disk storage or other magnetic storage; and any other media that can be used to store desired information and can be accessed by a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0183] This application has disclosed exemplary embodiments, and although specific terminology has been used, it is used and should be interpreted only in a general illustrative sense and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this application as set forth by the appended claims.

Claims

1. An image processing method, wherein, include: A hierarchical wavelet transform is performed on the image to be processed to obtain multiple sub-images to be processed, each of which includes a high-frequency sub-band image and a low-frequency sub-band image. The high-frequency sub-band image of each layer to be processed is input into the first image processing model to remove moiré patterns, thereby obtaining the high-frequency sub-band repair image of that layer; the low-frequency sub-band image of each layer to be processed is input into the second image processing model to repair color features, thereby obtaining the low-frequency sub-band repair image of that layer. The target image is obtained by performing inverse wavelet transform on the high-frequency subband repair image and the low-frequency subband repair image layer by layer.

2. The method according to claim 1, wherein, The step of inputting the high-frequency subband image of each layer of the sub-image to be processed into the first image processing model for moiré pattern removal to obtain the high-frequency subband restoration image of that layer includes: Based on multiple downsampling and upsampling methods, the high-frequency sub-band map in each layer of the sub-map to be processed is analyzed to obtain the high-frequency feature set corresponding to the high-frequency sub-band map. The high-frequency feature set includes multiple high-frequency features at different scales. The moiré weights of each high-frequency feature in the high-frequency feature set are determined respectively, wherein the moiré weight of each high-frequency feature is negatively correlated with the intensity of the moiré patterns it contains; An initial repair map of the high-frequency sub-band map is generated based on the high-frequency feature set corresponding to the high-frequency sub-band map and the moiré weight of each high-frequency feature in the high-frequency feature set. The initial repair map of the high-frequency subband map is enhanced with detail to obtain the high-frequency subband repair map of that layer.

3. The method according to claim 2, wherein, The method of multiple downsampling and upsampling is used to analyze the high-frequency sub-band map in each layer of the sub-graph to be processed, and to obtain the high-frequency feature set corresponding to the high-frequency sub-band map, including: The high-frequency sub-band map is downsampled multiple times according to the nearest neighbor interpolation algorithm to obtain a first feature set corresponding to the high-frequency sub-band map. The first feature set includes multiple first features with different sampling multiples. The receptive field of each of the first features is expanded to obtain a second feature set corresponding to the high-frequency sub-band map. The second feature set includes multiple second features. Based on the nearest neighbor interpolation algorithm and the downsampling factor corresponding to each of the second features, the second features in the second feature set corresponding to the high-frequency sub-band map are upsampled to obtain the high-frequency feature set corresponding to the high-frequency sub-band map.

4. The method according to claim 2, wherein, The process of enhancing the details of the initial repair map of the high-frequency subband image to obtain the high-frequency subband repair map of that layer includes: Extract the positional attention features from the initial repair map of the high-frequency subband map to obtain the first branch features, and expand the receptive field of the initial repair map of the high-frequency subband map to obtain the second branch features; Connect the first branch feature and the second branch feature to obtain the third feature; The third feature is fused with the initial repair map to obtain the high-frequency subband repair map of this layer.

5. The method according to claim 1, wherein, The step of inputting the low-frequency sub-band image of each layer of the sub-image to be processed into the second image processing model for color feature restoration to obtain the low-frequency sub-band restoration image of that layer includes: Feature extraction is performed on the low-frequency sub-band image, and the receptive field of the extracted low-frequency features is expanded to obtain multiple color features to be repaired. The low-frequency subband repair map of the layer is obtained by multiplying each of the color features to be repaired with its corresponding preset color weight.

6. The method according to any one of claims 1 to 5, wherein, Each layer of the subgraph to be processed includes multiple high-frequency subband graphs and one low-frequency subband graph.

7. The method according to any one of claims 1 to 5, wherein, The process involves performing hierarchical wavelet transforms on the image to be processed to obtain multiple sub-images to be processed, including: The image to be processed is decomposed for the first time according to the preset wavelet transform algorithm to obtain the high-frequency sub-band image and low-frequency sub-band image of the first layer; When the current decomposition count is less than or equal to n, the low-frequency sub-band map of the (i-1)th layer is decomposed according to the preset wavelet transform algorithm to obtain the high-frequency sub-band map and the low-frequency sub-band map of the i-th layer; where i is the current decomposition count, 1≤i≤n, and n is the preset decomposition count.

8. The method according to any one of claims 1 to 5, wherein, The step-by-step inverse wavelet transform of the high-frequency subband restoration map and the low-frequency subband restoration map to obtain the target image includes: Perform inverse wavelet transform on the high-frequency subband restoration map and the low-frequency subband restoration map of the nth layer respectively, and then fuse the obtained transformed high-frequency subband restoration map and the transformed low-frequency subband restoration map of the nth layer to obtain the low-frequency subband restoration map of the (n-1)th layer, where n is the preset number of decompositions; If the level corresponding to the obtained low-frequency subband repair map is greater than 1, perform inverse wavelet transform on the high-frequency subband repair map of the (j+1)th layer to obtain the transformed high-frequency subband repair map of the (j+1)th layer, and fuse the low-frequency subband repair map of the (j+1)th layer and the transformed high-frequency subband repair map of the (j+1)th layer to obtain the low-frequency subband repair map of the jth layer, where j≤n-2. Having obtained the low-frequency subband restoration map of layer 1, the target image is generated based on the low-frequency subband restoration map and the high-frequency subband restoration map of layer 1.

9. The method according to any one of claims 1 to 5, wherein, Before performing hierarchical wavelet transform on the image to be processed to obtain multiple sub-images to be processed, the method further includes: The acquired sample images are subjected to hierarchical wavelet transform to obtain multi-level sample sub-images, each of which includes a high-frequency sample sub-band image and a low-frequency sample sub-band image. The high-frequency sample sub-band image of each layer is input into the first image processing model to remove moiré patterns, thereby obtaining the high-frequency sub-band repair image of that layer; the low-frequency sub-band image of each layer is input into the second image processing model to repair color features, thereby obtaining the low-frequency sub-band repair image of that layer. The high-frequency sample subband restoration image and the low-frequency sample subband restoration image are subjected to inverse wavelet transform layer by layer to obtain the inverse transform result image; Based on the loss information between the inverse transform result image and the preset inverse transform standard image, determine the image loss function; The parameters of the first image processing model and the second image processing model are updated according to the image loss function to obtain the updated image processing model.

10. The method according to claim 9, wherein, The preset number of decompositions is n, where n is an integer greater than 1; the inverse transform standard image includes n preset sub-images; The step of determining the image loss function based on the loss information between the inverse transform result image and the preset inverse transform standard image includes: The inverse transform result image corresponding to the sample sub-image of each layer is compared with the preset sub-image of that layer to obtain the sub-image loss function; The image loss function is determined based on the n sub-image loss functions.

11. An electronic device comprising a memory and a processor; the memory storing a computer program executable by the processor, the computer program, when executed by the processor, implementing the image processing method as described in any one of claims 1 to 10.

12. A computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the image processing method as described in any one of claims 1 to 10.

13. A computer program product comprising a computer program that, when executed by a processor, implements the image processing method as described in any one of claims 1 to 10.

Images