Image processing method and device

Abstract

The invention discloses an image processing method and device, and belongs to the technical field of image processing. The method comprises the following steps: extracting high-frequency features of a first image to obtain at least one high-frequency feature map; extracting low-frequency features of the first image to obtain a first low-frequency feature map; extracting a channel separation feature and a channel mixing feature of a color channel in the first low-frequency feature map; fusing the channel separation features and the channel mixed features based on an illumination-guided attention mechanism to obtain a second low-frequency feature map; and performing detail enhancement processing on the second low-frequency feature map based on the at least one high-frequency feature map to obtain a second image.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to an image processing method and a device thereof. Background Art

[0002] Lighting is an important factor that affects photographic imaging. Images taken under low-light conditions often have problems such as more noise, lower color saturation, and lower clarity, resulting in poor image quality. This not only affects the visual presentation of the image, but also poses challenges to subsequent computer vision tasks such as image-based target detection and tracking. Image illumination adaptation refers to an image processing technology that automatically adjusts the image illumination based on the illumination characteristics of the image itself or in combination with external environmental factors. Its purpose is to optimize the visual effect of the image, making the details in the image clearer and the contrast more appropriate, and not overly restricted by the poor original lighting conditions. Therefore, image illumination adaptation becomes a key step in obtaining good visual perception and supporting downstream visual tasks.

[0003] There are multiple tasks in computer vision for image illumination adaptation, such as image retouching, tone mapping, low-light enhancement, exposure correction, HDR reconstruction, etc. Different tasks use different models to restore or enhance different image degradation problems. At present, these tasks are often based on the mixed features of all colors in the image, and use their own dedicated network architectures for image illumination adaptation. These network architectures usually do not take into account the common properties of different illumination-related tasks and have poor versatility. When applying a dedicated network architecture for a task to other tasks, specialized modifications are often required, otherwise problems such as low efficiency will occur and poor performance will occur. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image processing method and device thereof, which extracts channel separation features of color channels based on the same feature attributes between multiple illumination adaptation tasks, captures illumination differences across channels, and combines the difference features with channel mixing features of color channels, thereby effectively improving the model feature representation capability, thereby being able to demonstrate good performance in multiple illumination adaptation tasks and being suitable for multiple illumination adaptation tasks.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0006] Extracting high-frequency features of the first image to obtain at least one high-frequency feature map;

[0007] Extracting the first low-frequency feature of the image to obtain a first low-frequency feature map;

[0008] Extract channel separation features and channel mixing features of the color channel in the first low-frequency feature map;

[0009] Based on the illumination-guided attention mechanism, the channel separation features and channel mixing features are fused to obtain the second low-frequency feature map;

[0010] The second low-frequency feature map is subjected to detail enhancement processing based on at least one high-frequency feature map to obtain a second image.

[0011] In a second aspect, an embodiment of the present application provides an image processing device, including:

[0012] A high-frequency feature extraction module, used to extract high-frequency features of the first image to obtain at least one high-frequency feature map;

[0013] A low-frequency feature extraction module, used to extract low-frequency features of the first image to obtain a first low-frequency feature map;

[0014] An adaptive illumination module, used for extracting channel separation features and channel mixing features of the color channel in the first low-frequency feature map;

[0015] A feature fusion module is used to fuse channel separation features and channel mixing features based on the illumination-guided attention mechanism to obtain a second low-frequency feature map;

[0016] The iterative detail enhancement module is used to perform detail enhancement processing on the second low-frequency feature map based on at least one high-frequency feature map to obtain a second image.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0018] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0019] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0020] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0021] In an embodiment of the present application, high-frequency features of the first image are extracted to obtain at least one high-frequency feature map; low-frequency features of the first image are extracted to obtain a first low-frequency feature map; channel separation features and channel mixing features of the color channel in the first low-frequency feature map are extracted; channel separation features and channel mixing features are fused based on the illumination-guided attention mechanism to obtain a second low-frequency feature map; and detail enhancement processing is performed on the second low-frequency feature map based on at least one high-frequency feature map to obtain a second image. In an embodiment of the present application, channel separation features of color channels are extracted based on the same feature attributes between multiple illumination adaptive tasks, illumination differences across channels are captured, and difference features are combined with channel mixing features of color channels, effectively improving the model feature representation capability, thereby being able to demonstrate good performance in multiple illumination adaptive tasks and being applicable to multiple illumination adaptive tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flowchart of an image processing method provided by some embodiments of the present application;

[0023] Figure 2 It is a schematic diagram of the layer structure provided by some embodiments of the present application;

[0024] Figure 3 is a flowchart of an image processing method provided by some embodiments of the present application;

[0025] Figure 4 is a flowchart of an image processing method provided by some embodiments of the present application;

[0026] Figure 5 is a flowchart of an image processing method provided by some embodiments of the present application;

[0027] Figure 6 is a flowchart of an image processing method provided by some embodiments of the present application;

[0028] Figure 7 is a flowchart of an image processing method provided by some embodiments of the present application;

[0029] Figure 8 is a schematic diagram of the structure of a feature fusion module provided in some embodiments of the present application;

[0030] Fig. 9 is a schematic diagram of the structure of an iterative detail enhancement module provided in some embodiments of the present application;

[0031] Fig.10 is a flowchart of an image processing method provided by some embodiments of the present application;

[0032] Fig.11 is a schematic diagram of the structure of an image processing device provided in some embodiments of the present application;

[0033] Fig.12 is a block diagram of an electronic device provided by some embodiments of the present application;

[0034] Fig.13 It is a schematic diagram of the structure of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0036] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0037] This application aims to address the problem in existing methods that various tasks for image lighting adaptation lack consideration of common attributes, resulting in poor versatility of the network architectures used for each task. This application systematically analyzes the characteristic attributes between various lighting-related tasks and finds that lighting-related tasks have the same characteristic attributes, such as low frequencies containing brightness and color, and high frequencies mainly expressing textures. These characteristic attributes make it possible to unify the lighting adaptation problem. Based on this, a general network architecture is provided according to the same characteristic attributes between lighting-related tasks, which can achieve optimal performance in multiple tasks and effectively reduce resource consumption in hardware.

[0038] In conjunction with the accompanying drawings, the image processing method, device, electronic device, storage medium and program product based on the network architecture provided by the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0039] The image processing method provided in this application can be applied to image illumination adaptive processing scenarios or illumination adaptive model training scenarios. Figure 1-Figure 10The image processing method provided in the embodiment of the present application is described in detail. It should be noted that the image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, the image processing method provided in the embodiment of the present application is described by taking an image processing device executing the image processing method as an example.

[0040] See also Figure 1 , is a flowchart of an image processing method provided by some embodiments of the present application, such as Figure 1 As shown, the method includes the following steps 110 to 150, which are described in detail below.

[0041] Step 110: Extract high-frequency features of the first image to obtain at least one high-frequency feature map.

[0042] In some embodiments of the present application, in the image illumination adaptive processing scenario, the first image refers to the image to be processed. The first image may be an image selected by a user from an image storage application, wherein the image storage application may be, for example, an album App (Application), a gallery App, a camera App, an image processing software, etc.

[0043] In some other embodiments of the present application, in the image illumination adaptive model training scenario, the first image refers to a sample image used for model training. The first image can be an image downloaded by a user from an image database or the Internet. Among them, the image processing model is a model for realizing image illumination adaptation. Exemplarily, the image processing model includes but is not limited to models corresponding to tasks such as image modification, tone mapping, low light enhancement, exposure correction, and HDR reconstruction.

[0044] The high-frequency features of the first image refer to the parts of the first image where the grayscale or color changes dramatically. In the spectrum of the Fourier transform, the high-frequency components correspond to the information that changes rapidly in the image. From the perspective of the spatial domain, the high-frequency features usually include the edges, details, and textures of the image. Among them, the edge is the boundary between two different areas in the image. For example, in a simple image containing a black square and a white background, the outline of the square is the edge. The grayscale value of the pixel at this outline changes sharply from a low grayscale value to a high grayscale value, and this sharp change represents the high-frequency feature. From a mathematical point of view, the first-order derivative or the second-order derivative at the edge is large, which also indicates the drastic degree of grayscale change. Details refer to relatively small and complex structures in the image. For example, the hair of a person in a portrait photo has more frequent grayscale changes than the smoother areas such as the facial contour of the person. The boundaries of each hair and the interweaving parts between the hair will generate high-frequency information, and these detail information can help better distinguish the content in the image. Texture is composed of many repeated pattern units. Taking a piece of cloth image with texture as an example, fine pattern elements in the texture, such as the transition between patterns on the cloth, will produce high-frequency features. If the texture is regular, these high-frequency information will also show regularity to a certain extent. The more complex the texture, the richer its high-frequency components tend to be. The high-frequency feature map of the first image refers to the feature map containing the high-frequency features of the first image.

[0045] In some embodiments of the present application, the general network architecture provided by the present application includes a high-frequency feature extraction module for extracting high-frequency features of a first image. The first image is input into the high-frequency feature extraction module, and the high-frequency features of at least one scale of the first image are extracted by the high-frequency feature extraction module, thereby obtaining at least one high-frequency feature map corresponding to at least one scale. The resolutions of different high-frequency feature maps are different.

[0046] In some embodiments of the present application, the high-frequency feature extraction module can extract high-frequency features from the first image through a high-pass filter, a wavelet transform method, a difference-based method, etc. A high-pass filter is a linear filter that allows high-frequency signals to pass through while attenuating low-frequency signals. In the field of images, its function is to enhance high-frequency parts such as edges and details in the image, while weakening large smooth areas, i.e., low-frequency parts. Its principle is based on convolution operations, and a specific filter kernel, also known as a convolution kernel, is convolved with each pixel of the image to extract high-frequency features of the image. Wavelet transform is a time-frequency analysis method that can decompose an image into sub-bands of different scales and directions. By selecting appropriate wavelet basis functions such as Daubechies wavelets, Haar wavelets, etc. to decompose the image, a high-frequency feature map containing high-frequency features such as edges and details of the image can be obtained. The difference-based method mainly extracts high-frequency features by calculating the difference between adjacent pixels in the image. In a color image, if the color values ​​of adjacent pixels differ greatly, then the difference will be relatively large, indicating that there are high-frequency features here.

[0047] Step 120: Extract low-frequency features of the first image to obtain a first low-frequency feature map.

[0048] Here, the low-frequency feature of the first image refers to the part of the first image where the grayscale or color changes slowly. In the spectrum after Fourier transformation, the low-frequency component corresponds to the information in the image that changes relatively slowly. From the perspective of the spatial domain, the low-frequency feature usually includes large areas of pure color, smooth transition areas, etc. in the image. Among them, the pure color area is a typical representative of the low-frequency feature. For example, in a photo with a blue sky background, the pixel grayscale value or color value of the sky part is almost the same, and this part of the area belongs to the low-frequency part in the frequency domain. Because there is no obvious grayscale or color change, there is no drastic change corresponding to the high-frequency component. When there is an area in the image that slowly transitions from one color or grayscale to another, this also reflects the low-frequency feature. For example, in a photo of a sunset, the sky gradually transitions from orange to red, and this gradual process is relatively slow. From a mathematical point of view, in this transition area, the first-order derivative and the second-order derivative of the pixel value are relatively small, indicating that the grayscale or color change is not drastic and belongs to low-frequency information. The low-frequency feature map of the first image refers to a feature map containing the low-frequency features of the first image.

[0049] In some embodiments of the present application, the general network architecture provided by the present application includes a low-frequency feature extraction module for extracting low-frequency features of the first image. The first image is input into the low-frequency feature extraction module, and the low-frequency features of the first image are extracted by the low-frequency feature extraction module, thereby obtaining a first low-frequency feature map.

[0050] In some embodiments of the present application, the low-frequency feature extraction module can extract the low-frequency features of the first image by means of a low-pass filter, a Fourier transform method, a wavelet transform method, etc. The main function of the low-pass filter is to allow low-frequency signals to pass through while attenuating high-frequency signals. In image processing, it can be used to smooth images and highlight the low-frequency parts of images, such as large areas of color areas and slowly changing areas. Its working principle is to perform a convolution operation with an image through a specific filter kernel to obtain the low-frequency features of the image. The Fourier transform can convert an image from a spatial domain to a frequency domain. In the frequency domain, low-frequency components are concentrated in the central area of ​​the spectrum, and high-frequency components are distributed in the edge area of ​​the spectrum. By operating the spectrum in the frequency domain, the low-frequency features of the image can be extracted. The wavelet transform can decompose the image into sub-bands of different scales and directions. After a two-dimensional wavelet decomposition, a low-frequency sub-band (LL) and three high-frequency sub-bands (LH, HL, HH) are obtained. This low-frequency sub-band (LL) contains the low-frequency features of the image at this scale.

[0051] Step 130: Extract channel separation features and channel mixing features of the color channel in the first low-frequency feature map.

[0052] In some embodiments of the present application, the first image in the above step 110 is a color image including at least two color channels. For example, the first image is an RGB image including three color channels of red, green and blue. Based on this, the first low-frequency feature map also includes at least two color channels.

[0053] Here, the channel separation feature of the color channel refers to each color channel, such as the red, green, and blue channels of the RGB image, and the characteristics of each channel are analyzed separately. Each channel has its own independent information, which can more clearly show the characteristics of the image in different color dimensions after separation, including brightness distribution, color level, texture details, etc. Different color channels in the image have different light properties. Therefore, by extracting the characteristics of different color channels, that is, channel separation features, the illumination differences of different color channels can be captured, thereby improving the performance of image illumination adaptation.

[0054] The channel mixing feature of the color channel refers to the characteristics of the interaction and fusion between different color channels, such as the red, green, and blue channels in the RGB image. It reflects the characteristics of the image color information under the comprehensive effect of multiple channels. The channel mixing feature can be used to extract more expressive image features, adjust the image color style, or enhance certain specific visual effects of the image.

[0055] In some embodiments of the present application, the general network architecture provided by the present application includes an adaptive illumination module, and the adaptive illumination module includes two submodules, namely, a separation feature extraction submodule for extracting channel separation features and a mixed feature extraction submodule for extracting channel mixing features. After obtaining the first low-frequency feature map, the first low-frequency feature map is input into the separation feature extraction submodule and the mixed feature extraction submodule respectively, and the separation feature extraction submodule extracts the independent features of each color channel in the first low-frequency feature map to obtain the channel separation features of each color channel, and the mixed feature extraction submodule extracts the mixed features of all color channels in the first low-frequency feature image to obtain the channel mixing features.

[0056] Step 140. Based on the illumination-guided attention mechanism, the channel separation features and the channel mixing features are fused to obtain a second low-frequency feature map.

[0057] In some embodiments of the present application, the illumination-guided attention mechanism is used to guide the image processing device to pay more attention to illumination features that are important for illumination adaptation, thereby highlighting key information in the image.

[0058] In some embodiments of the present application, the general network architecture provided by the present application includes a feature fusion module for performing feature fusion on channel separation features and channel mixing features. The fused channel separation features and channel mixing features are input into the feature fusion module, and the channel separation features and channel mixing features are feature fused through the illumination-guided attention mechanism in the feature fusion module, thereby introducing the channel separation features into the channel mixing features, and obtaining a second low-frequency feature map after feature fusion.

[0059] Using the illumination-guided attention mechanism for feature fusion can guide the image processing device to focus on illumination features related to illumination adaptation. In this way, the image processing device can achieve color consistency and color balance in multiple illumination-related tasks, effectively improving the model feature representation capability.

[0060] Step 150: Perform detail enhancement processing on the second low-frequency feature map based on at least one high-frequency feature map to obtain a second image.

[0061] In some embodiments of the present application, the high-frequency feature map includes high-frequency features of the first image, and detail enhancement of the second low-frequency feature map is performed based on the high-frequency feature map, so that the high-frequency features of the second low-frequency feature map can be enhanced. By increasing the high-frequency features of the second low-frequency feature map, the second low-frequency feature map can be made to look clearer, and the second low-frequency feature map after detail enhancement is determined as the second image obtained after illumination adaptive processing is performed on the first image.

[0062] An image processing method provided in an embodiment of the present application extracts high-frequency features of a first image to obtain at least one high-frequency feature map; extracts low-frequency features of the first image to obtain a first low-frequency feature map; extracts channel separation features and channel mixing features of color channels in the first low-frequency feature map; fuses the channel separation features and channel mixing features based on an illumination-guided attention mechanism to obtain a second low-frequency feature map; performs detail enhancement processing on the second low-frequency feature map based on at least one high-frequency feature map to obtain a second image. An embodiment of the present application extracts channel separation features of color channels based on the same feature attributes between multiple illumination adaptive tasks, captures illumination differences across channels, and combines difference features with channel mixing features of color channels, effectively improving the model feature representation capability, thereby being able to demonstrate good performance in multiple illumination adaptive tasks and being applicable to multiple illumination adaptive tasks.

[0063] In some embodiments, high-frequency features and low-frequency features of the first image can be extracted through a pyramid framework. The pyramid framework is mainly composed of a multi-scale representation of the image and conversion operations between different scales. It is based on the original image and generates a pyramid structure containing multiple scale images through a series of processing steps, and can transfer and process information between different levels. This framework can be regarded as a layered image processing system, each layer corresponds to a different resolution image, from the original high-resolution image at the bottom to the low-resolution image at the top.

[0064] In some embodiments of the present application, in the above step 110, the first image can be input into a multi-layer structured learnable difference pyramid to obtain at least one high-frequency feature map output by at least one layer in the learnable difference pyramid.

[0065] Here, each layer of the learnable difference pyramid can perform high-frequency feature extraction based on the difference method. Thus, after the first image is input into the learnable difference pyramid with a multi-layer structure, the high-frequency feature extraction of the first image can be performed in sequence through the multi-layer structure in the learnable difference pyramid, thereby realizing multiple high-frequency feature extractions of the first image. After the high-frequency feature extraction of each layer, a high-frequency feature map can be output. In this way, multiple high-frequency feature maps can be obtained through the multi-layer structure of the learnable difference pyramid.

[0066] In some embodiments of the present application, the learnable difference pyramid consists of multiple layers such as Figure 2 The layer structure 200 shown is composed of Figure 2, each layer structure 200 includes a first convolution module 210, a difference module 220, a splicing module 230 and a first convolution residual module 240. Among them, the first convolution module 210 includes three convolution layers. The first convolution module 210 is used to extract features of the image input to the first convolution module 210 through the convolution layer therein. The difference module 220 is used to subtract adjacent features extracted by adjacent convolution layers in the first convolution module 210 to obtain differential features. The splicing module 230 is used to splice the differential features output by the difference module 220 to obtain splicing features. The first convolution residual module 240 is a convolution layer-residual block-convolution layer structure. The first convolution residual module 240 is used to further extract features from the splicing features. Based on this structure, see Figure 3 In the above step 110, at least one high-frequency feature map can be obtained by using the learnable difference pyramid using the following steps 310 to 340.

[0067] Step 310. For each layer in the learnable difference pyramid, the features of the image input to the layer are extracted in sequence through the three convolutional layers in the first convolutional module.

[0068] Step 320. Through the difference module, adjacent features extracted by adjacent convolution layers in the first convolution module are subtracted to obtain differential features.

[0069] Step 330. Use the splicing module to splice the differential features to obtain high-frequency splicing features.

[0070] Step 340. Perform feature extraction on the high-frequency splicing features through the first convolution residual module to obtain a high-frequency feature map corresponding to the layer.

[0071] In addition, participating Figure 2 The learnable differential pyramid also includes a maximum pooling layer 250, which is used to perform maximum pooling on the feature map output by the last convolution layer in the first convolution module 210, so as to obtain input data for the next layer.

[0072] For example, see Figure 2 , the image data input to this layer is F i , F i After being processed by the first convolution module 210, the difference module 220, the splicing module 230 and the first convolution residual module 240 in sequence, the high-frequency features output by this layer are obtained. Figure X i , F i After being processed by the first convolution module 210 and the maximum pooling layer 250, the input image F of the next layer is obtained. i+1 .

[0073] By extracting high-frequency features of images through a learnable difference pyramid, high-frequency features of images can be extracted at different scales. This multi-scale representation method makes it possible to capture detailed information of different sizes and frequencies in the image at the same time, thereby more comprehensively describing the feature information of the image. Moreover, compared with some complex frequency domain transformation methods such as Fourier transform, the construction and difference calculation of the learnable difference pyramid are mainly based on simple convolution and subtraction operations, with low computational complexity. When processing large-scale image data, it can quickly extract high-frequency features while ensuring a certain accuracy, improve the efficiency of image processing, and is suitable for application scenarios with high real-time requirements. In addition, since high-frequency features mainly reflect the intensity of grayscale changes in images rather than absolute grayscale values, they are robust to illumination changes to a certain extent. In other words, even if the illumination conditions of the image change, the relative position and intensity changes of high-frequency features such as edges and textures are relatively small. The high-frequency features extracted by the learnable difference pyramid can remain relatively stable under different illumination conditions, thereby improving the reliability and versatility of image processing algorithms based on these features.

[0074] In some embodiments of the present application, in the above step 120, the low-frequency features of the first image are extracted using a Gaussian pyramid, thereby obtaining a low-frequency and low-resolution first low-frequency feature map. The Gaussian pyramid is constructed by performing multiple downsampling and smoothing operations on the original image. In this process, the low-frequency features are retained and highlighted. Downsampling refers to reducing the number of pixels in an image, usually by deleting even rows and even columns. The smoothing operation is to convolve the image using a Gaussian filter. The function of the Gaussian filter is to update the value of the pixel for each pixel in the image according to the weighted average of the surrounding pixels. The Gaussian filter is a low-pass filter that allows low-frequency signals to pass through while attenuating high-frequency signals. When the image is smoothed using a Gaussian filter, the high-frequency part of the image is blurred, while the low-frequency part is retained and further integrated in the subsequent downsampling process, so that the image in each layer of the Gaussian pyramid gradually focuses more on low-frequency features.

[0075] By extracting the low-frequency features of the first image through the Gaussian pyramid, the low-frequency structural information can be well retained while removing the high-frequency details, so that the extracted low-frequency features can accurately reflect the overall framework of the first image, which is helpful for the subsequent understanding and analysis of the first image. In addition, the Gaussian pyramid is composed of a series of image layers with different resolutions, and each layer corresponds to low-frequency features of different scales. This multi-scale representation method makes it possible to analyze the first image at different levels of detail, gradually deepening from the rough overall structure to the finer local features. Moreover, compared with directly processing the high-frequency and low-frequency information of the original image, the low-frequency features extracted by the Gaussian pyramid can be analyzed on the basis of reducing the image resolution, which greatly reduces the amount of data and calculation. In the application scenarios of processing large-scale image data or requiring high real-time performance, this reduction in computational complexity is particularly important, which can improve the operating speed and response performance of the system. In addition, since the low-frequency features are relatively stable and less affected by factors such as illumination and contrast, the low-frequency features extracted based on the Gaussian pyramid have certain consistency and anti-interference ability under different image conditions. This allows low-frequency features to provide a more reliable basis for subsequent image processing and analysis when faced with complex and changing image environments, reducing feature fluctuations and misjudgments caused by changes in environmental factors.

[0076] In some embodiments, different color channels reflect different feature differences, and these differences are expressed differently in the spatial domain and the frequency domain. Based on this, in the above step 130, the separation feature extraction submodule can extract the features of each color channel in the first low-frequency feature map in the frequency domain and the spatial domain, respectively, to obtain the frequency domain features and spatial domain features of each color channel, and then determine the frequency domain features and spatial domain features of at least two color channels in the first low-frequency feature map as the channel separation features of the color channels in the first low-frequency feature map.

[0077] In some embodiments of the present application, see Figure 4 In the above step 130, the frequency domain features and spatial domain features of each color channel in the first low-frequency feature map can be extracted through the following steps 410-450.

[0078] Step 410: Convert the first low-frequency feature map into the frequency domain according to each color channel to obtain a frequency domain image corresponding to each color channel.

[0079] In some embodiments of the present application, the separation feature extraction submodule uses a two-dimensional Fourier transform to transform the first low-frequency feature map into the frequency domain for each color channel, thereby obtaining a frequency domain image S of each color channel. i =R i +j*I i , where S i Represents the frequency domain image corresponding to the i-th color channel, Ri and I i Represent the frequency domain image S i The real and imaginary parts of . The value of i is related to the number of color channels contained in the first low-frequency feature image. For example, if the first low-frequency feature image is an RGB image, the R channel is used as the first channel, the G channel is used as the second channel, and the B channel is used as the third channel. The value of i is 1, 2, 3, S1 represents the frequency domain image corresponding to the R channel, S2 represents the frequency domain image corresponding to the G channel, and S3 represents the frequency domain image corresponding to the B channel.

[0080] Step 420: Extract the features of the frequency domain image corresponding to each color channel respectively to obtain the frequency domain features of each color channel.

[0081] The frequency domain features of color channels refer to the features displayed after converting the color channels of an image, such as RGB channels, from the spatial domain to the frequency domain. Frequency domain features can reflect the distribution of color information on frequency components, including the intensity of different frequencies, the range of frequencies, and the relationship between frequencies.

[0082] In some embodiments of the present application, the separation feature extraction submodule uses 1×1 convolution to act on the real and imaginary parts of the frequency domain image corresponding to each color channel respectively to extract the features of each color channel in the frequency domain, thereby obtaining the frequency domain features of each color channel.

[0083] Step 430: Convert the frequency domain image corresponding to each color channel into the spatial domain to obtain the spatial domain image corresponding to each color channel.

[0084] In some embodiments of the present application, the separation feature extraction submodule uses an inverse two-dimensional Fourier transform to convert the frequency domain image corresponding to each color channel into the spatial domain, thereby obtaining a spatial domain image corresponding to each color channel.

[0085] Step 440: Stitching the spatial domain images corresponding to at least two color channels to obtain a spatial domain stitched image.

[0086] The spatial domain features of color channels refer to the features of each color channel in the spatial layout of the image, such as pixel distribution, shape and position of color regions, and spatial variation of colors. These features describe the characteristics of color channels from the spatial position of pixels in the image.

[0087] In some embodiments of the present application, before extracting the spatial domain features of each color channel, the separation feature extraction submodule stitches the spatial domain images of multiple color channels to obtain a complete stitched image, and then extracts the spatial domain features of each color channel based on the spatial layout of the stitched image.

[0088] Step 450: Extract the features of the spatial domain stitched image to obtain the spatial domain features of each color channel.

[0089] In some embodiments of the present application, the separation feature extraction submodule includes a channel attention module, and the spatial domain stitched image is input into the channel attention module. The channel attention module extracts the features of each color channel in the spatial domain, thereby obtaining the spatial domain features of each color channel.

[0090] The channel separation features extracted in the above manner, including the features of each color channel in the frequency domain and the spatial domain, can characterize the interaction between different color channels in the spatial domain and the frequency domain, and further enhance the model feature characterization capability.

[0091] In some embodiments, see Figure 5 In the above step 130, the channel mixing features of the color channels in the first low-frequency feature map can be extracted through the following steps 510-530.

[0092] Step 510: Perform wavelet transform on the first low-frequency feature map to obtain four sub-band images.

[0093] In some embodiments of the present application, the hybrid feature extraction submodule performs wavelet transform on the first low-frequency feature map through a wavelet function to obtain four sub-band images of different scales. Exemplarily, the hybrid feature extraction submodule performs feature extraction on the first low-frequency feature map through a wavelet function Haar to obtain a low-frequency sub-band image (LL) and three high-frequency sub-band images (LH, HL, HH).

[0094] Step 520: splice the four sub-band images to obtain a feature image.

[0095] In some embodiments of the present application, the hybrid feature extraction submodule obtains a feature image by splicing four sub-band images obtained by wavelet transform.

[0096] Step 530: Perform feature extraction on the feature image to obtain channel mixing features of the color channels in the first low-frequency feature image.

[0097] In some embodiments of the present application, the mixed feature extraction submodule inputs the feature image into a 3×3 convolutional layer, captures the mixed features of multiple color channels through the 3×3 convolutional layer, and thus obtains the channel mixed features of the color channels.

[0098] Wavelet transform can decompose images at multiple scales and analyze the channel mixing characteristics of images at different frequencies and spatial resolutions, thereby capturing a variety of information from the overall to the details, and thus comprehensively describing the channel mixing characteristics.

[0099] In some embodiments, see Figure 6 In the above step 140, the channel separation features and the channel mixing features are fused through the following steps 610 to 630.

[0100] Step 610. Input the channel mixing features and the channel separation features into the second convolution module respectively to obtain attention parameters, which include a query matrix, a key matrix and a value matrix.

[0101] In some embodiments of the present application, the feature fusion module includes a second convolution module, and the channel mixing feature and the channel separation feature are respectively input into the second convolution module, and the channel mixing feature and the channel separation feature are respectively extracted by the second convolution module, so as to obtain the attention parameters used in the illumination guided attention mechanism. The attention parameters include the query matrix Q, the key matrix K and the value matrix V.

[0102] In some embodiments of the present application, the second convolution module is composed of 1×1 convolution and 3×3 convolution. The channel mixing feature is input into the second convolution module, and the feature is extracted by 1×1 convolution and 3×3 convolution in the second convolution module in sequence to obtain the key matrix K and the value matrix V. The channel separation feature is input into the second convolution module, and the feature is extracted by 1×1 convolution and 3×3 convolution in the second convolution module in sequence to obtain the query matrix Q.

[0103] Step 620. Calculate the attention weight matrix based on the query matrix and the key matrix.

[0104] Step 630: Multiply the attention weight matrix by the value matrix to obtain a second low-frequency feature map.

[0105] In some embodiments of the present application, the feature fusion module calculates the attention weight matrix and the second low-frequency feature map based on the illumination-guided attention mechanism formula shown in the following formula (1).

[0106]

[0107] In formula (1), Attention(Q, K, V) represents the feature map after feature fusion, that is, the second low-frequency feature map, softmax() represents the normalization function, and d K represents the dimension of the key matrix K, and τ represents the scaling of the image. Used to calculate the attention weight matrix. Based on this, substitute the key matrix K and query matrix Q in the attention parameters into The attention weight matrix is ​​obtained, and the obtained attention weight matrix is ​​multiplied by the value matrix V in the attention parameter to obtain the feature fusion feature map, that is, the second low-frequency feature map.

[0108] Using the attention mechanism for feature fusion can assign different weights to different features. The attention mechanism can dynamically adjust the degree of attention to each feature according to the specific content of the input image, so that the model can better adapt to various image scenes and changes. This adaptability helps to improve the generalization ability of the model, so that it can also perform better when facing unseen image data.

[0109] In some embodiments, see Figure 7 When the channel separation feature and the channel mixing feature are fused, before the above step 610, the following steps 710 to 720 may be performed first.

[0110] Step 710: Perform feature extraction on the channel separation features through at least one group convolution to obtain independent features of each channel.

[0111] In some embodiments of the present application, see Figure 8 , is a schematic diagram of the structure of the feature fusion module 1140, such as Figure 8 As shown, the feature fusion module 1140 includes several group convolutions 810. Group convolution 810 is a convolution operation mode in a convolutional neural network (CNN). In traditional convolution, the convolution kernel slides on the entire input feature map to perform convolution calculations. Group convolution 810 divides the input feature map and the convolution kernel into several groups, and the convolution operation is only performed in the corresponding group. For example, if the input feature map is divided into G groups and the convolution kernel is also divided into G groups, then the convolution kernel of the mth group is only convolved with the input feature map of the mth group, where m=1,...G. By separating the channel feature F cs Input group convolution 810, channel separation feature F can be separated by group convolution 810 cs The independent features of each color channel are obtained by grouping. In this way, the features between color channels can be kept independent of each other.

[0112] Step 720: Extract channel mixed features through a state space model and a multi-layer perceptron to obtain global features of at least two channels.

[0113] In some embodiments of the present application, Figure 8As shown, the feature fusion module 1140 includes a plurality of combined structures composed of a state space model 820-a multilayer perceptron 830-an illumination-guided attention mechanism 840-a multilayer perceptron 850, and the number of the combined structures corresponds to the number of group convolutions. Among them, the state space model 820 is a mathematical model of a dynamic system, which is used to describe the evolution of the state of the system over time or other variables. The state space model 820 is mainly composed of two parts: a state equation and an observation equation. The state space model 820 can achieve the perception and utilization of global features by hierarchically representing the channel mixing features or using a recursive structure, so as to better maintain the overall structure and semantic consistency of the image, and avoid the situation where the local is good but the overall is not coordinated. The multilayer perceptron 830 and the multilayer perceptron 850 are a feedforward artificial neural network, which consists of a plurality of neurons, which are arranged in layers, including an input layer, one or more hidden layers and an output layer. The input layer receives external data, and the hidden layer performs a nonlinear transformation on the input data. Each neuron calculates the weighted sum of its input and processes it through an activation function. The output layer produces the final output result. The multi-layer perceptron 830 can extract various features of the image by performing weighted combination and nonlinear transformation on the pixels of the damaged image received by the input layer through the neurons in the hidden layer. It can be seen that the channel mixing feature F can be extracted by the state space model 820-multi-layer perceptron 830. cm Based on this, before the channel mixing feature is input into the second convolution module in the feature illumination guided attention mechanism 840, the channel mixing feature is first input into the state space model 820-multilayer perceptron 830 in sequence for feature extraction to obtain the global feature, which helps to better understand the content and structure of the image. Accordingly, in the above step 610, the channel mixing feature and the channel separation feature are respectively input into the second convolution module to obtain the attention parameters, including:

[0114] Input the independent features of each channel output by the group convolution 810 into the second convolution module to obtain a query matrix;

[0115] The global features of at least two channels obtained by extracting channel mixing features through the state space model 820-multilayer perceptron 830 are input into the second convolution module to obtain a key matrix and a value matrix.

[0116] In addition, in some embodiments of the present application, after the feature fusion of the channel separation feature and the channel fusion feature is completed by the illumination-guided attention mechanism 840, the output data of the illumination-guided attention mechanism 840 is input to Figure 8 The multi-layer perceptron 850 shown performs feature extraction to restore more details, and the output of the multi-layer perceptron 850 is determined as the second low-frequency feature map.

[0117] In some embodiments, the general network architecture provided by the embodiments of the present application includes an iterative detail enhancement module. In the above step 150, the second low-frequency feature map with low frequency and low resolution is sent to the iterative detail enhancement module to gradually restore the image resolution and enhance the details, thereby obtaining a second image.

[0118] In some embodiments of the present application, see Fig. 9 , the iterative detail enhancement module 1150 includes an upsampling module 910, a splicing module 920, a second convolution residual module 930, a multiplication module 940 and an addition module 950. Among them, the upsampling module 910 is used to upsample the second low-frequency feature map to obtain an upsampled image, the splicing module 920 is used to splice the upsampled image output by the upsampling module 910 with an image of corresponding resolution in at least one high-frequency feature map to obtain a spliced ​​image, the second convolution residual module 930 adopts a convolution-residual block-residual block-convolution structure, and is used to extract features from the spliced ​​image output by the splicing module 920 to obtain an image mask, the multiplication module 940 is used to multiply the image mask with an image of corresponding resolution in at least one high-frequency feature map as the upsampled image, and the addition module 950 is used to add the image output by the multiplication module 940 to the upsampled image to obtain a new second low-frequency feature map, and the iterative detail enhancement module can iteratively process the new second low-frequency feature map to finally obtain a second image that meets the requirements. Based on this, see Fig.10 In the above step 150, the second low-frequency feature map can be subjected to detail enhancement processing through the following steps 1010 to 1050.

[0119] Step 1010: Upsample the second low-frequency feature map to obtain an upsampled image.

[0120] Step 1020. Splice the upsampled image with a first high-frequency feature map in at least one high-frequency feature map to obtain a spliced ​​image, wherein the first high-frequency feature map is an image with a resolution consistent with that of the upsampled image.

[0121] Step 1030. Perform feature extraction on the spliced ​​image through the second convolution residual module to obtain an image mask.

[0122] Step 1040: Multiply the image mask by the first high-frequency feature map and then add the resultant result to the second low-frequency feature map to obtain a new second low-frequency feature map.

[0123] Step 1050. Return to execute the above step 1010 until the number of iterations is consistent with the number of images of at least one high-frequency feature map, and determine the final second low-frequency feature map as the second image.

[0124] Exemplarily, the above step 110 extracts high-frequency features from the first image to obtain three high-frequency feature maps: Among them, X0 has the highest resolution, and X2 has the lowest resolution. The second low-frequency feature map Y3 is obtained through the above steps 1010 to 1040. When the second low-frequency feature map Y3 is enhanced in detail, the second low-frequency feature map Y3 is first upsampled to obtain an upsampled image, and then the upsampled image is compared with the high-frequency feature maps of the corresponding resolution in the three high-frequency feature maps. Figure X 2 to obtain a stitched image, and then send the stitched image to convolution-residual block-residual block-convolution for feature extraction and predict an image mask to enhance high-frequency details. Figure X 2 and then added to the upsampled second low-frequency feature map Y3 to obtain the new second low-frequency feature map Y2 of the output of this scale. Figure X 1 Enhance the details of the new second low-frequency feature map Y2 to obtain a new second low-frequency feature map Y1, and then based on the high-frequency feature Figure X 0 performs detail enhancement on the new second low-frequency feature map Y1 to obtain a new second low-frequency feature map Y0. After the above three iterations, the final highest resolution Y0 is determined as the second image for output.

[0125] High-frequency feature maps of different resolutions contain detailed information from micro to macro. High-resolution high-frequency feature maps can capture tiny details in the image, while low-resolution high-frequency feature maps focus on reflecting the general outline and large-scale edges of the object. By using high-frequency feature maps of different resolutions to enhance the details of the second low-frequency feature map, the details of the image can be comprehensively improved, so that the image can show richer information at both the micro and macro levels. Moreover, when observing objects, the human eye naturally focuses on details of different scales. Images enhanced by high-frequency feature maps of different resolutions can better simulate this visual characteristic, thereby providing a more natural and comfortable visual experience.

[0126] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, an image processing device executing the image processing method is taken as an example to illustrate the image processing device provided in the embodiment of the present application.

[0127] See also Fig.11 , is a schematic diagram of an image processing device provided in some embodiments of the present application, such as Fig.11 As shown, the device 1100 includes the following modules:

[0128] A high-frequency feature extraction module 1110 is used to extract high-frequency features of the first image to obtain at least one high-frequency feature map;

[0129] A low-frequency feature extraction module 1120 is used to extract low-frequency features of the first image to obtain a first low-frequency feature map;

[0130] An adaptive illumination module 1130 is used to extract channel separation features and channel mixing features of the color channel in the first low-frequency feature map;

[0131] A feature fusion module 1140 is used to fuse the channel separation feature and the channel mixing feature based on the illumination guided attention mechanism to obtain a second low-frequency feature map;

[0132] The iterative detail enhancement module 1150 is used to perform detail enhancement processing on the second low-frequency feature map based on at least one high-frequency feature map to obtain a second image.

[0133] The image processing device provided in the embodiment of the present application extracts high-frequency features of the first image to obtain at least one high-frequency feature map; extracts low-frequency features of the first image to obtain a first low-frequency feature map; extracts channel separation features and channel mixing features of the color channel in the first low-frequency feature map; fuses the channel separation features and channel mixing features based on the illumination-guided attention mechanism to obtain a second low-frequency feature map; performs detail enhancement processing on the second low-frequency feature map based on at least one high-frequency feature map to obtain a second image. The embodiment of the present application extracts channel separation features of the color channel based on the same feature attributes between multiple illumination adaptive tasks, captures illumination differences across channels, and combines the difference features with the channel mixing features of the color channel, effectively improving the model feature representation capability, thereby being able to demonstrate good performance in multiple illumination adaptive tasks and being suitable for multiple illumination adaptive tasks.

[0134] In some embodiments, the high frequency feature extraction module 1110 is specifically used to:

[0135] The first image is input into a multi-layer learnable difference pyramid to obtain at least one high-frequency feature map output by at least one layer in the learnable difference pyramid.

[0136] In some embodiments, each layer of the learnable difference pyramid includes a first convolution module, a difference module, a concatenation module, and a first convolution residual module, and the first convolution module includes three convolution layers;

[0137] The high frequency feature extraction module 1110 is specifically used for:

[0138] For each layer in the learnable difference pyramid, the features of the image input to this layer are extracted sequentially through the three convolutional layers in the first convolutional module;

[0139] Through the difference module, adjacent features extracted by adjacent convolution layers in the first convolution module are subtracted to obtain differential features;

[0140] Through the splicing module, the differential features are spliced ​​to obtain high-frequency splicing features;

[0141] The high-frequency splicing features are extracted through the first convolution residual module to obtain the high-frequency feature map corresponding to the layer.

[0142] In some embodiments, the first low-frequency feature map includes at least two color channels, and the adaptive illumination module 1130 includes a separation feature extraction submodule, which is specifically used to:

[0143] Convert the first low-frequency feature map to the frequency domain according to each color channel to obtain a frequency domain image corresponding to each color channel;

[0144] Extract the features of the frequency domain image corresponding to each color channel respectively to obtain the frequency domain features of each color channel;

[0145] The frequency domain image corresponding to each color channel is converted into the spatial domain to obtain the spatial domain image corresponding to each color channel;

[0146] splicing the spatial domain images corresponding to at least two color channels to obtain a spatial domain spliced ​​image;

[0147] Extract the features of the spliced ​​image in the spatial domain to obtain the spatial domain features of each color channel;

[0148] The frequency domain features and the space domain features of at least two color channels are determined as channel separation features of the color channels in the first low-frequency feature map.

[0149] In some embodiments, the adaptive illumination module 1130 includes a hybrid feature extraction submodule, which is specifically used to:

[0150] Performing wavelet transform on the first low-frequency feature map to obtain four sub-band images;

[0151] The four sub-band images are spliced ​​to obtain a feature image;

[0152] Feature extraction is performed on the feature image to obtain a channel mixing feature of the color channel in the first low-frequency feature image.

[0153] In some embodiments, the feature fusion module 1140 is specifically configured to:

[0154] The channel mixing features and the channel separation features are respectively input into the second convolution module to obtain attention parameters, which include a query matrix, a key matrix, and a value matrix;

[0155] Based on the query matrix and the key matrix, calculate the attention weight matrix;

[0156] Multiply the attention weight matrix by the value matrix to obtain the second low-frequency feature map.

[0157] In some embodiments, the feature fusion module 1140 is further configured to:

[0158] The channel mixture features and the channel separation features are respectively input into the second convolution module. Before obtaining the attention parameters, the channel separation features are subjected to feature extraction through at least one group convolution to obtain independent features of each channel.

[0159] The channel mixed features are extracted through the state space model and the multi-layer perceptron module to obtain the global features of at least two channels;

[0160] The feature fusion module 1140 is specifically used for:

[0161] Input the independent features of each channel into the second convolution module to obtain the query matrix;

[0162] The global features of at least two channels are input into the second convolution module to obtain a key matrix and a value matrix.

[0163] In some embodiments, the iterative detail enhancement module 1150 is specifically configured to:

[0164] Upsampling the second low-frequency feature map to obtain an upsampled image;

[0165] Splicing the upsampled image with a first high-frequency feature map in at least one high-frequency feature map to obtain a spliced ​​image, wherein the first high-frequency feature map has a resolution consistent with that of the upsampled image;

[0166] Perform feature extraction on the spliced ​​image through the second convolution residual module to obtain an image mask;

[0167] The image mask is multiplied by the first high-frequency feature map and then added to the upsampled image to obtain a new second low-frequency feature map;

[0168] Return to the step of upsampling the second low-frequency feature map to obtain an upsampled image until the number of iterations is consistent with the number of images of at least one high-frequency feature map, and determine the final second low-frequency feature map as the second image.

[0169] The image processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0170] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0171] The image processing device provided in the embodiment of the present application can achieve Figures 1 to 10 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0172] Alternatively, if Fig.12 As shown, an embodiment of the present application further provides an electronic device 1200, including a processor 1201 and a memory 1202, wherein the memory 1202 stores programs or instructions that can be executed on the processor 1201, and when the program or instructions are executed by the processor 1201, the various steps of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0173] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0174] Fig.13 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0175] The electronic device 1300 includes but is not limited to: a radio frequency unit 1301, a network module 1302, an audio output unit 1303, an input unit 1304, a sensor 1305, a display unit 1306, a user input unit 1307, an interface unit 1308, a memory 1309, and a processor 1310 and other components.

[0176] Those skilled in the art will appreciate that the electronic device 1300 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1310 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.13 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0177] The processor 1310 is used to extract high-frequency features of the first image to obtain at least one high-frequency feature map;

[0178] The processor 1310 is further configured to extract low-frequency features of the first image to obtain a first low-frequency feature map;

[0179] The processor 1310 is further configured to extract channel separation features and channel mixing features of the color channel in the first low-frequency feature map;

[0180] The processor 1310 is further configured to fuse the channel separation feature and the channel mixing feature based on the illumination guided attention mechanism to obtain a second low-frequency feature map;

[0181] The processor 1310 is further configured to perform detail enhancement processing on the second low-frequency feature map based on at least one high-frequency feature map to obtain a second image.

[0182] The electronic device provided in the embodiment of the present application extracts high-frequency features of a first image to obtain at least one high-frequency feature map; extracts low-frequency features of the first image to obtain a first low-frequency feature map; extracts channel separation features and channel mixing features of color channels in the first low-frequency feature map; fuses the channel separation features and channel mixing features based on an illumination-guided attention mechanism to obtain a second low-frequency feature map; performs detail enhancement processing on the second low-frequency feature map based on at least one high-frequency feature map to obtain a second image. The embodiment of the present application extracts channel separation features of color channels based on the same feature attributes between multiple illumination adaptive tasks, captures illumination differences across channels, and combines difference features with channel mixing features of color channels, effectively improving the model feature representation capability, thereby being able to demonstrate good performance in multiple illumination adaptive tasks and being applicable to multiple illumination adaptive tasks.

[0183] In some embodiments, the processor 1310 is specifically configured to:

[0184] The first image is input into a multi-layer learnable difference pyramid to obtain at least one high-frequency feature map output by at least one layer in the learnable difference pyramid.

[0185] In some embodiments, each layer of the learnable difference pyramid includes a first convolution module, a difference module, a concatenation module, and a first convolution residual module, and the first convolution module includes three convolution layers;

[0186] The processor 1310 is specifically configured to:

[0187] For each layer in the learnable difference pyramid, the features of the image input to this layer are extracted sequentially through the three convolutional layers in the first convolutional module;

[0188] Through the difference module, adjacent features extracted by adjacent convolution layers in the first convolution module are subtracted to obtain differential features;

[0189] Through the splicing module, the differential features are spliced ​​to obtain high-frequency splicing features;

[0190] The high-frequency splicing features are extracted through the first convolution residual module to obtain the high-frequency feature map corresponding to the layer.

[0191] In some embodiments, the first low-frequency feature map includes at least two color channels, and the processor 1310 is specifically configured to:

[0192] Convert the first low-frequency feature map to the frequency domain according to each color channel to obtain a frequency domain image corresponding to each color channel;

[0193] Extract the features of the frequency domain image corresponding to each color channel respectively to obtain the frequency domain features of each color channel;

[0194] The frequency domain image corresponding to each color channel is converted into the spatial domain to obtain the spatial domain image corresponding to each color channel;

[0195] splicing the spatial domain images corresponding to at least two color channels to obtain a spatial domain spliced ​​image;

[0196] Extract the features of the spliced ​​image in the spatial domain to obtain the spatial domain features of each color channel;

[0197] The frequency domain features and the space domain features of at least two color channels are determined as channel separation features of the color channels in the first low-frequency feature map.

[0198] In some embodiments, the processor 1310 is specifically configured to:

[0199] Performing wavelet transform on the first low-frequency feature map to obtain four sub-band images;

[0200] The four sub-band images are spliced ​​to obtain a feature image;

[0201] Feature extraction is performed on the feature image to obtain a channel mixing feature of the color channel in the first low-frequency feature image.

[0202] In some embodiments, the processor 1310 is specifically configured to:

[0203] The channel mixing features and the channel separation features are respectively input into the second convolution module to obtain attention parameters, which include a query matrix, a key matrix, and a value matrix;

[0204] Based on the query matrix and the key matrix, calculate the attention weight matrix;

[0205] Multiply the attention weight matrix by the value matrix to obtain the second low-frequency feature map.

[0206] In some embodiments, the processor 1310 is further configured to:

[0207] The channel mixture features and the channel separation features are respectively input into the second convolution module. Before obtaining the attention parameters, the channel separation features are subjected to feature extraction through at least one group convolution to obtain independent features of each channel.

[0208] The channel mixed features are extracted through the state space model and the multi-layer perceptron module to obtain the global features of at least two channels;

[0209] The processor 1310 is specifically configured to:

[0210] Input the independent features of each channel into the second convolution module to obtain the query matrix;

[0211] The global features of at least two channels are input into the second convolution module to obtain a key matrix and a value matrix.

[0212] In some embodiments, the processor 1310 is specifically configured to:

[0213] Upsampling the second low-frequency feature map to obtain an upsampled image;

[0214] Splicing the upsampled image with a first high-frequency feature map in at least one high-frequency feature map to obtain a spliced ​​image, wherein the first high-frequency feature map has a resolution consistent with that of the upsampled image;

[0215] Perform feature extraction on the spliced ​​image through the second convolution residual module to obtain an image mask;

[0216] The image mask is multiplied by the first high-frequency feature map and then added to the upsampled image to obtain a new second low-frequency feature map;

[0217] Return to the step of upsampling the second low-frequency feature map to obtain an upsampled image until the number of iterations is consistent with the number of images of at least one high-frequency feature map, and determine the final second low-frequency feature map as the second image.

[0218] It should be understood that in the embodiment of the present application, the input unit 1304 may include a graphics processor (Graphics Processing Unit, GPU) 13041 and a microphone 13042, and the graphics processor 13041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1306 may include a display panel 13061, and the display panel 13061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1307 includes a touch panel 13071 and at least one of other input devices 13072. The touch panel 13071 is also called a touch screen. The touch panel 13071 may include two parts: a touch detection device and a touch controller. Other input devices 13072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0219] The memory 1309 can be used to store software programs and various data. The memory 1309 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1309 may include a volatile memory or a non-volatile memory, or the memory x09 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1309 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0220] The processor 1310 may include one or more processing units; optionally, the processor 1310 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1310.

[0221] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0222] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0223] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0224] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0225] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0226] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0227] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0228] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: include: Extracting high-frequency features of the first image to obtain at least one high-frequency feature map; Extracting low-frequency features of the first image to obtain a first low-frequency feature map; Extracting channel separation features and channel mixing features of the color channel in the first low-frequency feature map; Based on the illumination-guided attention mechanism, the channel separation feature and the channel mixing feature are fused to obtain a second low-frequency feature map; Based on the at least one high-frequency feature map, detail enhancement processing is performed on the second low-frequency feature map to obtain a second image.

2. The method according to claim 1, characterized in that The step of extracting high-frequency features of the first image to obtain at least one high-frequency feature map includes: The first image is input into a multi-layer learnable difference pyramid to obtain at least one high-frequency feature map output by at least one layer in the learnable difference pyramid.

3. The method according to claim 2, characterized in that Each layer of the learnable difference pyramid includes a first convolution module, a difference module, a concatenation module and a first convolution residual module, and the first convolution module includes three convolution layers; The step of inputting the first image into a multi-layer structured learnable difference pyramid to obtain at least one high-frequency feature map output by at least one layer in the learnable difference pyramid comprises: For each layer in the learnable difference pyramid, extracting features of the image input to the layer in sequence through the three convolutional layers in the first convolutional module; By using the difference module, adjacent features extracted by adjacent convolution layers in the first convolution module are subtracted to obtain differential features; The differential features are spliced ​​by the splicing module to obtain high-frequency splicing features; The first convolution residual module is used to extract the high-frequency splicing features to obtain a high-frequency feature map corresponding to the layer.

4. The method according to claim 1, characterized in that: The first low-frequency feature map includes at least two color channels, and extracting channel separation features of the color channels in the first low-frequency feature map includes: Converting the first low-frequency feature map to the frequency domain according to each color channel to obtain a frequency domain image corresponding to each color channel; Extract the features of the frequency domain image corresponding to each color channel respectively to obtain the frequency domain features of each color channel; The frequency domain image corresponding to each color channel is converted into the spatial domain to obtain the spatial domain image corresponding to each color channel; Stitching the spatial domain images corresponding to the at least two color channels to obtain a spatial domain stitched image; Extracting features of the spatial domain stitched image to obtain spatial domain features of each color channel; The frequency domain features and the space domain features of the at least two color channels are determined as channel separation features of the color channels in the first low-frequency feature map.

5. The method according to claim 1, characterized in that The performing detail enhancement processing on the second low-frequency feature map based on the at least one high-frequency feature map to obtain a second image includes: Upsampling the second low-frequency feature map to obtain an upsampled image; Splicing the upsampled image with a first high-frequency feature map in the at least one high-frequency feature map to obtain a spliced ​​image, wherein the first high-frequency feature map has a resolution consistent with that of the upsampled image; Performing feature extraction on the spliced ​​image through a second convolution residual module to obtain an image mask; Multiplying the image mask by the first high-frequency feature map and then adding the resultant result to the upsampled image to obtain a new second low-frequency feature map; Return to the step of upsampling the second low-frequency feature map to obtain an upsampled image until the number of iterations is consistent with the number of images of the at least one high-frequency feature map, and determine the second low-frequency feature map finally obtained as the second image.

6. An image processing device, characterized in that: include: A high-frequency feature extraction module, used to extract high-frequency features of the first image to obtain at least one high-frequency feature map; A low-frequency feature extraction module, used to extract low-frequency features of the first image to obtain a first low-frequency feature map; An adaptive illumination module, used for extracting channel separation features and channel mixing features of the color channel in the first low-frequency feature map; A feature fusion module, used for fusing the channel separation feature and the channel mixing feature based on an illumination-guided attention mechanism to obtain a second low-frequency feature map; The iterative detail enhancement module is used to perform detail enhancement processing on the second low-frequency feature map based on the at least one high-frequency feature map to obtain a second image.

7. The device according to claim 6, characterized in that The high-frequency feature extraction module is specifically used for: The first image is input into a multi-layer learnable difference pyramid to obtain at least one high-frequency feature map output by at least one layer in the learnable difference pyramid.

8. The device according to claim 7, characterized in that Each layer of the learnable difference pyramid includes a first convolution module, a difference module, a concatenation module and a first convolution residual module, and the first convolution module includes three convolution layers; The high-frequency feature extraction module is specifically used for: For each layer in the learnable difference pyramid, extracting features of the image input to the layer in sequence through the three convolutional layers in the first convolutional module; By using the difference module, adjacent features extracted by adjacent convolution layers in the first convolution module are subtracted to obtain differential features; The differential features are spliced ​​by the splicing module to obtain high-frequency splicing features; The first convolution residual module is used to extract the high-frequency splicing features to obtain a high-frequency feature map corresponding to the layer.

9. The device according to claim 6, characterized in that The first low-frequency feature map includes at least two color channels, and the adaptive illumination module includes a separation feature extraction submodule, which is specifically used to: Converting the first low-frequency feature map to the frequency domain according to each color channel to obtain a frequency domain image corresponding to each color channel; Extract the features of the frequency domain image corresponding to each color channel respectively to obtain the frequency domain features of each color channel; The frequency domain image corresponding to each color channel is converted into the spatial domain to obtain the spatial domain image corresponding to each color channel; Stitching the spatial domain images corresponding to the at least two color channels to obtain a spatial domain stitched image; Extracting features of the spatial domain stitched image to obtain spatial domain features of each color channel; The frequency domain features and the space domain features of the at least two color channels are determined as channel separation features of the color channels in the first low-frequency feature map.

10. The device according to claim 6, characterized in that The iterative detail enhancement module is specifically used for: Upsampling the second low-frequency feature map to obtain an upsampled image; Splicing the upsampled image with a first high-frequency feature map in the at least one high-frequency feature map to obtain a spliced ​​image, wherein the first high-frequency feature map has a resolution consistent with that of the upsampled image; Performing feature extraction on the spliced ​​image through a second convolution residual module to obtain an image mask; Multiplying the image mask by the first high-frequency feature map and then adding the resultant result to the upsampled image to obtain a new second low-frequency feature map; Return to the step of upsampling the second low-frequency feature map to obtain an upsampled image until the number of iterations is consistent with the number of images of the at least one high-frequency feature map, and determine the second low-frequency feature map finally obtained as the second image.

Images