Image processing method and device, storage medium and electronic equipment

Abstract

Embodiments of the present application disclose an image processing method and device, a storage medium and an electronic device. The image processing method comprises: obtaining a RAW image; extracting a time domain feature map of the RAW image according to a first convolutional neural network of a preset convolutional neural network model, wherein the preset convolutional neural network model comprises the first convolutional neural network and a second convolutional neural network arranged side by side; extracting a frequency domain feature map of the RAW image according to the second convolutional neural network; and performing fusion processing on the time domain feature map and the frequency domain feature map to obtain an RGB image corresponding to the RAW image. The RAW image is processed in the time domain and the frequency domain based on one convolutional neural network model, and the time domain feature and the frequency domain feature are fused to obtain the RGB image, thereby avoiding error accumulation caused by processing the image through multiple stages, and improving the image processing precision and processing efficiency.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to an image processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] Currently, mobile phone photography performance is increasingly approaching that of DSLRs, thanks to the ISP (Image Signal Processor) module built into the phone's operating system. The ISP is used to convert RAW format image signals into sRGB format images.

[0003] The ISP module processes RAW format images in stages through multiple cascaded image processing modules, namely: image de-mosaic processing, image noise reduction processing, white balance and color space transformation processing, color enhancement processing, and tone mapping processing.

[0004] Because image processing requires multiple stages, this multi-stage processing method causes error accumulation and reduces the accuracy of image processing. Summary of the Invention

[0005] This application provides an image processing method, apparatus, storage medium, and electronic device, which avoids the need for multiple stages of image processing in the prior art and can improve the processing accuracy of images.

[0006] In a first aspect, embodiments of this application provide an image processing method, including:

[0007] Acquire RAW images;

[0008] The temporal feature map of the RAW image is extracted according to the first convolutional neural network of the preset convolutional neural network model, wherein the preset convolutional neural network model includes a first convolutional neural network and a second convolutional neural network in parallel.

[0009] Frequency domain feature maps of RAW images are extracted using the second convolutional neural network;

[0010] The time-domain feature map and the frequency-domain feature map are fused to obtain the RGB image corresponding to the RAW image.

[0011] Secondly, embodiments of this application also provide an image processing apparatus, comprising:

[0012] The image acquisition module is used to acquire RAW images;

[0013] The model acquisition module is used to acquire a preset convolutional neural network model, which includes a first convolutional neural network and a second convolutional neural network in parallel.

[0014] The image processing module is used to process RAW images using a preset convolutional neural network model. Specifically, it extracts the temporal feature map of the RAW image through a first convolutional neural network and extracts the frequency domain feature map of the RAW image through a second convolutional neural network. The temporal feature map and the frequency domain feature map are then fused to obtain the RGB image corresponding to the RAW image.

[0015] Thirdly, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when run on a computer, causes the computer to perform the image processing method provided in any embodiment of this application.

[0016] Fourthly, embodiments of this application also provide an electronic device, including a processor and a memory, the memory having a computer program, and the processor executing an image processing method as provided in any embodiment of this application by calling the computer program.

[0017] The technical solution provided in this application uses a preset convolutional neural network model to process RAW images, thereby obtaining an RGB image that integrates time-domain and frequency-domain features. This solution can process RAW images through a preset convolutional neural network model, thus avoiding the need for multi-stage processing of RAW images in the prior art, reducing the accumulation of errors caused by multi-stage processing of RAW images, and improving the processing accuracy and efficiency of RAW images. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic flowchart of an image processing method provided in an embodiment of this application.

[0020] Figure 2 This is a schematic diagram of the structure of a preset convolutional neural network model provided in an embodiment of this application.

[0021] Figure 3 This is a schematic diagram of a convolution operation provided in an embodiment of this application.

[0022] Figure 4 This is a schematic diagram of the structure of the first convolutional neural network provided in an embodiment of this application.

[0023] Figure 5This is a schematic diagram of the processing flow of the attention mechanism provided in the embodiments of this application.

[0024] Figure 6 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application.

[0025] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0027] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0028] This application provides an image processing method. The execution subject of this image processing method can be the image processing device provided in this application, or an electronic device integrating the image processing device. The image processing device can be implemented in hardware or software. The electronic device can be a smartphone, tablet computer, PDA, laptop computer, or desktop computer, etc.

[0029] Please see Figure 1 , Figure 1 This is a schematic flowchart of the image processing method provided in an embodiment of this application. The specific flow of the image processing method provided in this embodiment of the application can be as follows:

[0030] 101. Obtain the RAW image.

[0031] RAW images represent raw images, which are images directly acquired from the image sensor of a digital camera, scanner, or film scanner and have not yet been processed. Raw images possess a wide color gamut and retain most of the image information captured.

[0032] RAW images are obtained by capturing images using the electronic device provided in this application embodiment. The electronic device integrates a camera to achieve a photo-taking function, and the camera can be a front-facing camera or a rear-facing camera; this is not limited to either. Alternatively, RAW images can be captured using a device with a camera, and then acquired and processed by the electronic device. Since there are multiple methods for acquiring RAW images, only the appropriate method needs to be selected in practical applications, and this should not be construed as a limitation of this application.

[0033] 102. Extract the temporal feature map of the RAW image according to the first convolutional neural network of the preset convolutional neural network model, wherein the preset convolutional neural network model includes a first convolutional neural network and a second convolutional neural network in parallel.

[0034] In this embodiment, a convolutional neural network model is pre-trained using sample images, and then the trained convolutional neural network model is directly used to process RAW images to obtain images in a visualization format corresponding to the RAW images, such as RGB or sRGB formats.

[0035] like Figure 2 As shown, Figure 2 This is a schematic diagram of the structure of a preset convolutional neural network model in an embodiment of this application. The trained convolutional neural network model includes a two-part network structure, namely a first convolutional neural network and a second convolutional neural network, wherein the first convolutional neural network and the second convolutional neural network are connected in parallel.

[0036] When processing RAW images using a trained convolutional neural network model, the RAW image is input into the convolutional neural network model, and the RAW image is simultaneously acquired by two parallel first and second convolutional neural networks in the convolutional neural network model. The first convolutional neural network extracts temporal features from the RAW image to obtain a temporal feature map, and the second convolutional neural network extracts frequency features from the RAW image to obtain a frequency domain feature map.

[0037] 103. Extract the frequency domain feature map of the RAW image based on the second convolutional neural network.

[0038] The first convolutional neural network is used to extract features of the RAW image in the spatial domain, while the second convolutional neural network is used to extract features of the RAW image in the frequency domain.

[0039] For example, in this embodiment, the temporal domain refers to processing the RAW image in the spatial dimension, that is, processing the RAW image in the spatial domain, so as to extract the features of the RAW image in the spatial dimension.

[0040] The frequency domain refers to the frequency domain, which means transforming the image to another domain for processing and calculation, and then performing an inverse domain transformation, so that the features of the RAW image can be extracted from the frequency domain.

[0041] The first and second convolutional neural networks can extract features of RAW images in different domains, thereby obtaining more feature information of the RAW images, realizing the enhancement processing of RAW images, and obtaining the corresponding time-domain feature maps and frequency-domain feature maps.

[0042] In some embodiments, the first convolutional neural network and the second convolutional neural network employ different algorithms. The first convolutional neural network uses a convolutional sampling algorithm to extract features of the RAW image in the spatial domain, while the second convolutional neural network uses a wavelet transform algorithm to extract features of the RAW image in the frequency domain.

[0043] For example, step 102 includes: 1021, using a first convolutional neural network based on a preset convolutional neural network model to extract the temporal feature map of the RAW image using a convolutional sampling algorithm.

[0044] Step 103 includes: 1031. Based on the second convolutional neural network, the frequency domain feature map of the RAW image is extracted using the wavelet transform sampling algorithm.

[0045] During the convolution operation on a RAW image using either a first or second convolutional neural network, the RAW image is converted into an RGB image. Therefore, the first convolutional neural network maps the RAW image to a temporal feature map in RGB format, while the second convolutional neural network maps it to a frequency domain feature map in RGB format. This achieves color gamut conversion and tone mapping for the RAW image.

[0046] Specifically, when extracting the temporal feature map of a RAW image using the convolutional sampling algorithm, a convolution operation is performed by assigning values ​​to the convolution kernel. The convolution kernel is a two-dimensional grid data. During the convolution operation, the product of each pixel in the RAW image and its surrounding pixels with the convolution kernel is calculated and added together. The pixel value of each pixel is then recalculated, resulting in a temporal feature map. The pixel value of each pixel in the temporal feature map is the recalculated pixel value.

[0047] As shown in Figure 3 Figure 3 This is a schematic diagram of the convolution operation in an embodiment of this application. Figure 3 The convolution kernel is 3*3. When recalculating the pixel value of a target pixel, the center of the convolution kernel grid is aligned with the target pixel, the values ​​of the pixels around the target pixel are multiplied by the corresponding values ​​in the convolution kernel grid, and finally all the products are added together to get the pixel value of the target pixel.

[0048] For example, the convolution kernel is The original pixel value of the target pixel is 1, and its surrounding pixels are represented by a matrix. The calculated pixel value of the target pixel is 8.

[0049] By converting the pixel values ​​of RAW images, feature information can be obtained from the surrounding area of ​​each pixel and then added to each pixel. This method facilitates white balance processing, tone enhancement processing, and sharpening processing of images, thereby enhancing the image processing.

[0050] In the sampling wavelet transform algorithm for extracting the frequency domain feature map of a RAW image, the digital signal of the RAW image is decomposed into sine waves of different frequencies through wavelet transform to achieve feature extraction in the frequency domain. During wavelet transform processing, each row and column of the RAW image is filtered to obtain the low-frequency and high-frequency components in three dimensions. These components are then modified using high-frequency and low-frequency coefficients.

[0051] By processing the frequency distribution of RAW images, noisy high-frequency components can be filtered out, thus achieving image denoising. At the same time, processing low-frequency components can change the smoothness of the image.

[0052] 104. The time-domain feature map and the frequency-domain feature map are fused to obtain the RGB image corresponding to the RAW image.

[0053] By fusing temporal and frequency domain feature maps, the effects of image enhancement in the spatial domain and image filtering in the frequency domain can be combined. Image enhancement can achieve dehazing, contour enhancement, and grayscale equalization, replacing the de-mosaic and white balance stages performed by the ISP module in existing technologies. Similarly, image filtering can replace the image denoising stage performed by the ISP model in existing technologies. Furthermore, converting RAW images to RGB images through convolution operations enables color space transformation, color enhancement, and tone mapping. Therefore, this embodiment achieves the effects of multiple image processing stages by training only a single convolutional neural network model, avoiding the error accumulation caused by multiple stages of image processing, thus improving both image processing accuracy and efficiency.

[0054] In some embodiments, the first convolutional neural network includes a first encoding subnetwork and a first decoding subnetwork. The first encoding subnetwork includes a plurality of first downsampling units connected in stages, and the first decoding subnetwork includes a plurality of first upsampling units connected in stages.

[0055] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of the first convolutional neural network in an embodiment of this application. The first convolutional neural network includes a first encoding subnetwork for downsampling and a first decoding subnetwork connected to the first encoding subnetwork for upsampling. The first encoding subnetwork includes a plurality of first downsampling units connected in sequence, and each first downsampling unit is distributed at different levels. The first decoding subnetwork includes a plurality of first upsampling units connected in sequence, and each first upsampling unit is distributed at different levels. Each level has one first upsampling unit and one first downsampling unit. The structures of the first encoding subnetwork and the first decoding subnetwork are symmetrical, forming an encoding-decoding network structure.

[0056] The encoder-decoder network structure, also known as the Unet (U-shaped network) structure, is a deep learning framework that includes encoding and decoding processes. The encoding process is the downsampling process of the image, and the decoding process is the inverse operation of the encoding process. The output feature map of the encoded image is continuously upsampled to obtain a feature map with the same size as the original input.

[0057] In some embodiments, step 1021 includes:

[0058] The first feature map is obtained by performing progressive convolution downsampling on the RAW image through multiple first downsampling units. In other words, the first feature map is obtained by performing multiple convolution operations on the RAW image through multiple first downsampling units.

[0059] For example, in this embodiment, convolutional downsampling is used to process the image step by step, wherein a convolutional downsampling with a larger stride can be used.

[0060] In this process, multiple first downsampling units perform progressive convolutional downsampling on the RAW image. The number of downsampling operations is related to the number of first downsampling units; each first downsampling unit performs one downsampling operation. Each downsampling process yields a feature map with a reduced size. The following example uses a 224*224 image with four first downsampling units for illustration:

[0061] A feature map of size 112*112 can be obtained through the first downsampling unit;

[0062] The 112*112 feature map is then processed in the second first downsampling unit to obtain a 56*56 feature map.

[0063] A 56*56 feature map can be processed by a third first downsampling unit to obtain a 28*28 feature map.

[0064] A 28*28 feature map can be transformed into a 14*14 feature map by passing through the fourth first downsampling unit.

[0065] The first downsampling unit includes a first convolutional unit and a second convolutional unit. The first convolutional unit includes a convolutional layer, an activation layer, and a batch normalization (BN) layer. The second convolutional unit includes a convolutional layer, a BN layer, and an activation layer. The stride of the convolutional layer in the first convolutional unit is 1, and the stride of the convolutional layer in the second convolutional unit is 2. The second convolutional unit primarily functions to reduce the feature map size and achieve downsampling.

[0066] As shown above, convolutional downsampling can produce multiple output feature maps with progressively smaller sizes. The number of output feature maps in the downsampling part is related to the number of the first downsampling units, which can be determined according to actual needs and is not limited here.

[0067] In this embodiment, the first feature map refers to the smallest output feature map obtained after multiple downsampling processes by multiple first downsampling units.

[0068] The first feature map is subjected to progressive deconvolution upsampling processing by multiple first upsampling units to obtain a temporal feature map. The fused feature map obtained by fusing the output feature map of each first upsampling unit with the output feature map of the corresponding level first downsampling unit is used as the input feature map of the next first upsampling unit.

[0069] Deconvolution, also known as transposed convolution, has the advantages of fewer parameters and higher speed. The first upsampling unit includes a third convolutional unit and an upsampling layer. The third convolutional unit includes a deconvolutional layer, an activation layer, and a batch normalization (BN) layer. The third convolutional unit mainly expands the feature map size and achieves upsampling, while the upsampling layer mainly extracts features.

[0070] Since the multiple first upsampling units in the decoding section and the multiple first upsampling units in the encoding section are structurally symmetrical and have the same number, the number of multiple first upsampling units can be referred to the description of the multiple first downsampling units mentioned above, and will not be repeated here.

[0071] Multiple first upsampling units can be used to perform progressively upward deconvolution operations on the first feature map, ultimately obtaining the temporal feature map corresponding to the RAW image.

[0072] For example, the first downsampling unit and the first upsampling unit in the same layer are connected by a feature fusion unit. The feature fusion unit is used to pass the output feature map of the first downsampling unit to the first upsampling unit. The feature fusion of the two output feature maps is implemented in the first upsampling unit, and then the fused feature map is used as the input feature map of the next first upsampling unit.

[0073] Based on this, top-down feature fusion of the feature map is achieved through the encoding part (i.e., the first encoding sub-network) in the encoder-decoder network structure, and bottom-up feature fusion of the feature map is achieved through the decoding part (i.e., the first decoding sub-network).

[0074] For example, the first upsampling unit uses concatenation as a feature map fusion method, where more features are obtained by concatenating the number of channels.

[0075] In some embodiments, the second convolutional neural network includes a second encoding subnetwork and a second decoding subnetwork. The second encoding subnetwork includes a plurality of second downsampling units connected in stages, and the second decoding subnetwork includes a plurality of second upsampling units connected in stages.

[0076] The second convolutional neural network has the same structure as the first convolutional neural network, both being encoder-decoder network structures. The second encoder subnetwork serves as the encoder part and includes multiple second downsampling units, which perform downsampling. The second decoder subnetwork serves as the decoder part and includes multiple second upsampling units, with the number of second upsampling units being the same as the number of second downsampling units, and they are symmetrically distributed. The multiple second upsampling units are used to upsample the output feature map of the encoder part.

[0077] In some embodiments, step 1031 includes:

[0078] The second feature map is obtained by performing successive wavelet transform downsampling on the RAW image through multiple second downsampling units. That is, the second feature map is obtained by performing multiple wavelet transforms and convolution operations on the RAW image through multiple second downsampling units.

[0079] For example, the RAW image is downsampled by the second downsampling unit, that is, the RAW image is converted into a time-frequency diagram, thereby realizing the processing of the RAW image in the frequency domain.

[0080] The second downsampling unit includes a fourth convolutional unit and a wavelet transform downsampling layer. The fourth convolutional unit comprises a convolutional layer, a batch normalization (BN) layer, and an activation layer. The fourth convolutional unit has the same structure as the first convolutional unit and is also used to reduce the feature map size to achieve downsampling. The wavelet transform downsampling layer is used to extract features from the feature map in the frequency domain.

[0081] The second feature map refers to the smallest output feature map obtained after downsampling by the second downsampling unit.

[0082] The second feature map is subjected to stepwise inverse wavelet transform upsampling processing by multiple second upsampling units to obtain a frequency domain feature map. The fused feature map obtained by fusing the output feature map of each second upsampling unit with the output feature map of the corresponding level second downsampling unit is used as the input feature map of the next first upsampling unit.

[0083] The difference between the second upsampling unit and the second downsampling unit lies in that the second upsampling unit includes a fifth convolutional unit and an inverse wavelet transform upsampling layer. The fifth convolutional unit comprises a deconvolutional layer, a batch normalization (BN) layer, and an activation layer. The fifth convolutional unit has the same structure as the third convolutional unit and is also used to expand the feature map size and achieve upsampling. The inverse wavelet transform upsampling layer mainly performs an inverse transform on the frequency domain.

[0084] Multiple second upsampling units can be used to perform a step-by-step upward inverse wavelet transform on the second feature map, ultimately obtaining the frequency domain feature map corresponding to the RAW image.

[0085] For example, referring to the connection method of the first downsampling unit and the first upsampling unit, similarly, the second downsampling unit and the second upsampling unit in the same layer are connected through a feature fusion unit to pass the output feature map of the second downsampling unit to the second upsampling unit, and to realize the fusion of the two output feature maps at the same level in the second upsampling unit. Then, the fused feature map is used as the input feature map of the next second upsampling unit.

[0086] Understandably, the second upsampling unit can achieve feature map fusion by channel splicing.

[0087] In some embodiments, the feature fusion unit may also perform attention mechanism processing on the feature map. After processing the output feature map of each first downsampling unit or each second downsampling unit through the attention mechanism, the processed third feature map is passed to the first upsampling unit or the second upsampling unit at the same level, and feature fusion processing is performed on its corresponding output feature map in the first upsampling unit or the second upsampling unit. The resulting fused feature map is then used as the input feature map of the next first upsampling unit or the second upsampling unit.

[0088] Please see Figure 5 , Figure 5 This is a schematic diagram of the attention mechanism processing flow provided in the embodiments of this application. When the feature fusion unit performs attention mechanism processing on the output feature map of each first downsampling unit or each second downsampling unit, it executes the following steps:

[0089] 201. Divide each channel of the output feature map into a first part and a second part respectively.

[0090] For example, the image is divided into three channels based on the RGB mode: the R channel, the G channel, and the B channel, with each channel divided into two equal parts. Understandably, dividing the channels equally can directly divide the output feature map into two parts (top and bottom, left and right), or it can be divided into two parts based on the number of pixels. The specific division method is not limited here, as long as it achieves the desired channel division.

[0091] 202. The first part is processed based on the channel attention mechanism to obtain the first part features, and the second part is processed based on the spatial attention mechanism to obtain the second part features.

[0092] For example, the channel attention mechanism processes the first part in two steps. First, the first part is compressed into a global compressed feature vector. Then, weights are assigned to the global compressed feature vector. The first part of each channel has different weights. That is, the weights of the first part of the same channel are the same, while the weights of the first part of different channels are different. Then, the weights of the channels are weighted and summed with the pixel values ​​of each pixel in the first part to obtain the first part features, thereby suppressing irrelevant features in the first part.

[0093] When processing the second part based on the spatial attention mechanism, each pixel is assigned a weight, and the pixel value of each pixel is summed with its weight to obtain the features of the second part, thereby strengthening the useful features in the second part.

[0094] By selectively utilizing the first part in the channel dimension and the second part in the spatial dimension, useful features can be enhanced while irrelevant features are suppressed.

[0095] 203. The first part of the features and the second part of the features in the same channel are concatenated to obtain the third feature map. The third feature map is then fused with the output feature map of the first upsampling unit or the second upsampling unit at the corresponding level to obtain the fused feature map.

[0096] The first part of the features is a channel-dimensional feature map obtained through a channel attention mechanism, and the second part of the features is a spatial-dimensional feature map obtained through a spatial attention mechanism. The first and second parts of the features can be concatenated by channel-based fusion to obtain the concatenated features of each channel. After obtaining the concatenated feature map of each channel, the channel can be dimensionality reduced, and then the channels can be concatenated into a third feature map. The third feature map is then input into the first or second upsampling unit at the same level to achieve feature fusion between the third feature map and the output feature map of the first or second upsampling unit, resulting in a fused feature map that serves as the input feature map for the next first or second upsampling unit.

[0097] For example, the spliced ​​feature maps of each channel can also be shuffled and then spliced ​​into a third feature map.

[0098] By extracting useful information from the output feature map through the attention mechanism, the computational cost of the convolutional neural network model can be reduced, thereby improving the model's processing performance.

[0099] In some embodiments, the method further includes the following steps before constructing the first convolutional neural network and the second convolutional neural network:

[0100] 301. Based on the preset training set, the preset Unet network is searched using a differentiable architecture search algorithm to obtain the target upsampling unit and the target downsampling unit.

[0101] The preset training set can be the SID dataset, which contains 5094 original short-exposure images, each with a corresponding long-exposure reference image.

[0102] Differentiable Architecture Search (DNAS) is an algorithm that automatically searches by constructing a differentiable neural network through continuous relaxation. DNAS supports de-hierarchical search space; instead, it represents the search space through a hypernet whose operators are randomly executed. DNAS can find the distribution of optimal structures within the search space.

[0103] The search space can be configured with the number of network layers of the Unet network to be searched. Each network layer corresponds to one upsampling unit, one downsampling unit, and one feature fusion unit. The random combination of units in each network layer constitutes the search space. The dataset formed by the random combination of units in each network layer is called the preset Unet network. During the search, an alternating optimization method can be used to learn the parameters of each layer and the weights of the units. Then, the preset convolutional neural network is updated and optimized through the parameters and / or weights until all parameters converge.

[0104] In this embodiment, based on the ISD dataset and the search space corresponding to the preset Unet network, a parameter can be obtained through neural network structure search. This parameter indicates the importance of each unit to improving network performance. Based on this parameter, the search architecture of DNAS is assigned a value, allowing the search for target upsampling units and target downsampling units with optimal structures within the search space. The target downsampling unit represents the encoding part, and the target upsampling unit represents the decoding part.

[0105] 302. Construct the first convolutional neural network based on the target upsampling unit and the target downsampling unit.

[0106] When constructing the first convolutional neural network, it is based on the structure of the Unet network model. The target downsampling unit is used as the first downsampling unit, the target upsampling unit is used as the first upsampling unit, and the resulting multiple first downsampling units and multiple first upsampling units are constructed into the first convolutional neural network through a multi-layer stacking algorithm.

[0107] Among them, the multi-layer stacking algorithm is to stack a shallow sequence recommendation model multiple times to obtain a deep sequence recommendation model. That is, after training the parameters of the shallow model, the parameters are transferred to the deep model. By fine-tuning the parameters, the deep model can converge quickly and achieve optimal performance.

[0108] For example, convolution operators can be set at the head and tail of the first convolutional neural network to adjust the depth of the feature map, so that the size of the RAW image decreases by a factor of 4 after being input into the first convolutional neural network, while the size increases by a factor of 4 when the RGB image is output from the first convolutional neural network, thereby enabling image preprocessing to obtain the feature map after the RAW image is input into the first convolutional neural network.

[0109] 303. Construct a second convolutional neural network based on the first convolutional neural network.

[0110] After obtaining the trained first convolutional neural network, the second encoding subnetwork can be obtained by replacing the convolutional sampling algorithm of the first encoding subnetwork with the wavelet transform sampling algorithm, and the second decoding subnetwork can be obtained by replacing the deconvolutional sampling algorithm of the first decoding subnetwork with the inverse wavelet transform sampling algorithm. This realizes the updating of the sampling algorithm and obtains the second convolutional neural network based on the framework of the first convolutional neural network.

[0111] In some embodiments, after constructing the first convolutional neural network and the second convolutional neural network, the first convolutional neural network and the second convolutional neural network are connected in parallel to form a convolutional neural network model. Then, the convolutional neural network model is trained, and the training steps are as follows:

[0112] Obtain the sample RAW images and their corresponding sample RGB images. Each sample RAW image and its corresponding sample RGB image constitute one training data point. The sample RGB images can be obtained using traditional conversion methods.

[0113] A convolutional neural network model is trained based on multiple training data sets to determine the model parameters.

[0114] In the training process, the convolutional neural network model learns the mapping relationship between RAW images and RGB images. When a new RAW image is input into the trained model, the trained model can calculate the corresponding RGB image of the new RAW image.

[0115] In this embodiment, by searching for the target upsampling unit and the target downsampling unit, it is convenient to optimize the structure of the first convolutional neural network, so that the final convolutional neural network model has a small computational load and can utilize the frequency domain and time domain information in the RAW image to enhance the image processing performance of the convolutional neural network model.

[0116] In some embodiments, step 104 includes:

[0117] The time-domain feature map and the frequency-domain feature map are subjected to feature averaging to obtain the RGB image corresponding to the RAW image.

[0118] For example, the average value of the feature values ​​at corresponding positions in the time-domain feature map and the frequency-domain feature map can be calculated to obtain the RGB image.

[0119] In practice, this application is not limited by the execution order of the described steps. Without causing conflicts, some steps may be performed in other orders or simultaneously.

[0120] As can be seen from the above, the image processing method provided in this application uses a trained convolutional neural network model to process RAW images, replacing the multi-stage processing method of RAW images in the prior art, thereby avoiding the superposition of errors during image processing and improving the accuracy and efficiency of image processing.

[0121] In one embodiment, an image processing apparatus 300 is also provided. See also... Figure 6 , Figure 6 This is a schematic diagram of the structure of an image processing apparatus 300 provided in an embodiment of this application. The image processing apparatus 300 is applied to an electronic device and includes an image acquisition module 301, a model acquisition module 302, and an image processing module 303, as follows:

[0122] Image acquisition module 301 is used to acquire RAW images.

[0123] The model acquisition module 302 is used to acquire a preset convolutional neural network model, which includes a first convolutional neural network and a second convolutional neural network in parallel.

[0124] The image processing module 303 is used to process RAW images using a preset convolutional neural network model. Specifically, it extracts the temporal feature map of the RAW image through a first convolutional neural network, extracts the frequency domain feature map of the RAW image through a second convolutional neural network, and fuses the temporal and frequency domain feature maps to obtain the RGB image corresponding to the RAW image.

[0125] In some embodiments, the image processing module 303 is further configured to:

[0126] The first convolutional neural network, based on a pre-defined convolutional neural network model, uses a convolutional sampling algorithm to extract temporal feature maps from RAW images.

[0127] Based on the second convolutional neural network, the frequency domain feature map of the RAW image is extracted using the wavelet transform sampling algorithm.

[0128] In some embodiments, the first convolutional neural network includes a first encoding subnetwork and a first decoding subnetwork. The first encoding subnetwork includes a plurality of first downsampling units connected in stages, and the first decoding subnetwork includes a plurality of first upsampling units connected in stages.

[0129] Image processing module 303 is also used for:

[0130] The first feature map is obtained by performing progressive convolutional downsampling on the RAW image through multiple first downsampling units.

[0131] The first feature map is subjected to progressive deconvolution upsampling processing by multiple first upsampling units to obtain a temporal feature map. The fused feature map obtained by fusing the output feature map of each first upsampling unit with the output feature map of the corresponding level first downsampling unit is used as the input feature map of the next first upsampling unit.

[0132] In some embodiments, the second convolutional neural network includes a second encoding subnetwork and a second decoding subnetwork. The second encoding subnetwork includes a plurality of second downsampling units connected in stages, and the second decoding subnetwork includes a plurality of second upsampling units connected in stages.

[0133] Image processing module 303 is also used for:

[0134] The second feature map is obtained by performing successive wavelet transform downsampling on the RAW image through multiple second downsampling units.

[0135] The second feature map is subjected to stepwise inverse wavelet transform upsampling processing by multiple second upsampling units to obtain a frequency domain feature map. The fused feature map obtained by fusing the output feature map of each second upsampling unit with the output feature map of the corresponding level second downsampling unit is used as the input feature map of the next first upsampling unit.

[0136] In some embodiments, the image processing module 303 further includes a feature fusion module connecting a first upsampling unit and a first downsampling unit at the same level, and a second upsampling unit and a second downsampling unit at the same level. The feature fusion module is used for:

[0137] For the output feature map of each first downsampling unit or each second downsampling unit, each channel of the output feature map is divided into a first part and a second part respectively;

[0138] The first part is processed based on the channel attention mechanism to obtain the first part features, and the second part is processed based on the spatial attention mechanism to obtain the second part features;

[0139] The first and second features in the same channel are concatenated to obtain a third feature map. The third feature map is then fused with the output feature map of the corresponding level's first or second upsampling unit to obtain a fused feature map.

[0140] In some embodiments, the image processing apparatus 300 further includes a search module, which is used for:

[0141] Based on the preset training set, the preset Unet network is searched using a differentiable architecture search algorithm to obtain the target upsampling unit and the target downsampling unit;

[0142] The first convolutional neural network is constructed based on the target upsampling unit and the target downsampling unit.

[0143] Construct a second convolutional neural network based on the first convolutional neural network.

[0144] In some embodiments, the image processing module 303 is further configured to:

[0145] The time-domain feature map and the frequency-domain feature map are subjected to feature averaging to obtain the RGB image corresponding to the RAW image.

[0146] It should be noted that the image processing apparatus provided in this application embodiment and the image processing method in the above embodiment belong to the same concept. The image processing apparatus can implement any of the methods provided in the image processing method embodiment. For details of its implementation process, please refer to the image processing method embodiment, which will not be repeated here.

[0147] As can be seen from the above, the image processing apparatus proposed in this application embodiment can process RAW images in both the time and frequency domains to obtain an RGB image that integrates the time and frequency domain features, thereby replacing the existing method of processing RAW images through multiple stages, thus avoiding the superposition of errors during image processing and improving image processing accuracy and efficiency.

[0148] This application also provides an electronic device, which can be a terminal, such as a smartphone, tablet computer, laptop computer, touch screen, game console, personal computer (PC), personal digital assistant (PDA), or other terminal device. Figure 7 As shown, Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 400 includes a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, and a computer program stored in the memory 402 and executable on the processor. The processor 401 and the memory 402 are electrically connected.

[0149] Those skilled in the art will understand that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0150] The processor 401 is the control center of the electronic device 400. It connects various parts of the electronic device 400 through various interfaces and lines. By running or loading software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, it performs various functions of the electronic device 400 and processes data, thereby monitoring the electronic device 400 as a whole.

[0151] In this embodiment, the processor 401 in the electronic device 400 loads the instructions corresponding to the processes of one or more applications into the memory 402 according to the following steps, and the processor 401 runs the applications stored in the memory 402 to realize various functions:

[0152] Acquire RAW images;

[0153] The temporal feature map of the RAW image is extracted according to the first convolutional neural network of the preset convolutional neural network model, wherein the preset convolutional neural network model includes a first convolutional neural network and a second convolutional neural network in parallel.

[0154] Frequency domain feature maps of RAW images are extracted using the second convolutional neural network;

[0155] The time-domain feature map and the frequency-domain feature map are fused to obtain the RGB image corresponding to the RAW image.

[0156] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0157] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0158] As can be seen from the above, the electronic device provided in this embodiment can process RAW images in the time domain and frequency domain to obtain RGB images that integrate time domain and frequency domain features, thereby replacing the existing technology of processing RAW images through multiple stages, thus avoiding the superposition of errors during image processing and improving image processing accuracy and efficiency.

[0159] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0160] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of computer programs that can be loaded by a processor to execute the steps of any of the image processing methods provided in embodiments of this application. For example, the computer program can execute the following steps:

[0161] Acquire RAW images;

[0162] The temporal feature map of the RAW image is extracted according to the first convolutional neural network of the preset convolutional neural network model, wherein the preset convolutional neural network model includes a first convolutional neural network and a second convolutional neural network in parallel.

[0163] Frequency domain feature maps of RAW images are extracted using the second convolutional neural network;

[0164] The time-domain feature map and the frequency-domain feature map are fused to obtain the RGB image corresponding to the RAW image.

[0165] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0166] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk, or optical disk, etc. Since the computer program stored in the storage medium can execute the steps of any of the image processing methods provided in the embodiments of this application, it can achieve the beneficial effects achievable by any of the image processing methods provided in the embodiments of this application, as detailed in the preceding embodiments, and will not be repeated here.

[0167] The foregoing has provided a detailed description of an image processing method, apparatus, medium, and electronic device provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image processing method, characterized in that, include: Acquire RAW images; The RAW image is subjected to progressive convolutional downsampling processing by multiple first downsampling units in a preset convolutional neural network model to obtain a first feature map. The preset convolutional neural network model includes a first convolutional neural network and a second convolutional neural network in parallel. The first convolutional neural network is used for convolution and deconvolution processing, and the second convolutional neural network is used for wavelet transform and inverse wavelet transform processing. The first convolutional neural network includes a first encoding subnetwork and a first decoding subnetwork. The first encoding subnetwork includes multiple progressively connected first downsampling units, and the first decoding subnetwork includes multiple progressively connected first upsampling units. The second convolutional neural network includes a second encoding subnetwork and a second decoding subnetwork. The second encoding subnetwork includes multiple progressively connected second downsampling units, and the second decoding subnetwork includes multiple progressively connected second upsampling units. The first feature map is subjected to progressive deconvolution upsampling processing by the plurality of first upsampling units to obtain a temporal feature map. The fused feature map obtained by fusing the output feature map of each first upsampling unit with the output feature map of the corresponding level of the first downsampling unit is used as the input feature map of the next first upsampling unit. The RAW image is subjected to successive wavelet transform downsampling processing through the multiple second downsampling units to obtain the second feature map; The second feature map is subjected to stepwise inverse wavelet transform upsampling processing by the multiple second upsampling units to obtain a frequency domain feature map. The fused feature map obtained by fusing the output feature map of each second upsampling unit with the output feature map of the corresponding level of the second downsampling unit is used as the input feature map of the next second upsampling unit. The time-domain feature map and the frequency-domain feature map are fused to obtain the RGB image corresponding to the RAW image.

2. The image processing method as described in claim 1, characterized in that, The method further includes: For the output feature map of each first downsampling unit or each second downsampling unit, each channel of the output feature map is divided into a first part and a second part respectively; The first part is processed based on the channel attention mechanism to obtain the first part features, and the second part is processed based on the spatial attention mechanism to obtain the second part features; The first part of the features and the second part of the features in the same channel are concatenated to obtain a third feature map. The third feature map is then fused with the output feature map of the corresponding level first upsampling unit or second upsampling unit to obtain the fused feature map.

3. The image processing method as described in claim 1, characterized in that, The method further includes: Based on the preset training set, the preset Unet network is searched using a differentiable architecture search algorithm to obtain the target upsampling unit and the target downsampling unit; The first convolutional neural network is constructed based on the target upsampling unit and the target downsampling unit; The second convolutional neural network is constructed based on the first convolutional neural network.

4. The image processing method as described in claim 1, characterized in that, The step of fusing the time-domain feature map and the frequency-domain feature map to obtain the RGB image corresponding to the RAW image includes: The time-domain feature map and the frequency-domain feature map are subjected to feature averaging to obtain the RGB image corresponding to the RAW image.

5. An image processing apparatus, characterized in that, include: The image acquisition module is used to acquire RAW images; The model acquisition module is used to acquire a preset convolutional neural network model, which includes a first convolutional neural network and a second convolutional neural network in parallel. The first convolutional neural network is used for convolution and deconvolution processing, and the second convolutional neural network is used for wavelet transform and inverse wavelet transform processing. The first convolutional neural network includes a first encoding subnetwork and a first decoding subnetwork. The first encoding subnetwork includes a plurality of first downsampling units connected in stages, and the first decoding subnetwork includes a plurality of first upsampling units connected in stages. The second convolutional neural network includes a second encoding subnetwork and a second decoding subnetwork. The second encoding subnetwork includes a plurality of second downsampling units connected in stages, and the second decoding subnetwork includes a plurality of second upsampling units connected in stages. The image processing module is used to perform progressive convolutional downsampling on the RAW image through multiple first downsampling units in a preset convolutional neural network model to obtain a first feature map; to perform progressive deconvolutional upsampling on the first feature map through multiple first upsampling units to obtain a temporal feature map; wherein the output feature map of each first upsampling unit is fused with the output feature map of the corresponding level of the first downsampling unit to obtain a fused feature map, which is used as the input feature map of the next first upsampling unit; to perform progressive wavelet transform downsampling on the RAW image through multiple second downsampling units to obtain a second feature map; to perform progressive inverse wavelet transform upsampling on the second feature map through multiple second upsampling units to obtain a frequency domain feature map; wherein the output feature map of each second upsampling unit is fused with the output feature map of the corresponding level of the second downsampling unit to obtain a fused feature map, which is used as the input feature map of the next second upsampling unit; and to fuse the temporal feature map and the frequency domain feature map to obtain the RGB image corresponding to the RAW image.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run on a computer, it causes the computer to perform the image processing method as described in any one of claims 1 to 4.

7. An electronic device comprising a processor and a memory, the memory storing a computer program, characterized in that, The processor executes the image processing method as described in any one of claims 1 to 4 by invoking the computer program.

Images