A method, electronic device, apparatus, and computing system for image processing
By constructing time-domain and frequency-domain UNET subnetworks of the image processing neural network, feature extraction and fusion are performed separately, which solves the problem of low performance of existing ISP imaging processing models and achieves more efficient image processing results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
- Filing Date
- 2021-06-04
- Publication Date
- 2026-07-07
AI Technical Summary
Existing AI-based ISP imaging processing models have low performance and high complexity, making it difficult to meet the needs of practical applications.
An image processing neural network is constructed, including a time-domain UNET subnetwork and a frequency-domain UNET subnetwork. Time-domain and frequency-domain features are extracted respectively, and sRGB images are generated through feature fusion.
The image processor performance has been optimized, the image processing effect has been improved, the defects of time domain feature information have been compensated for, and the frequency domain information of the image has been fully utilized.
Smart Images
Figure CN115512197B_ABST
Abstract
Description
Technical Field
[0001] This application relates to image processing technology, and more particularly to a method, electronic device, apparatus, and computing system for image processing. Background Technology
[0002] Currently, the camera functions of mobile terminals are becoming increasingly sophisticated, and their camera performance is getting closer and closer to that of SLR cameras. This is mainly due to the important role played by the image signal processor (ISP) built into the mobile terminal system.
[0003] Traditional ISP imaging processing workflows consist of multiple cascaded image processing modules completed in stages: image depigmentation, image noise reduction, white balance and color space transformation, color enhancement, and tone mapping. These cascaded steps map the original RAW image to the sRGB (standard Red Green Blue) universal color standard, helping people obtain color images that are more consistent with the characteristics of human vision.
[0004] In recent years, against the backdrop of the rapid development of artificial intelligence (AI) technology and the increasing prominence of the advantages of image big data, researchers have begun to consider using AI technologies such as convolutional neural networks (CNN) to replace the traditional multi-stage hierarchical ISP imaging processing architecture.
[0005] However, existing AI-based ISP imaging processing models have low performance and high complexity, making it difficult for trained models to meet the needs of practical applications, and they urgently need to be optimized. Summary of the Invention
[0006] To address the aforementioned technical problems, embodiments of this application aim to provide a method, electronic device, apparatus, and computing system for image processing.
[0007] The technical solution of this application is implemented as follows:
[0008] Firstly, a method for image processing is provided, comprising:
[0009] Construct an image processing neural network, which includes a time-domain UNET subnetwork and a frequency-domain UNET subnetwork;
[0010] Obtain the raw RAW image;
[0011] The original RAW image is input into the image processing neural network;
[0012] The image processing neural network is used to process the original RAW image. The processing includes: extracting temporal features from the original RAW image using the temporal UNET sub-network to obtain a temporal feature map; extracting frequency features from the original RAW image using the frequency UNET sub-network to obtain a frequency feature map; and fusing the temporal feature map and the frequency feature map to output an sRGB image.
[0013] Secondly, an electronic device is provided, comprising:
[0014] Image processing neural network, comprising a time-domain UNET subnetwork, a frequency-domain UNET subnetwork, and a fusion subnetwork; and
[0015] A processor is configured to process an original RAW image using the image processing neural network, the processing including: extracting temporal features from the original RAW image using the temporal UNET sub-network to obtain a temporal feature map; extracting frequency features from the original RAW image using the frequency UNET sub-network to obtain a frequency feature map; and fusing the temporal feature map and the frequency feature map using the fusion sub-network to output an sRGB image.
[0016] Thirdly, an apparatus for image processing includes:
[0017] The building module is used to construct an image processing neural network, which includes a time-domain UNET subnetwork and a frequency-domain UNET subnetwork;
[0018] The acquisition module is used to acquire raw RAW images;
[0019] The input module is used to input the original RAW image into the image processing neural network;
[0020] The processing module is used to process the original RAW image using the image processing neural network. The processing includes: extracting temporal features from the original RAW image using the temporal UNET sub-network to obtain a temporal feature map; extracting frequency features from the original RAW image using the frequency UNET sub-network to obtain a frequency feature map; and fusing the temporal feature map and the frequency feature map to output an sRGB image.
[0021] Fourthly, a computing system is provided, the computing system comprising: a processor and a memory configured to store computer programs capable of running on the processor.
[0022] Wherein, the processor is configured to execute the steps of the aforementioned method when running the computer program.
[0023] Fifthly, a computer storage medium is also provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the aforementioned method.
[0024] This application provides a method, electronic device, apparatus, and computing system for image processing. The method includes: inputting a raw RAW image into an image processing neural network; extracting temporal features from the raw RAW image using a temporal UNET sub-network to obtain a temporal feature map; extracting frequency features from the raw RAW image using a frequency UNET sub-network to obtain a frequency feature map; and fusing the temporal and frequency feature maps to output an sRGB image. Thus, when mapping the raw RAW image to an sRGB image, the parallel temporal and frequency UNET sub-networks, which extract temporal and frequency feature maps respectively, yield a frequency feature map that compensates for the information deficiencies of the temporal feature map. By comprehensively utilizing both temporal and frequency information of the image, the performance of the image processor is further optimized, and the image processing effect is improved. Attached Figure Description
[0025] Figure 1 This is a first flowchart illustrating the image processing method in an embodiment of this application;
[0026] Figure 2 This is a schematic diagram of the first component structure of the image processing neural network in an embodiment of this application;
[0027] Figure 3 This is a schematic diagram of the composition structure of the UNET sub-network in the embodiments of this application;
[0028] Figure 4 This is a second flowchart illustrating the image processing method in an embodiment of this application;
[0029] Figure 5 This is a flowchart illustrating the image processing neural network construction method in an embodiment of this application;
[0030] Figure 6 This is a schematic diagram of the NAS search process in an embodiment of this application;
[0031] Figure 7 This is a schematic diagram of the structure of the second downsampling module in an embodiment of this application;
[0032] Figure 8 This is a schematic diagram of the structure of the second upsampling module in an embodiment of this application;
[0033] Figure 9 This is a schematic diagram of the second component structure of the image processing neural network in an embodiment of this application;
[0034] Figure 10 This is a schematic diagram of the composition structure of the electronic device in the embodiments of this application;
[0035] Figure 11 This is a schematic diagram of the composition of the apparatus for image processing in the embodiments of this application;
[0036] Figure 12 This is a schematic diagram of the composition structure of a computing system according to an embodiment of this application. Detailed Implementation
[0037] In order to gain a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this application.
[0038] Figure 1 This is a first flowchart illustrating the image processing method in an embodiment of this application, as shown below. Figure 1 As shown, the method may specifically include:
[0039] Step 101: Construct an image processing neural network, which includes a time-domain UNET subnetwork and a frequency-domain UNET subnetwork;
[0040] In order to fully exploit the temporal and frequency domain information of images, this application provides a novel image processing neural network for converting RAW images into sRGB images.
[0041] Here, the time-frequency domain subnetworks in the image processing neural network are all designed based on the UNET network structure. UNET is a deep learning framework that typically includes encoding and decoding processes, with the input and output dimensions remaining constant.
[0042] Figure 2 This is a schematic diagram of the first component structure of the image processing neural network in an embodiment of this application, as shown below. Figure 2 As shown, the image processing neural network 20 includes: a temporal UNET subnetwork 201, a frequency domain UNET subnetwork 202, and a fusion subnetwork 203. The temporal UNET subnetwork 201 is used to extract temporal features from the RAW image to obtain temporal features; the frequency domain UNET subnetwork 202 is used to extract frequency features from the RAW image to obtain frequency domain features; and the fusion subnetwork 203 is used to fuse the temporal and frequency domain features to obtain an sRGB image.
[0043] Figure 3 This is a schematic diagram of the composition structure of the UNET sub-network in an embodiment of this application, as shown below. Figure 3As shown, the UNET subnetwork is divided into two parts: the encoder on the left and the decoder on the right. The encoder contains a downsampling module, and the decoder contains an upsampling module. There are skip connections between each downsampling and upsampling module. The RAW image is processed by the encoder through convolutional layers (e.g., Conv+BN+LReLU) and downsampling layers (e.g., Conv with stride = 2 / BN / LreLU) to reduce the image size and extract some shallow features. The downsampled feature map is then processed by convolutional layers (e.g., Conv+BN+LReLU) for identity mapping, and then by the decoder through deconvolutional layers (e.g., Transpose_Conv+BN+LReLU) and upsampling layers (containing upsampling operators) to obtain deeper features. The skip connections between each downsampling and upsampling module combine the feature maps obtained in the encoding and decoding stages, integrating both deep and shallow features to refine the image.
[0044] It should be noted that, Figure 3 The UNET subnetwork undergoes three downsampling and three upsampling operations. In practical applications, the number of downsampling and upsampling operations can be flexibly set according to image processing requirements, and this application embodiment does not impose specific limitations.
[0045] For example, in some embodiments, constructing an image processing neural network may specifically include: pre-constructing a time-domain UNET subnetwork and a frequency-domain UNET subnetwork to generate an initial image processing neural network; obtaining a training image set; and training the image processing neural network using the training image set until the loss function meets the convergence condition to obtain a trained image processing neural network.
[0046] This can be understood as follows: first, construct a time-domain UNET subnetwork and a frequency-domain UNET subnetwork respectively; connect the two subnetworks in parallel to obtain an image processing neural network; then train the image processing neural network to obtain the final network model that can be used for image processing.
[0047] For example, in some embodiments, the loss function of the image processing neural network is: argmin(L1+L cd ); where L1 is the norm loss term, L cd This represents the loss due to color chromaticity difference.
[0048] Step 102: Obtain the raw RAW image;
[0049] Here, the raw RAW image is the original data captured by a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor and converted into a digital signal.
[0050] It should be noted that the image processing neural network provided in this application can map to a single frame of RAW image or to multiple frames of RAW images in a video.
[0051] Step 103: Input the original RAW image into the image processing neural network;
[0052] Step 104: Process the original RAW image using the image processing neural network;
[0053] Here, the processing includes:
[0054] Step 1041: Use the temporal UNET sub-network to extract temporal features from the original RAW image to obtain a temporal feature map;
[0055] For example, in some embodiments, the time-domain UNET subnet can use a traditional UNET network structure (such as... Figure 3 The temporal feature extraction can be achieved by using the first downsampling module and the second upsampling module obtained by Neural Architecture Search (NAS) to replace the downsampling module and the upsampling module in the traditional UNET network model.
[0056] Step 1042: Use the frequency domain UNET sub-network to extract frequency domain features from the original RAW image to obtain a frequency domain feature map;
[0057] For example, in some embodiments, the frequency domain UNET subnetwork and the time domain UNET subnetwork have similar structures, the difference being that the downsampling module and the upsampling module are replaced by a second downsampling module based on wavelet transform and a second upsampling module based on inverse wavelet transform, respectively.
[0058] In some embodiments, the downsampling module and upsampling module in the frequency domain UNET subnetwork can also be replaced by a second downsampling module based on Fourier transform and a second upsampling module based on inverse Fourier transform.
[0059] Step 1043: Fuse the time-domain feature map and the frequency-domain feature map to output an sRGB image.
[0060] For example, feature averaging is performed on the time-domain feature map and the frequency-domain feature map to obtain an sRGB image.
[0061] The image processing method provided in this application embodiment can be applied to an image signal processor (ISP). The ISP is configured in an electronic device with shooting function. The ISP maps the raw RAW image acquired by the CMOS or CCD image sensor to an sRGB image, reducing manual intervention in the ISP image processing process.
[0062] By adopting the above technical solution, when mapping the original RAW image to an sRGB image, time-domain feature extraction and frequency-domain feature extraction are performed through parallel time-domain UNET subnetwork and frequency-domain UNET subnetwork, respectively. The obtained frequency-domain features can make up for the information defects of time-domain features, make full use of the frequency-domain information of the image, further optimize the performance of the image processor, and improve the image processing effect.
[0063] Figure 4 This is a second flowchart illustrating the image processing method in an embodiment of this application, as shown below. Figure 4 As shown, the method may specifically include:
[0064] Step 401: Construct an image processing neural network, which includes a time-domain UNET subnetwork and a frequency-domain UNET subnetwork;
[0065] Step 402: Obtain the raw RAW image;
[0066] Step 403: Input the original RAW image into the temporal UNET sub-network, and use the temporal UNET sub-network to extract temporal features from the original RAW image to obtain a temporal feature map;
[0067] For example, in some embodiments, the temporal UNET subnetwork is searched based on NAS and used to extract temporal feature maps of the image.
[0068] For example, in some embodiments, a first downsampling module and a first upsampling module are automatically searched based on a neural network structure search; an encoder for the temporal UNET subnetwork is constructed using at least one of the first downsampling modules, and a decoder for the temporal UNET subnetwork is constructed using at least one of the first upsampling modules, thereby obtaining the temporal UNET subnetwork.
[0069] This can be understood as using the first downsampling module and the first upsampling module obtained through Neural Architecture Search (NAS) to replace the traditional UNET network model (such as...). Figure 3 The downsampling module and upsampling module are shown in the figure.
[0070] Step 404: Input the original RAW image into the frequency domain UNET sub-network, and use the frequency domain UNET sub-network to extract frequency domain features from the original RAW image to obtain a frequency domain feature map;
[0071] For example, in some embodiments, the frequency domain UNET subnetwork is constructed based on wavelet transform and inverse wavelet transform algorithms to extract frequency domain feature maps of images.
[0072] For example, in some embodiments, a second downsampling module is constructed based on a wavelet transform algorithm; a second upsampling module is constructed based on an inverse wavelet transform algorithm; an encoder for the frequency domain UNET subnetwork is constructed using at least one of the second downsampling modules; and a decoder for the frequency domain UNET subnetwork is constructed using at least one of the second upsampling modules, thereby obtaining the frequency domain UNET subnetwork.
[0073] Step 405: Perform feature averaging on the time-domain feature map and the frequency-domain feature map;
[0074] Step 406: Output sRGB image.
[0075] Based on the above embodiments, the construction and training of image processing neural networks are further illustrated with examples. Figure 5 This is a flowchart illustrating the image processing neural network construction method in an embodiment of this application, as shown below. Figure 5 As shown, the method specifically includes:
[0076] Step 501: Automatically search for the first downsampling module and the first upsampling module based on the neural network structure;
[0077] The core idea of Neural Architecture Search (NAS) is to use search algorithms to discover the neural network structures needed to solve a specific problem.
[0078] Figure 6 This is a schematic diagram of the NAS search process in an embodiment of this application, such as... Figure 6 As shown, Neural Network Architecture (NAS) mainly consists of three parts: search space, search strategy, and performance evaluation strategy. The principle of NAS is to use a certain search strategy to search for a network structure from a set of candidate neural network structures given a search space. The performance of the searched network structures is then evaluated, using metrics such as accuracy and speed, until the optimal network structure is determined.
[0079] Search strategies include: random search, Bayesian optimization, evolutionary algorithms, reinforcement learning, and gradient-based algorithms.
[0080] Specifically, the automatic search for the first downsampling module and the first upsampling module based on the neural network structure includes:
[0081] A first search space is predefined when automatically searching for the first downsampling module based on the neural network structure, and a second search space is predefined when automatically searching for the first upsampling module based on the neural network structure.
[0082] The first downsampling module is obtained by performing a network search in the first search space;
[0083] The first upsampling module is obtained by performing a network search in the second search space;
[0084] The first search space includes M network nodes, a downsampling operator, and a first identity mapping operator; the second search space includes N network nodes, an upsampling operator, and a second identity mapping operator; M and N are both integers greater than or equal to 2.
[0085] In other words, when searching for the downsampling module and the upsampling module, different search spaces need to be predefined, including the number of network nodes and the types of operators. For example, for the first downsampling module, the first search space includes the option to select 4 network nodes, a downsampling operator, and a nonlinear operator. For the first upsampling module, the first search space includes the option to select 4 network nodes, an upsampling operator, and a nonlinear operator.
[0086] In some embodiments, the objective function for automatically searching the first downsampling module and the first upsampling module based on the neural network structure includes: a loss function and constraints;
[0087] The constraints include: number of parameters < r1, latency < r2, and floating-point operation volume < r3.
[0088] Wherein, the number of parameters refers to the total number of network parameters of the time-domain UNET sub-network, the latency refers to the latency generated during the inference process of the time-domain UNET sub-network, the floating-point operation volume is the number of floating-point operations per second (FLOPs), which is the total number of floating-point operations of the time-domain UNET sub-network, r1 is the maximum tolerance value of the number of parameters, r2 is the maximum tolerance value of the latency, and r3 is the maximum tolerance value of the floating-point operation volume.
[0089] Here, when NAS searches for the first downsampling module and the first upsampling module, in order to make the final network model more lightweight, this application introduces some hardware-related constraints in the loss function, namely, constraining the network parameters, operator delay and / or floating-point operation volume with limited constraints, so that the searched network model is more lightweight, while also taking into account accuracy and efficiency, thereby making the network model easier to deploy on mobile devices such as mobile phones, and effectively reducing the computing power consumption of mobile devices.
[0090] For example, the loss function is argmin(L1+L) cd ), where L1 is the norm loss term, L cd This represents the loss due to color chromaticity difference.
[0091] Here, when NAS searches for the first downsampling module and the first upsampling module, in order to make the final network model more lightweight, this application adopts some hardware-related constraints in the definition of the loss function, namely, constraints on the number of network parameters, operator delay, and floating-point operation volume, so that the searched network model is more lightweight, thereby making the model easier to deploy on mobile devices such as mobile phones for inference operations.
[0092] Step 502: Construct an encoder for the temporal UNET sub-network using at least one of the first downsampling modules, and construct a decoder for the temporal UNET sub-network using at least one of the first upsampling modules to obtain the temporal UNET sub-network;
[0093] Here, the first downsampling module found in the search is used to replace... Figure 3 The traditional downsampling module in the encoder is replaced by the first upsampling module found in the search. Figure 3 The traditional upsampling module in the decoder is used to obtain the time-domain UNET subnetwork.
[0094] Step 503: Construct the second downsampling module based on the wavelet transform algorithm, and construct the second upsampling module based on the inverse wavelet transform algorithm;
[0095] For example, the second downsampling module includes: a first convolutional layer, a discrete wavelet transform layer, and a concatenation layer; wherein, the input feature map of the second downsampling module is input to the first convolutional layer to obtain a first intermediate feature map; the first intermediate feature map is input to the discrete wavelet transform layer to obtain a second intermediate feature map; the first intermediate feature map and the second intermediate feature map are input to the concatenation layer to obtain the output feature map of the second downsampling module.
[0096] Here, the first convolutional layer is used to downsample the input feature map, the discrete wavelet transform layer is used to extract the frequency domain features of the first intermediate feature map, and the concatenation layer is used to concatenate the first and second intermediate feature maps of the same size.
[0097] Figure 7 This is a schematic diagram of the structure of the second downsampling module in an embodiment of this application, as shown below. Figure 7 As shown, the feature map is input into the first convolutional layer (e.g., Conv with stride = 2 / BN / LreLU) for downsampling. The output feature map passes through two paths: one is the Discrete Wavelet Transform (DWT) layer, and the other is concatenated with the feature map output from the DWT layer to obtain the output feature map of the second downsampling module.
[0098] Specifically, the second upsampling module includes: a time-frequency channel separation layer, a second convolutional layer, a third convolutional layer, an inverse discrete wavelet transform layer, and a stitching layer; wherein, the input feature map of the second upsampling module is input to the time-frequency channel separation layer to obtain a first time-domain feature map and a first frequency-domain feature map; the first time-domain feature map is input to the second convolutional layer to obtain a third intermediate feature map, and the first frequency-domain feature map is sequentially passed through the third convolutional layer, the inverse discrete wavelet transform layer, and the third convolutional layer to obtain a fourth intermediate feature map; the third intermediate feature map and the fourth intermediate feature map are input to the stitching layer to obtain the output feature map of the second upsampling module.
[0099] Here, the time-frequency channel separation layer is used to separate the time-domain channel and frequency-domain channel of the input feature map, the second convolutional layer is used to upsample the first time-domain feature map, the third convolutional layer, the inverse discrete wavelet transform layer and the third convolutional layer are combined to perform inverse wavelet transform on the first frequency-domain feature to obtain more frequency-domain features, and the stitching layer is used to stitch together the third intermediate feature map and the fourth intermediate feature map of the same size.
[0100] Figure 8 This is a schematic diagram of the structure of the second upsampling module in an embodiment of this application, as shown below. Figure 8As shown, the input feature map first passes through a time-frequency channel separation layer to separate the time-domain and frequency-domain channels. The separated time-domain feature map is then processed by a second convolutional layer (e.g., Conv with stride = 2 / BN / LreLU). The frequency-domain feature map then passes through a third convolutional layer (e.g., Conv with stride = 1), an inverse discrete wavelet transform (IDWT) layer, and another third convolutional layer. The two processed feature maps are then concatenated to obtain the output feature map of the second upsampling module.
[0101] In some embodiments, step 503 can also be replaced by: constructing the second downsampling module based on the Fourier transform algorithm, and constructing the second upsampling module based on the inverse Fourier transform algorithm;
[0102] Accordingly, the second downsampling module includes: a first convolutional layer, a discrete Fourier transform layer, and a stitching layer. The second upsampling module includes: a time-frequency channel separation layer, a second convolutional layer, a third convolutional layer, an inverse discrete Fourier transform layer, and a stitching layer.
[0103] Step 504: Construct an encoder for the frequency domain UNET sub-network using at least one of the second downsampling modules, and construct a decoder for the frequency domain UNET sub-network using at least one of the second upsampling modules to obtain the frequency domain UNET sub-network;
[0104] Here, the second downsampling module found in the search is used to replace... Figure 3 The traditional downsampling module in the encoder is replaced by the found second upsampling module. Figure 3 The traditional upsampling module in the decoder yields the frequency domain UNET subnetwork.
[0105] Step 505: Construct the image processing neural network using the completed time-domain UNET subnetwork and frequency-domain UNET subnetwork;
[0106] Specifically, the time-domain UNET subnetwork and the frequency-domain UNET subnetwork are connected in parallel, and the outputs of the two subnetworks are fused through a feature fusion module to obtain an image processing neural network.
[0107] Figure 9 This is a schematic diagram of the second component structure of the image processing neural network in an embodiment of this application, as shown below. Figure 9As shown, in the time-domain UNET subnetwork, the first downsampling module consists of the traditional Conv+BN+LReLU operator and the Conv_stride=2+BN+LReLU operator, and the first upsampling module consists of the traditional Depth2Space operator and the Upsampling operator. The frequency-domain UNET subnetwork is similar to the time-domain UNET subnetwork, except that the second downsampling module consists of the Conv+BN+LReLU operator and the DWT_downsampling operator, and the second upsampling module consists of the Transpose_Conv+BN+LReLU operator and the IDWT_upsampling operator. Finally, the outputs of the time-domain UNET subnetwork and the frequency-domain UNET subnetwork are fused using feature averaging to output the final sRGB image.
[0108] In other words, time-domain UNET subnets can use traditional UNET network structures (such as...) Figure 3 (As shown) to achieve temporal feature extraction. In other embodiments, in the temporal UNET subnetwork, the first downsampling module and the first upsampling module can also be obtained in the early stage through neural network structure search.
[0109] Step 506: Obtain the training image set;
[0110] Step 507: Train the image processing neural network using the training image set until the loss function meets the convergence condition, and obtain the trained image processing neural network.
[0111] The aforementioned image processing neural network, combined with NAS, fully leverages the advantages of these cutting-edge artificial intelligence methods to improve the processor's image processing performance. The added frequency-domain UNET subnetwork obtains more frequency-domain information, and the fusion of time and frequency information, along with the use of frequency-domain information, compensates for the shortcomings of classic deep networks that only utilize time-domain information while neglecting frequency-domain information, thus optimizing image processor performance and improving image processing results.
[0112] This application also provides an electronic device, such as... Figure 10 As shown, the electronic device 100 includes: an image processing neural network 1001, which includes a time-domain UNET sub-network 10011, a frequency-domain UNET sub-network 10012, and a fusion sub-network 10013; and
[0113] Processor 1002 is configured to process the original RAW image using the image processing neural network, the processing including:
[0114] Temporal feature maps are obtained by extracting temporal features from the original RAW image using the temporal UNET sub-network 10011.
[0115] Frequency domain feature maps are obtained by extracting frequency domain features from the original RAW image using the frequency domain UNET subnetwork 10012; and
[0116] The time-domain feature map and the frequency-domain feature map are fused using the fusion sub-network 10013 to output an sRGB image.
[0117] In some embodiments, the encoder of the time-domain UNET subnetwork 10011 includes at least one first downsampling module, and the decoder of the time-domain UNET subnetwork 10011 includes at least one first upsampling module.
[0118] The first downsampling module and the first upsampling module were automatically searched based on neural network structure search.
[0119] In some embodiments, the automatic search for the first downsampling module and the first upsampling module based on the neural network structure search includes:
[0120] A first search space is predefined when automatically searching for the first downsampling module based on the neural network structure, and a second search space is predefined when automatically searching for the first upsampling module based on the neural network structure.
[0121] The first downsampling module is obtained by performing a network search in the first search space;
[0122] The first upsampling module is obtained by performing a network search in the second search space;
[0123] The first search space includes M network nodes, a downsampling operator, and a first identity mapping operator; the second search space includes N network nodes, an upsampling operator, and a second identity mapping operator; M and N are both integers greater than or equal to 2.
[0124] In some embodiments, the objective function for automatically searching the first downsampling module and the first upsampling module based on the neural network structure includes: a loss function and constraints;
[0125] The constraints include: number of parameters < r1, latency < r2, and floating-point operation volume < r3.
[0126] Wherein, the number of parameters refers to the total number of network parameters of the time-domain UNET sub-network, the latency refers to the latency generated by the time-domain UNET sub-network during inference, the floating-point operation volume is the total floating-point operation volume of the time-domain UNET sub-network, r1 is the maximum tolerance value of the number of parameters, r2 is the maximum tolerance value of the latency, and r3 is the maximum tolerance value of the floating-point operation volume.
[0127] In some embodiments, the encoder of the frequency domain UNET subnetwork includes at least one second downsampling module, and the decoder of the frequency domain UNET subnetwork includes at least one second upsampling module.
[0128] The second downsampling module includes a first convolutional layer, a discrete wavelet transform layer, and a splicing layer;
[0129] Specifically, the input feature map of the second downsampling module is input into the first convolutional layer to obtain the first intermediate feature map; the first intermediate feature map is input into the discrete wavelet transform layer to obtain the second intermediate feature map; the first intermediate feature map and the second intermediate feature map are input into the concatenation layer to obtain the output feature map of the second downsampling module.
[0130] The second upsampling module includes: a time-frequency channel separation layer, a second convolutional layer, a third convolutional layer, an inverse discrete wavelet transform layer, and a splicing layer;
[0131] Specifically, the input feature map of the second upsampling module is input to the time-frequency channel separation layer to obtain a first time-domain feature map and a first frequency-domain feature map; the first time-domain feature map is input to the second convolutional layer to obtain a third intermediate feature map, and the first frequency-domain feature map is sequentially passed through the third convolutional layer, the inverse discrete wavelet transform layer, and the third convolutional layer to obtain a fourth intermediate feature map; the third intermediate feature map and the fourth intermediate feature map are input to the splicing layer to obtain the output feature map of the second upsampling module.
[0132] In some embodiments, the second downsampling module is constructed based on the Fourier transform algorithm, and the second upsampling module is constructed based on the inverse Fourier transform algorithm;
[0133] Accordingly, the second downsampling module includes: a first convolutional layer, a discrete Fourier transform layer, and a stitching layer. The second upsampling module includes: a time-frequency channel separation layer, a second convolutional layer, a third convolutional layer, an inverse discrete Fourier transform layer, and a stitching layer.
[0134] It should be noted that after constructing the image processing neural network using the completed time-domain UNET subnetwork and the completed frequency-domain UNET subnetwork, the image processing neural network still needs to be trained using the training image set until the loss function meets the convergence condition, thus obtaining the trained image processing neural network.
[0135] In some embodiments, the processor includes at least one of the following: a central processing unit (CPU); a graphics processing unit (GPU); a digital signal processor (DSP); an image signal processor (ISP); and a neural network processing unit (NPU).
[0136] It should be noted that the image processing neural network described in the embodiments of this application can also be deployed on any type of hardware computing unit.
[0137] This application also provides an apparatus for image processing, such as... Figure 11 As shown, the device 110 includes:
[0138] Module 1101 is used to construct an image processing neural network, which includes a time-domain UNET subnetwork and a frequency-domain UNET subnetwork.
[0139] Acquisition module 1102 is used to acquire raw RAW images;
[0140] Input module 1103 is used to input the original RAW image into the image processing neural network;
[0141] Processing module 1104 is used to process the original RAW image using the image processing neural network, the processing including:
[0142] The original RAW image is subjected to temporal feature extraction using the temporal UNET sub-network to obtain a temporal feature map;
[0143] The original RAW image is subjected to frequency domain feature extraction using the frequency domain UNET sub-network to obtain a frequency domain feature map;
[0144] The time-domain feature map and the frequency-domain feature map are fused to output an sRGB image.
[0145] In some embodiments, the construction module 1101 is further configured to: automatically search for a first downsampling module and a first upsampling module based on a neural network structure search; construct an encoder for the temporal UNET subnetwork using at least one of the first downsampling modules; construct a decoder for the temporal UNET subnetwork using at least one of the first upsampling modules; and obtain the temporal UNET subnetwork.
[0146] In some embodiments, the construction module 1101 is further configured to: predefine a first search space corresponding to automatically searching for a first downsampling module based on the neural network structure, and a second search space corresponding to automatically searching for a first upsampling module based on the neural network structure; perform a network search in the first search space to obtain the first downsampling module; and perform a network search in the second search space to obtain the first upsampling module.
[0147] The first search space includes M network nodes, a downsampling operator, and a first identity mapping operator; the second search space includes N network nodes, an upsampling operator, and a second identity mapping operator; M and N are both integers greater than or equal to 2.
[0148] In some embodiments, the objective function for automatically searching the first downsampling module and the first upsampling module based on the neural network structure includes: a loss function and constraints;
[0149] The constraints include: number of parameters < r1, latency < r2, and floating-point operation volume < r3.
[0150] Wherein, the number of parameters refers to the total number of network parameters of the time-domain UNET sub-network, the latency refers to the latency generated by the time-domain UNET sub-network during inference, the floating-point operation volume is the total floating-point operation volume of the time-domain UNET sub-network, r1 is the maximum tolerance value of the number of parameters, r2 is the maximum tolerance value of the latency, and r3 is the maximum tolerance value of the floating-point operation volume.
[0151] In some embodiments, the construction module 1101 is further configured to: construct a second downsampling module based on a wavelet transform algorithm; construct a second upsampling module based on an inverse wavelet transform algorithm; construct an encoder for the frequency domain UNET subnetwork using at least one of the second downsampling modules; construct a decoder for the frequency domain UNET subnetwork using at least one of the second upsampling modules; and obtain the frequency domain UNET subnetwork.
[0152] In some embodiments, the second downsampling module includes: a first convolutional layer, a discrete wavelet transform layer, and a concatenation layer; wherein, the input feature map of the second downsampling module is input to the first convolutional layer to obtain a first intermediate feature map; the first intermediate feature map is input to the discrete wavelet transform layer to obtain a second intermediate feature map; the first intermediate feature map and the second intermediate feature map are input to the concatenation layer to obtain the output feature map of the second downsampling module;
[0153] The second upsampling module includes: a time-frequency channel separation layer, a second convolutional layer, a third convolutional layer, an inverse discrete wavelet transform layer, and a stitching layer; wherein, the input feature map of the second upsampling module is input to the time-frequency channel separation layer to obtain a first time-domain feature map and a first frequency-domain feature map; the first time-domain feature map is input to the second convolutional layer to obtain a third intermediate feature map, and the first frequency-domain feature map is sequentially passed through the third convolutional layer, the inverse discrete wavelet transform layer, and the third convolutional layer to obtain a fourth intermediate feature map; the third intermediate feature map and the fourth intermediate feature map are input to the stitching layer to obtain the output feature map of the second upsampling module.
[0154] In some embodiments, the construction module 1101 is further configured to: train the image processing neural network after constructing the image processing neural network, the training including:
[0155] Obtain a training image set; use the training image set to train the image processing neural network until the loss function meets the convergence condition, thus obtaining the trained image processing neural network.
[0156] For example, in some embodiments, the loss function of the image processing neural network is: argmin(L1+L cd ); where L1 is the norm loss term, L cd This represents the loss due to color chromaticity difference.
[0157] Using the above-mentioned device, when mapping the original RAW image to an sRGB image, time-domain feature map extraction and frequency-domain feature map extraction are performed through parallel time-domain UNET sub-networks and frequency-domain UNET sub-networks, respectively. The obtained frequency-domain feature map can make up for the information defects of the time-domain feature map. By comprehensively utilizing the time-domain and frequency-domain information of the image, the performance of the image processor is further optimized and the image processing effect is improved.
[0158] This application also provides a computing system, such as... Figure 12 As shown, the computing system 120 includes a processor 1201 and a memory 1202 configured to store computer programs capable of running on the processor 1201.
[0159] When the processor 1201 is configured to run a computer program, it executes the method steps described in the foregoing embodiments.
[0160] The processor 1201 includes at least one of the following: a central processing unit (CPU); a graphics processing unit (GPU); a digital signal processor (DSP); an image signal processor (ISP); and a neural network processor (NPU).
[0161] Of course, in practical applications, such as Figure 12 As shown, the various components in this computing system are coupled together via a bus system. It can be understood that the bus system is used to enable communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus, and a status signal bus.
[0162] The aforementioned memory can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provides instructions and data to the processor.
[0163] The electronic devices described in this application may include mobile phones, tablets, laptops, handheld computers, personal digital assistants (PDAs), portable media players (PMPs), wearable devices, smart bracelets, cameras, etc.
[0164] In an exemplary embodiment, this application also provides a computer-readable storage medium, such as a memory including a computer program, which can be executed by a processor of an electronic device to perform the steps of the aforementioned method.
[0165] It should be understood that the terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items. The expressions “having,” “may have,” “comprising,” and “including,” or “may include” and “may contain” used herein may be used to indicate the presence of a corresponding feature (e.g., an element such as a number, function, operation, or component), but do not exclude the presence of additional features.
[0166] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and are not necessarily used to describe a specific order or sequence. For example, without departing from the scope of this invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information.
[0167] The technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.
[0168] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatus, and devices can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical, or other forms.
[0169] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.
[0170] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.
[0171] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.
Claims
1. A method for image processing, comprising: Construct an image processing neural network, which includes a time-domain UNET subnetwork and a frequency-domain UNET subnetwork; Obtain the raw RAW image; The original RAW image is input into the image processing neural network; The original RAW image is processed using the image processing neural network, the processing including: The original RAW image is subjected to temporal feature extraction using the temporal UNET sub-network to obtain a temporal feature map; The original RAW image is subjected to frequency domain feature extraction using the frequency domain UNET sub-network to obtain a frequency domain feature map; The time-domain feature map and the frequency-domain feature map are fused to output an sRGB image; The construction of the image processing neural network includes: Automatic search for the first downsampling module and the first upsampling module based on neural network structure search; An encoder for the time-domain UNET subnetwork is constructed using at least one of the first downsampling modules, and a decoder for the time-domain UNET subnetwork is constructed using at least one of the first upsampling modules, thereby obtaining the time-domain UNET subnetwork; The first downsampling module is obtained by automatically searching the first search space based on neural network structure search; The first upsampling module obtains the second search space automatically based on neural network structure search; The first search space includes: M network nodes, a downsampling operator, and a first identity mapping operator; the second search space includes: N network nodes, an upsampling operator, and a second identity mapping operator; M and N are both integers greater than or equal to 2.
2. The method according to claim 1, characterized in that, The objective function used to automatically search the first downsampling module and the first upsampling module based on the neural network structure includes: a loss function and constraints; The constraints include: the number of parameters. Delay and floating-point operation volume , Wherein, the number of parameters refers to the total number of network parameters of the time-domain UNET sub-network, the latency refers to the latency incurred by the time-domain UNET sub-network during inference, and the floating-point operation volume refers to the total floating-point operation volume of the time-domain UNET sub-network. This represents the maximum tolerance value for the number of parameters. This is the maximum tolerable delay value. This is the maximum tolerable value for the floating-point operation.
3. The method according to claim 1, characterized in that, The construction of the image processing neural network also includes: A second downsampling module is constructed based on the wavelet transform algorithm; A second upsampling module is constructed based on the inverse wavelet transform algorithm; An encoder for the frequency domain UNET subnetwork is constructed using at least one of the second downsampling modules, and a decoder for the frequency domain UNET subnetwork is constructed using at least one of the second upsampling modules, thereby obtaining the frequency domain UNET subnetwork.
4. The method according to claim 3, characterized in that, The second downsampling module includes: a first convolutional layer, a discrete wavelet transform layer, and a splicing layer; Specifically, the input feature map of the second downsampling module is input into the first convolutional layer to obtain the first intermediate feature map; the first intermediate feature map is input into the discrete wavelet transform layer to obtain the second intermediate feature map; the first intermediate feature map and the second intermediate feature map are input into the concatenation layer to obtain the output feature map of the second downsampling module. The second upsampling module includes: a time-frequency channel separation layer, a second convolutional layer, a third convolutional layer, an inverse discrete wavelet transform layer, and a splicing layer; Specifically, the input feature map of the second upsampling module is input to the time-frequency channel separation layer to obtain a first time-domain feature map and a first frequency-domain feature map; the first time-domain feature map is input to the second convolutional layer to obtain a third intermediate feature map, and the first frequency-domain feature map is sequentially passed through the third convolutional layer, the inverse discrete wavelet transform layer, and the third convolutional layer to obtain a fourth intermediate feature map; the third intermediate feature map and the fourth intermediate feature map are input to the splicing layer to obtain the output feature map of the second upsampling module.
5. The method according to claim 3, characterized in that, The method further includes training the image processing neural network after constructing it, the training comprising: Obtain the training image set; The image processing neural network is trained using the training image set until the loss function meets the convergence condition, thus obtaining the trained image processing neural network.
6. An electronic device, characterized in that, The electronic device includes: Image processing neural network, comprising a time-domain UNET subnetwork, a frequency-domain UNET subnetwork, and a fusion subnetwork; and A processor is configured to process a raw RAW image using the image processing neural network, the processing including: Temporal feature maps are obtained by extracting temporal features from the original RAW image using the temporal UNET sub-network. Frequency domain feature maps are obtained by extracting frequency domain features from the original RAW image using the frequency domain UNET subnetwork; and The time-domain feature map and the frequency-domain feature map are fused using the fusion sub-network to output an sRGB image; The encoder of the time-domain UNET sub-network includes at least one first downsampling module, and the decoder of the time-domain UNET sub-network includes at least one first upsampling module. The first downsampling module obtains the first search space automatically based on neural network structure search; The first upsampling module obtains the second search space automatically based on neural network structure search; The first search space includes: M network nodes, a downsampling operator, and a first identity mapping operator; the second search space includes: N network nodes, an upsampling operator, and a second identity mapping operator; M and N are both integers greater than or equal to 2.
7. The electronic device according to claim 6, characterized in that, The encoder of the frequency domain UNET subnetwork includes at least one second downsampling module, and the decoder of the frequency domain UNET subnetwork includes at least one second upsampling module. The second downsampling module includes a first convolutional layer, a discrete wavelet transform layer, and a splicing layer; Specifically, the input feature map of the second downsampling module is input into the first convolutional layer to obtain the first intermediate feature map; the first intermediate feature map is input into the discrete wavelet transform layer to obtain the second intermediate feature map; the first intermediate feature map and the second intermediate feature map are input into the concatenation layer to obtain the output feature map of the second downsampling module. The second upsampling module includes: a time-frequency channel separation layer, a second convolutional layer, a third convolutional layer, an inverse discrete wavelet transform layer, and a splicing layer; Specifically, the input feature map of the second upsampling module is input to the time-frequency channel separation layer to obtain a first time-domain feature map and a first frequency-domain feature map; the first time-domain feature map is input to the second convolutional layer to obtain a third intermediate feature map, and the first frequency-domain feature map is sequentially passed through the third convolutional layer, the inverse discrete wavelet transform layer, and the third convolutional layer to obtain a fourth intermediate feature map; the third intermediate feature map and the fourth intermediate feature map are input to the splicing layer to obtain the output feature map of the second upsampling module.
8. The electronic device according to claim 6, characterized in that, The processor includes at least one of the following: a central processing unit (CPU); a graphics processing unit (GPU); a digital signal processor (DSP); an image signal processor (ISP); and a neural network processor (NPU).
9. An apparatus for image processing, characterized in that, The device includes: The building module is used to construct an image processing neural network, which includes a time-domain UNET subnetwork and a frequency-domain UNET subnetwork; The acquisition module is used to acquire raw RAW images; The input module is used to input the original RAW image into the image processing neural network; The processing module is used to process the original RAW image using the image processing neural network, the processing including: The original RAW image is subjected to temporal feature extraction using the temporal UNET sub-network to obtain a temporal feature map; The original RAW image is subjected to frequency domain feature extraction using the frequency domain UNET sub-network to obtain a frequency domain feature map; The time-domain feature map and the frequency-domain feature map are fused to output an sRGB image; The building module is further used for: Automatic search for the first downsampling module and the first upsampling module based on neural network structure search; An encoder for the time-domain UNET subnetwork is constructed using at least one of the first downsampling modules, and a decoder for the time-domain UNET subnetwork is constructed using at least one of the first upsampling modules, thereby obtaining the time-domain UNET subnetwork; The first downsampling module obtains the first search space automatically based on neural network structure search; The first upsampling module obtains the second search space automatically based on neural network structure search; The first search space includes: M network nodes, a downsampling operator, and a first identity mapping operator; the second search space includes: N network nodes, an upsampling operator, and a second identity mapping operator; M and N are both integers greater than or equal to 2.
10. The apparatus according to claim 9, characterized in that, The objective function used to automatically search the first downsampling module and the first upsampling module based on the neural network structure includes: a loss function and constraints; The constraints include: the number of parameters. Delay and floating-point operation volume , Wherein, the number of parameters refers to the total number of network parameters of the time-domain UNET sub-network, the latency refers to the latency incurred by the time-domain UNET sub-network during inference, and the floating-point operation volume refers to the total floating-point operation volume of the time-domain UNET sub-network. This represents the maximum tolerance value for the number of parameters. This is the maximum tolerable delay value. This is the maximum tolerable value for the floating-point operation.
11. The apparatus according to claim 9, characterized in that, The building module is further used for: A second downsampling module is constructed based on the wavelet transform algorithm; A second upsampling module is constructed based on the inverse wavelet transform algorithm; An encoder for the frequency domain UNET subnetwork is constructed using at least one of the second downsampling modules, and a decoder for the frequency domain UNET subnetwork is constructed using at least one of the second upsampling modules, thereby obtaining the frequency domain UNET subnetwork.
12. The apparatus according to claim 11, characterized in that, The second downsampling module includes: a first convolutional layer, a discrete wavelet transform layer, and a splicing layer; Specifically, the input feature map of the second downsampling module is input into the first convolutional layer to obtain the first intermediate feature map; the first intermediate feature map is input into the discrete wavelet transform layer to obtain the second intermediate feature map; the first intermediate feature map and the second intermediate feature map are input into the concatenation layer to obtain the output feature map of the second downsampling module. The second upsampling module includes: a time-frequency channel separation layer, a second convolutional layer, a third convolutional layer, an inverse discrete wavelet transform layer, and a splicing layer; Specifically, the input feature map of the second upsampling module is input to the time-frequency channel separation layer to obtain a first time-domain feature map and a first frequency-domain feature map; the first time-domain feature map is input to the second convolutional layer to obtain a third intermediate feature map, and the first frequency-domain feature map is sequentially passed through the third convolutional layer, the inverse discrete wavelet transform layer, and the third convolutional layer to obtain a fourth intermediate feature map; the third intermediate feature map and the fourth intermediate feature map are input to the splicing layer to obtain the output feature map of the second upsampling module.
13. The apparatus according to claim 11, characterized in that, The building module is further configured to: train the image processing neural network after constructing the image processing neural network, the training including: Obtain the training image set; The image processing neural network is trained using the training image set until the loss function meets the convergence condition, thus obtaining the trained image processing neural network.
14. A computing system, characterized in that, The computing system includes: a processor and a memory configured to store computer programs capable of running on the processor. Wherein, when the processor is configured to run the computer program, it performs the steps of the method according to any one of claims 1 to 5.
15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.