Image processing method, device, electronic device and computer readable storage medium

Abstract

The embodiment of the present application discloses an image processing method, device, electronic device and computer-readable storage medium, which can improve the processing efficiency and image effect of image signal processing based on neural network. The method includes: extracting and predicting features based on Bayer domain images through the first network branch in the target image processing network to determine the Bayer prediction features; extracting and predicting features of the first color domain of the Bayer prediction features through the second network branch to determine the first color prediction features; extracting and predicting features of the second color domain of the first color prediction features through the third network branch to determine the second color prediction features; reconstructing the image based on the second color prediction features to determine the reconstructed image corresponding to the original image; wherein the target image processing network is obtained by supervised training of at least one network branch among the initial first network branch, the initial second network branch and the initial third network branch.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] At present, many image signal processing (ISP) solutions based on deep learning use deep learning neural networks to replace some functional modules in traditional ISP. However, due to the strong dependence between ISP functional modules, they need to work together to achieve better image processing effects. Therefore, after some functional modules in ISP are replaced by neural networks, a lot of manual debugging is still required, which is inefficient and difficult to debug to an optimal level. In addition, multiple neural networks will also compete for resources with other functional modules in ISP during operation, reducing image processing efficiency. At present, there are also solutions for training end-to-end convolutional neural networks to complete ISP processing, but this technical solution trains ISP as an image restoration task, which ultimately leads to a large gap between the ISP processing effect of the neural network and the processing effect of the traditional ISP solution, and poor image processing effect. Summary of the invention

[0003] The embodiments of the present application are intended to provide an image processing method, device, electronic device, and computer-readable storage medium that can improve the image processing effect and image processing efficiency of image signal processing based on neural networks.

[0004] The technical solution of this application is implemented as follows:

[0005] The present application embodiment provides an image processing method, which is applied to a target image processing network, including:

[0006] Resampling the original image acquired by the image sensor into a target Bayer format to obtain a Bayer domain image through the first network branch, extracting and predicting features based on the Bayer domain image, and determining Bayer prediction features;

[0007] Extracting and predicting features in a first color domain on the Bayer prediction features through a second network branch to determine a first color prediction feature;

[0008] By using a third network branch, extracting and predicting features of a second color domain on the first color prediction feature, to determine a second color prediction feature;

[0009] Performing image reconstruction based on the second color prediction feature to determine a reconstructed image corresponding to the original image;

[0010] The target image processing network is obtained by performing supervised training on at least one of the initial first network branch, the initial second network branch and the initial third network branch.

[0011] The present application provides an image processing device, including:

[0012] A network prediction module is used to resample the original image collected by the image sensor into a target Bayer format through a first network branch to obtain a Bayer domain image, perform feature extraction and prediction based on the Bayer domain image, and determine a Bayer prediction feature; perform feature extraction and prediction of a first color domain on the Bayer prediction feature through a second network branch to determine a first color prediction feature; perform feature extraction and prediction of a second color domain on the first color prediction feature through a third network branch to determine a second color prediction feature;

[0013] The target image processing network is obtained by performing supervised training on at least one of the initial first network branch, the initial second network branch and the initial third network branch;

[0014] A reconstruction module is used to perform image reconstruction based on the second color prediction feature and determine a reconstructed image corresponding to the original image.

[0015] An embodiment of the present application provides an electronic device, including:

[0016] A memory for storing executable instructions;

[0017] The processor is used to implement the image processing method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0018] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute and implement the image processing method provided in the embodiment of the present application.

[0019] An embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the image processing method provided in the embodiment of the present application.

[0020] The embodiment of the present application provides an image processing method, device, electronic device and computer-readable storage medium, which realizes Bayer domain image processing in the ISP processing flow through the first network branch in the target image processing network; realizes image processing in the first color domain in the ISP processing flow through the second network branch, and realizes image processing in the second color domain in the ISP processing flow through the third network branch; and then reconstructs the image based on the second color prediction feature predicted by the third network branch to obtain the reconstructed image of the ISP, thereby realizing the end-to-end ISP processing flow through the target image processing network. In this way, the complete processing flow of the ISP is uniformly processed end-to-end through the target image processing network, which will not destroy the high coupling between the processes and will not cause resource preemption, thereby improving the image processing effect and image processing efficiency of image signal processing based on neural networks. Furthermore, since the target image processing network is obtained by supervised training of at least one of the initial first network branch, the initial second network branch and the initial third network branch, that is, for network branches used to implement different image processing processes, such as the network branch implementing the Bayer domain processing process, and / or the network branch implementing the first color domain processing process, and / or the network branch implementing the second color domain processing process, corresponding training targets can be set according to different image processing processes, and segmented supervised training can be performed to improve the image processing effect of the corresponding network branch. Thus, the image processing effect of image signal processing based on the neural network is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Schematic diagram of the image processing method provided in the embodiment of the present application Figure 1 ;

[0022] Figure 2 is a schematic diagram of the image signal processing flow;

[0023] Figure 3 Schematic diagram of the image processing method provided in the embodiment of the present application Figure 2 ;

[0024] Figure 4A Schematic diagram of the network structure of the second network branch provided in the embodiment of the present application Figure 1 ;

[0025] Figure 4B Schematic diagram of the network structure of the second network branch provided in the embodiment of the present application Figure 2 ;

[0026] Figure 5 A schematic diagram of the network structure of a region growing module provided in an embodiment of the present application;

[0027] Figure 6Schematic diagram of the image processing method provided in the embodiment of the present application Figure 3 ;

[0028] Fig. 7A Schematic diagram of the network structure of the second network branch provided in the embodiment of the present application Figure 1 ;

[0029] Figure 7B Schematic diagram of the network structure of the second network branch provided in the embodiment of the present application Figure 2 ;

[0030] Figure 8 Schematic diagram 4 of the flow chart of the image processing method provided in the embodiment of the present application;

[0031] Fig. 9 A schematic diagram of the network structure of the third network branch provided in an embodiment of the present application;

[0032] Fig.10 Schematic diagram of the image processing method provided in the embodiment of the present application Figure 5 ;

[0033] Fig.11 Schematic diagram of the image processing method provided in the embodiment of the present application Figure 6 ;

[0034] Fig. 12A A schematic diagram of a process for determining a Bayer domain reconstructed image provided in an embodiment of the present application Figure 1 ;

[0035] Fig. 12B A schematic diagram of a process for determining a Bayer domain reconstructed image provided in an embodiment of the present application Figure 2 ;

[0036] Fig. 12C A schematic diagram of a process for determining a second color domain reconstructed image provided in an embodiment of the present application;

[0037] Fig.13 A schematic diagram of a segmented network structure provided in an embodiment of the present application;

[0038] Fig.14 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0039] Fig.15 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0041] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0042] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0044] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0045] 1) Region Seeds Growing (RSG): RSG is the process of aggregating pixels or sub-regions into larger regions based on pre-defined criteria. The basic idea is to start with a set of growing points (a growing point can be a single pixel or a small area), and merge the adjacent pixels or regions with similar properties to the growing point. Then treat these new pixels as new growing points, continue the above operation, and repeat this process until no pixels that meet the conditions can be included. At this time, it means that the growing point can no longer grow, and a region is grown. The similarity between the growing point and the similar area can be judged based on image information such as grayscale value, texture, and color.

[0046] 2) LSC (Lens Shading Correction) is an important component of the ISP algorithm, which is used to correct the lens shading phenomenon. Lens shading is the phenomenon of uneven image brightness caused by the optical characteristics of the lens and image sensor. Usually, the center of the image is brighter, while the edge is darker. The goal of the LSC algorithm is to correct this uneven brightness so that the image has the same brightness level throughout the field of view. The LSC algorithm calculates a correction table by comparing the actual captured image with the ideal flat illumination image. The correction table can be represented as a one-dimensional or two-dimensional lookup table, and each element in the table represents the brightness correction coefficient of the corresponding pixel position. The correction table is converted into a lookup table for fast lookup and application during real-time image processing. The lookup table can be stored in hardware or software for efficient calculation and correction. In this way, during real-time image processing, the brightness correction coefficient corresponding to the original pixel value in the lookup table can be applied to the original pixel value to achieve correction of each pixel value of the original image.

[0047] 3) Attention mechanism: A computer vision mechanism that focuses on local information in an image, such as a certain image area in an image. As computer vision tasks change, the area of ​​attention often changes. The essence of the attention mechanism is to locate information of interest and suppress useless information. The processing results are usually displayed in the form of probability maps or probability feature vectors. In principle, the attention mechanism can be divided into three types: spatial attention mechanism, channel attention mechanism, and spatial and channel dual attention mechanism. Among them, the spatial attention mechanism focuses on the spatial position in the image, and searches for areas in the image that contribute greatly to the task for processing. The channel attention mechanism focuses on each feature channel in the neural network, models the importance of each feature channel, and then enhances or suppresses different channels for different tasks. The spatial and channel dual attention mechanism combines the two attention mechanisms of space and channel.

[0048] 4) HSV color space domain: H (Hue) represents hue. The HSV color space domain uses degrees to describe hue. For example, red corresponds to 0 degrees, green corresponds to 120 degrees, and blue corresponds to 240 degrees. S (Saturation) represents the saturation, which represents the depth of the color. V (Value) represents the hue, which represents the brightness of the color.

[0049] 5) VGG network: VGG is a convolutional neural network proposed by the Visual Geometry Group. The abbreviation VGG also comes from the name of this group. The VGG network explores the importance of improving the depth of the network to the final image recognition accuracy. At the same time, small convolution kernels are used in VGG to build a deep convolutional network to improve the performance of the network by increasing the depth of the network.

[0050] 6) Norm loss: Also known as L1 loss, also known as Least Absolute Deviations (LAD) or Least Absolute Errors (LAE). Norm loss is used to minimize the sum of the absolute differences between the target value and the estimated value.

[0051] ISP is a general term for an image signal processor, which receives the original data (raw data) output by the image sensor, performs a series of algorithm processing on it, and finally outputs the result image in the YUV and RGB domains. The ISP algorithm processing process will effectively improve the image quality, brightness, color and other aspects of the image signal. The main functions of ISP include linear correction, noise removal, bad pixel removal, interpolation, white balance, automatic exposure control, etc. ISP can restore the details of the scene well under different optical conditions, which largely determines the imaging quality of the camera.

[0052] At present, the complete ISP processing flow based on computer vision involves many algorithm modules, such as Bayer Denoise, Demosaic, Global Tone Mapping (GTM) and Local Tone Mapping (LTM), Spatial Denoise, Temporal Denoise, and Sharpen, etc. There is a strong dependency between different modules. For example, GTM and LTM need to depend on each other, and their synergy can improve the tone and brightness of the image.

[0053] In recent years, with the development of deep learning technology, many ISP solutions based on deep learning have been born. One solution is to combine the traditional ISP solution with the deep learning technology solution, and use the deep learning neural network module to replace some modules in the traditional ISP. This solution still cannot avoid the problems caused by the high coupling between modules, and still requires a lot of manual debugging, and it is difficult to debug the image processing effect to an optimal level. In addition, if there are multiple neural network modules based on deep learning in an ISP, it will also lead to problems such as resource preemption and low operating efficiency.

[0054] At present, there is also a solution that directly learns end-to-end convolutional neural networks to complete ISP processing. This technical solution simply regards ISP as an image restoration task, and it is difficult to ensure that each functional module in image processing has better performance. As a result, the final network effect is far behind the traditional CV solution, which means that the image processing effect is reduced.

[0055] The embodiments of the present application provide an image processing method, device, electronic device and computer-readable storage medium, which can improve the image processing effect and image processing efficiency of image signal processing based on neural network. The exemplary application of the electronic device provided by the embodiments of the present application is described below. The electronic device provided by the embodiments of the present application can be implemented as various types of user terminals such as laptop computers, tablet computers, desktop computers, set-top boxes, mobile devices (for example, mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), and can also be implemented as servers.

[0056] See also Figure 1 , Figure 1 is an optional flow chart of the image processing method provided in the embodiment of the present application, which will be combined with Figure 1 The steps shown are explained.

[0057] S101. Resample the original image captured by the image sensor into a target Bayer format through a first network branch to obtain a Bayer domain image, extract and predict features based on the Bayer domain image, and determine Bayer prediction features.

[0058] In the embodiment of the present application, the target image processing network is a pre-trained neural network for ISP processing. The target image processing network includes a first network branch, a second network branch and a third network branch. Figure 2The ISP processing flow shown in the figure, the common ISP processing flow includes Bayer domain signal processing, RGB domain signal processing and YUV domain signal processing. Among them, the task of Bayer domain signal processing is to correct the data offset caused by the physical characteristics of the image sensor, and interpolate the signal to restore the complete RGB signal. The task of RGB domain signal processing is to compensate for the color and restore the real feeling of the human eye. The task of YUV domain signal processing is to separate the brightness signal and the chrominance signal, process the two signals separately, and perform JPG compression encoding. The above-mentioned first network branch can be obtained by supervised training of the initial first network branch according to the task of Bayer domain signal processing; and / or, the above-mentioned second network branch can be obtained by supervised training of the initial second network branch according to the task of first color domain signal processing, such as the task of RGB domain signal processing; and / or, the above-mentioned third network branch can be obtained by supervised training of the initial third network branch according to the task of second color domain signal processing, such as the task of RGB domain signal processing. That is to say, the target image processing network is obtained by supervised training of at least one network branch among the initial first network branch, the initial second network branch and the initial third network branch.

[0059] In the embodiment of the present application, since the target image processing network is obtained by supervised training of at least one of the initial first network branch, the initial second network branch and the initial third network branch, the target image processing network can be used to implement the ISP function.

[0060] In the embodiment of the present application, the first network branch is obtained by training the initial first network branch, the second network branch is obtained by training the initial second network branch, and the third network branch is obtained by training the initial third network branch. Here, at least one of the initial first network branch, the initial second network branch, and the initial third network branch can be supervised and trained, and the first network branch, the second network branch, and the third network branch can be obtained by continuing to train the supervised trained network branch; or, the initial first network branch, the initial second network branch, and the initial third network branch can be supervised and trained to obtain the first network branch, the second network branch, and the third network branch. The specific selection is based on the actual situation, and the embodiment of the present application is not limited.

[0061] In an embodiment of the present application, the first network branch is used to process a Bayer domain image. Among them, the Bayer domain image can be determined by resampling the original image collected by the image sensor to a target Bayer format. The original image collected by the image sensor is a raw image. Since the pixel units in the image sensor adopt a Bayer arrangement, such an arrangement is a Bayer domain. In the Bayer domain, four neighboring pixels are used as units to form the entire picture, and each unit has two green photosensitive pixels G, one red photosensitive pixel R, and one blue photosensitive pixel B. Due to the difference in image sensor hardware, the arrangement order of the above four photosensitive pixels may include four types: GRBG, BGGR, GBGR, and RGGB. Exemplarily, the target Bayer format may be BGGR. By resampling the original image according to the BGGR format through the first network branch, the original images of different Bayer domain arrangement forms corresponding to different image sensors can be unified into a Bayer domain image in the BGGR format, so as to perform further Bayer domain image processing based on the unified target Bayer format.

[0062] In the embodiment of the present application, the first network branch extracts and predicts features based on the Bayer domain image to determine the Bayer prediction features. The first network branch extracts features based on the Bayer domain image to determine the Bayer features; the first network branch predicts based on the extracted Bayer features, and the Bayer domain signal processing is implemented through the neural network prediction process to obtain the Bayer prediction features.

[0063] In some embodiments, the prediction of the first network branch may include one or more neural network prediction processes corresponding to Bayer domain image signal processing, such as black level correction (Black Level Correction), light offset correction (Shading Correction), green channel correction (G Channels Correction), and white balance (White Balance).

[0064] S102: extract and predict the first color domain of the Bayer prediction feature through the second network branch to determine the first color prediction feature.

[0065] In an embodiment of the present application, the second network branch in the target image processing network performs feature extraction on the Bayer prediction feature output by the first network branch in the first color domain to obtain a first color feature; performs prediction based on the first color feature, implements first color domain signal processing through a neural network prediction process, and obtains a first color prediction feature.

[0066] In some embodiments, the first color domain may be an RGB color space domain. The prediction of the second network branch may include one or more neural network prediction processes corresponding to RGB domain image signal processing, such as color correction and gamma correction. Alternatively, the first color domain may also be other custom color spaces, and the prediction of the second network branch includes the corresponding image signal processing flow in the corresponding color space. The specific selection is made according to the actual situation, and the embodiments of the present application are not limited thereto.

[0067] S103: extract and predict the features of the second color domain on the first color prediction feature through the third network branch to determine the second color prediction feature.

[0068] In an embodiment of the present application, the third network branch in the target image processing network performs feature extraction on the first color prediction feature output by the second network branch in the second color domain to obtain a second color feature; prediction is performed based on the second color feature, and second color domain signal processing is implemented through a neural network prediction process to obtain a second color prediction feature.

[0069] In some embodiments, the second color domain may be a YUV color space domain. The prediction of the third network branch may include one or more neural network prediction processes corresponding to YUV domain image signal processing, such as noise reduction and edge enhancement. Alternatively, the second color domain may also be an HSV color space domain, and the prediction of the second network branch may include an image signal processing flow corresponding to the HSV color space domain, etc. The specific selection is made according to the actual situation, and the embodiments of the present application are not limited.

[0070] S104: Reconstruct an image based on the second color prediction feature to determine a reconstructed image corresponding to the original image.

[0071] In an embodiment of the present application, the second color prediction feature is obtained by performing image signal processing prediction on the Bayer domain image through the first network branch, the second network branch and the third network branch. Image reconstruction is performed based on the second color prediction feature, and the reconstructed image corresponding to the original image is determined as the image signal processing result of the target image processing network for the Bayer domain image.

[0072] In some embodiments, the image reconstruction module may be included in the target image processing network, or may be implemented through another module. The specific selection is made based on actual conditions, and the embodiments of the present application do not limit this.

[0073] It can be understood that the Bayer domain image processing in the ISP processing flow is realized through the first network branch in the target image processing network; the image processing of the first color domain in the ISP processing flow is realized through the second network branch, and the image processing of the second color domain in the ISP processing flow is realized through the third network branch; and then the image is reconstructed based on the second color prediction feature predicted by the third network branch to obtain the reconstructed image of the ISP, thereby realizing the end-to-end ISP processing flow through the target image processing network. In this way, the complete processing flow of the ISP is uniformly processed end-to-end through the target image processing network, which will not destroy the high coupling between the processes, nor will it cause resource preemption, thereby improving the image processing effect and image processing efficiency of image signal processing based on the neural network. In addition, since the target image processing network is obtained by supervised training of at least one network branch among the initial first network branch, the initial second network branch and the initial third network branch, that is, for the network branches used to realize different image processing flows, such as the network branches realizing the Bayer domain processing flow, and / or the network branches realizing the first color domain processing flow, and / or the network branches realizing the second color domain processing flow, corresponding training targets can be set according to different image processing flows, and segmented supervised training can be performed to improve the image processing effect of the corresponding network branches. Thereby, the image processing effect of image signal processing based on neural network is further improved.

[0074] In some embodiments, the first network branch includes: at least one level of Bayer feature extraction module and at least one level of Bayer feature prediction module. Figure 1 In the above S101, the first network branch is used to extract and predict features based on the Bayer domain image, and the process of determining the Bayer prediction features can be as follows: Figure 3 As shown, including S1011-S1012, as follows:

[0075] S1011. Perform at least one level of feature extraction based on the Bayer domain image by at least one level of Bayer feature extraction module to determine Bayer features.

[0076] In the embodiment of the present application, at least one level of Bayer feature extraction module corresponds to at least one scale of Bayer domain feature extraction. The input of the first level of Bayer feature extraction module in the at least one level of Bayer feature extraction module is the Bayer domain image, and the input of each subsequent level of Bayer feature extraction module is the output of the previous level of Bayer feature extraction module. In this way, the first network branch performs scale-by-scale progressive Bayer domain feature extraction based on the Bayer domain image through at least one level of Bayer feature extraction module, and performs feature extraction of the next scale based on the features extracted at the current scale, until feature extraction of at least one scale is completed to obtain a Bayer feature.

[0077] In some embodiments, the first network branch extracts features from the Bayer domain image through the first-level Bayer feature extraction module in at least one level of Bayer feature extraction module to determine the first initial Bayer feature; downsamples and extracts features from the i-1th initial Bayer feature corresponding to the i-1th level Bayer feature extraction module through the i-th level Bayer feature extraction module in at least one level of Bayer feature extraction module to determine the i-th initial Bayer feature, until the Mth initial Bayer feature is determined as the Bayer feature through the M-th level Bayer feature extraction module; i is not less than 2 and not greater than M; M is the number of levels of at least one level of Bayer feature extraction module. M is an integer not less than 1, and M can be taken according to actual conditions, and is not limited in the embodiments of the present application.

[0078] S1012: Perform at least one level of Bayer domain image processing prediction on the Bayer feature by at least one level of Bayer feature prediction module to determine a Bayer prediction feature.

[0079] In an embodiment of the present application, the first network branch performs at least one level of Bayer domain image processing prediction on the Bayer feature through at least one level of Bayer feature prediction module. In the Bayer domain image processing prediction process, prediction is performed through the first level of Bayer feature prediction module, and each level of Bayer feature prediction module after the first level restores the prediction features output by the previous level of Bayer feature prediction module scale by scale and performs Bayer domain image processing prediction based on the predicted features restored at the scale, until the Bayer domain image processing prediction of at least one scale is completed to obtain the Bayer prediction feature.

[0080] In some embodiments, the M-th level Bayer feature extraction module is connected to the first level Bayer feature prediction module in at least one level Bayer feature prediction module; that is, the last level Bayer feature extraction module is connected to the first level Bayer feature prediction module in at least one level Bayer feature prediction module. In this way, the first level Bayer feature prediction module can be used to perform Bayer domain image processing prediction on the Bayer feature to determine the first initial Bayer prediction feature; the i-1th initial Bayer prediction feature corresponding to the i-1th level Bayer feature prediction module is upsampled and predicted by Bayer domain image processing through the i-th level Bayer feature prediction module in at least one level Bayer feature prediction module to determine the i-th initial Bayer prediction feature, until the M-th initial Bayer prediction feature is determined by the M-th level Bayer feature prediction module as the Bayer prediction feature. Among them, the number of levels of the at least one level Bayer feature prediction module is the same as the number of levels of the at least one level Bayer feature extraction module, which is also M levels.

[0081] In some embodiments, the first-level Bayer feature extraction module in the first network branch may include at least one convolution layer (conv) for resampling and an RSG module for feature extraction, where the at least one convolution layer for resampling is used to resample the original image according to the target Bayer format to obtain a Bayer domain image. The i-th level Bayer feature extraction module may include a downsampling module and an RSG module. The first-level Bayer feature prediction module may include an RSG module, and the i-th level Bayer feature prediction module may include an upsampling module and an RSG module. Exemplarily, the downsampling module may be implemented by a convolution layer, and the upsampling module may be implemented by a deconvolution layer (deconv). Figure 4A As shown, the first network branch includes two convolutional layers 40 for resampling, a three-level Bayer feature extraction module and a three-level Bayer feature prediction module. Among them, the first-level Bayer feature extraction module 41 includes an RSG module for feature extraction, and the second-level Bayer feature extraction module 42 and the third-level Bayer feature extraction module 43 respectively include a convolutional layer for downsampling and an RSG module for feature extraction. The first-level Bayer feature prediction module 44 includes an RSG module for feature prediction; the second-level Bayer feature prediction module 45 and the third-level Bayer feature prediction module 46 respectively include a deconvolution layer for upsampling and an RSG module for feature prediction.

[0082] Figure 4A In the figure, the third-level Bayer feature extraction module 43 is connected to the first-level Bayer feature prediction module 44, the original image is input into the two convolutional layers 40 for resampling, and the output Bayer domain image enters the first-level Bayer feature extraction module 41. Through the feature extraction and prediction of the three-level Bayer feature extraction module and the three-level Bayer feature prediction module, the third-level Bayer feature prediction module 46 outputs the Bayer prediction feature to the second network branch.

[0083] In some embodiments, the M-th level Bayer feature extraction module is connected to the first level Bayer feature prediction module in at least one level Bayer feature prediction module; the M-(i-1)-th level Bayer feature extraction module is connected to the i-th level Bayer feature prediction module. The first network branch can perform Bayer domain image processing prediction on the Bayer feature through the first level Bayer feature prediction module to determine the first initial Bayer prediction feature; upsample the i-1th initial Bayer prediction feature corresponding to the i-1th level Bayer feature prediction module through the i-th level prediction module in at least one level Bayer feature prediction module, perform feature fusion on the upsampled i-1th initial Bayer prediction feature and the M-(i-1)th initial Bayer prediction feature, and perform Bayer domain image processing prediction on the fused feature to determine the i-th initial Bayer prediction feature, until the M-th initial Bayer prediction feature is determined by the M-th level Bayer feature prediction module as the Bayer prediction feature.

[0084] For example, Figure 4B As shown, the third-level Bayer feature extraction module 43 is connected to the first-level Bayer feature prediction module 44; the second-level Bayer feature prediction module 45 is connected to the first-level Bayer feature prediction module 44 and the second-level Bayer feature extraction module 42; the third-level Bayer feature prediction module 46 is connected to the second-level Bayer feature prediction module 45 and the third-level Bayer feature extraction module 41. The original image is input into the two convolutional layers 40 for resampling, and the output Bayer domain image enters the first-level Bayer feature extraction module 41. Through the feature extraction of the three-level Bayer feature extraction modules, three initial Bayer features of different scales are obtained respectively. The third initial Bayer feature is predicted by the first-level Bayer feature prediction module 44 to obtain the first initial Bayer prediction feature. The first initial Bayer prediction feature is upsampled by the second-level Bayer feature prediction module 45, the upsampled first initial Bayer prediction feature is fused with the second initial Bayer feature, and the fused feature is processed and predicted by the Bayer domain image to obtain the second initial Bayer prediction feature. The second initial Bayer prediction feature is upsampled by the third-level Bayer feature prediction module 46, the upsampled second initial Bayer prediction feature is fused with the third initial Bayer feature, and the fused feature is subjected to Bayer domain image processing prediction to obtain the third initial Bayer prediction feature as the Bayer prediction feature.

[0085] In some embodiments, the RSG module in the Bayer feature extraction module and the Bayer feature prediction module includes: a region growing network based on a spatial and channel dual attention mechanism. In the region growing network of the RSG module, the spatial attention module based on the spatial attention mechanism and the channel attention module based on the channel attention mechanism can be combined in series or in parallel. Exemplarily, the network structure of the RSG module can be as follows: Figure 5 As shown, it includes: three convolutional layers, a channel attention module, a spatial attention module and a fusion module.

[0086] It can be understood that the multi-level feature extraction and prediction of the Bayer domain image is performed through the multi-level pyramid structure of the first network branch to obtain the Bayer prediction features to input into the second network branch. Since the Bayer prediction features output by the last level Bayer feature prediction module, rather than the processed BGGR image, enter the subsequent second network branch, the semantic information contained in the features has richer dimensions and content than the image content, and sending it to the subsequent branches is conducive to improving the final image processing effect. Furthermore, through the dual attention mechanism network in the RSG module, the accuracy of feature extraction and feature prediction can be improved, thereby improving the image processing performance of the first network branch, and then improving the image processing effect of the target image processing network.

[0087] In some embodiments, the second network branch includes: at least one level first color feature extraction module and at least one level first color feature prediction module. Figure 1 or Figure 3 In the above S102, the second network branch is used to extract and predict the feature of the first color domain of the Bayer prediction feature, and the process of determining the first color prediction feature can be as follows: Figure 6 As shown, it is implemented through S1021-S1022 as follows:

[0088] S1021. Perform at least one level of feature extraction of a first color domain based on the Bayer prediction feature by using at least one level of first color feature extraction module to determine a first color feature.

[0089] In the embodiment of the present application, at least one level of first color feature extraction module corresponds to the first color feature extraction of at least one scale. The input of the first level of first color feature extraction module in at least one level of first color feature extraction module is the Bayer prediction feature, and the input of each level of first color feature extraction module thereafter is the output of the first color feature extraction module of the previous level. In this way, the second network branch performs scale-by-scale progressive first color domain feature extraction based on the Bayer prediction feature through at least one level of first color feature extraction module, performs feature extraction of the next scale based on the feature extracted at the current scale, until the feature extraction of at least one scale is completed, and a first color feature is obtained.

[0090] In some embodiments, the second network branch extracts the first color domain feature of the Bayer prediction feature through the first-level first color feature extraction module in at least one first-level first color feature extraction module to determine the first initial first color feature; downsamples and extracts the j-1th initial first color feature corresponding to the j-1th first color feature extraction module through the j-th first color feature extraction module in at least one first-level first color feature extraction module to determine the j-th initial first color feature, until the K-th initial first color feature is determined as the first color feature through the K-th first color feature extraction module; j is not less than 2 and not greater than K; K is the number of the at least one first color feature extraction module. K is an integer not less than 1, and K can be taken according to actual conditions, and the embodiment of the present application is not limited.

[0091] S1022. Perform at least one level of first color domain image processing prediction on the first color feature through at least one level of first color feature prediction module to determine a first color prediction feature.

[0092] In an embodiment of the present application, the second network branch performs at least one level of first color domain image processing prediction on the first color feature through at least one level of first color feature prediction module. During the first color domain image processing prediction process, prediction is performed through the first level first color feature prediction module. Through each level of first color feature prediction module after the first level, the prediction features output by the previous level first color domain prediction module are restored scale by scale and the first color domain image processing prediction is performed based on the scale-restored prediction features until the first color domain image processing prediction at at least one scale is completed to obtain the first color prediction feature.

[0093] In some embodiments, the K-th level first color feature extraction module is connected to the first level first color feature prediction module in at least one level first color feature prediction module. The second network branch determines the first initial first color prediction feature by performing first color domain image processing prediction based on the first color feature through the first level first color feature prediction module; upsampling and first color domain image processing prediction are performed based on the j-1th initial first color prediction feature corresponding to the j-1th level first color feature prediction module through the j-th level first color feature prediction module in at least one level first color feature prediction module, and determines the j-th initial first color prediction feature, until the K-th first color prediction feature is determined through the K-th level first color feature prediction module as the first color prediction feature; the number of levels of the at least one level first color feature extraction module is the same as the number of levels of the at least one level first color feature prediction module.

[0094] For example, the network structure of the second network branch is the same as the network structure of the first network branch, that is, the network structure of the second network branch can be directly reused. Figure 4A or Figure 4BThe network structure shown in Fig. 7A As shown, the second network branch includes 3 levels of first color feature extraction modules and 3 levels of first color feature prediction modules. Among them, the first level first color feature extraction module 41 includes a RSG module for feature extraction; the second level first color feature extraction module 71 and the third level first color feature extraction module 73 respectively include a convolution layer for downsampling and an RSG module for feature extraction. The first level first color feature prediction module 74 includes an RSG module for feature prediction; the second level first color feature prediction module 75 and the third level first color feature prediction module 76 respectively include a deconvolution layer for upsampling and an RSG module for feature prediction.

[0095] It can be seen that both the first color feature extraction module and the first color feature prediction module include an RSG module, wherein the RSG module includes a region growing network based on a dual attention mechanism of space and channel. Figure 5 As shown, no further description is given here.

[0096] Fig. 7A In the figure, the third-level first color feature extraction module 73 is connected to the first-level first color feature prediction module 74. The Bayer prediction feature is input into the first-level first color feature extraction module 71, and through the feature extraction and prediction of the three-level first color feature extraction module and the three-level first color feature prediction module, the third-level first color feature prediction module 76 outputs the first color prediction feature to the third network branch.

[0097] In some embodiments, the K-th level first color feature extraction module is connected to the first level first color feature prediction module in at least one level first color feature prediction module; the K-(j-1)-th level first color feature extraction module is connected to the j-th level first color feature prediction module. The second network branch performs first color domain image processing prediction on the first color feature through the first level first color feature prediction module to determine the first initial first color prediction feature; the j-1-th initial first color prediction feature corresponding to the j-1-th level first color feature prediction module is upsampled through the j-th level first color feature prediction module in at least one level first color feature prediction module, the upsampled j-1-th initial first color prediction feature is fused with the K-(j-1)-th initial first color feature, and the fused feature is subjected to first color domain image processing prediction to determine the j-th initial first color prediction feature, until the K-th initial first color prediction feature is determined by the K-th level first color feature prediction module as the first color prediction feature.

[0098] For example, Figure 7BAs shown, the third-level first color feature extraction module 73 is connected to the first-level first color feature prediction module 74, the second-level first color feature prediction module 75 is connected to the first-level first color feature prediction module 74 and the second-level first color feature extraction module 72; the third-level first color feature prediction module 76 is connected to the second-level first color feature prediction module 75 and the first-level first color feature extraction module 71. The Bayer prediction feature is input into the first-level first color feature extraction module 71, and three initial first color features of different scales are obtained through feature extraction of the three-level first color feature extraction modules. The third initial first color feature is predicted and upsampled by the first-level first color feature prediction module 74 to obtain the first initial first color prediction feature. The first initial first color prediction feature is upsampled by the second-level first color feature prediction module 75, the upsampled first initial first color prediction feature is fused with the second initial first color feature, and the fused feature is predicted by the first color domain image processing to obtain the second initial first color prediction feature. The second initial first color prediction feature is upsampled by the third-level first color feature prediction module 76, the upsampled second initial first color prediction feature is fused with the third initial first color feature, and the fused feature is subjected to first color domain image processing prediction to obtain the third initial first color prediction feature as the first color prediction feature.

[0099] It can be understood that the multi-level feature extraction and prediction of the Bayer prediction feature is performed through the multi-level pyramid structure of the second network branch to obtain the first color prediction feature to input into the third network branch. The semantic information contained in the feature has richer dimensions and content, and sending it to the subsequent branches is conducive to improving the final image processing effect. Furthermore, through the dual attention mechanism network in the RSG module, the accuracy of feature extraction and feature prediction can be improved, thereby improving the image processing performance of the second network branch, and then improving the image processing effect of the target image processing network.

[0100] In some embodiments, based on Figure 1 , Figure 3 ,or Figure 6 In any one of the above, in S103, the process of extracting and predicting the features of the second color domain by the third network branch on the first color prediction feature and determining the second color prediction feature can be as follows: Figure 8 As shown, it is implemented through S201-S202 as follows:

[0101] S201. Perform feature extraction of a second color domain on a first color prediction feature through each level of second color feature extraction module in at least one level of second color feature extraction module, determine a second color feature corresponding to each level of second color feature extraction module, and thereby determine at least one second color feature.

[0102] In the embodiment of the present application, the third network branch includes at least one level of second color feature extraction module and at least one level of second color feature prediction module. Among them, each level of second color feature extraction module in the at least one level of second color feature extraction module is connected to the last level of first color feature prediction module in the second network branch, and is used to extract the features of the second color domain of the first color prediction feature output by the second network branch to determine the second color feature.

[0103] In the embodiment of the present application, the scales of feature extraction performed by each level of second color feature extraction module are different, and the resolutions of the corresponding extracted second color features are different. Through each level of second color feature extraction module, the first color prediction feature is subjected to feature extraction of the second color domain to obtain the second color feature corresponding to each level of second color feature extraction module, thereby obtaining a second color feature of at least one resolution through at least one level of second color feature extraction module as at least one second color feature.

[0104] S202: Perform image processing prediction based on at least one second color feature through at least one level of second color feature prediction module to determine a second color prediction feature.

[0105] In the embodiment of the present application, the at least one level second color feature prediction module is used to perform at least one level of image processing prediction in the second color domain on at least one second color feature to determine the second color prediction feature.

[0106] In some embodiments, the level of at least one level of second color feature prediction module is S-1 level, that is, the level of the second color feature prediction module is one level less than the level of the second color feature extraction module. At least one level of second color feature prediction module performs image processing prediction based on at least one second color feature, and the process of determining the second color prediction feature may include: taking the Sth second color feature in at least one second color feature as the first initial second color prediction feature; S is the level of at least one level of second color feature extraction module; upsampling the Lth initial second color prediction feature through the Lth level second color feature prediction module in at least one level of second color feature prediction module, performing feature fusion on the upsampled Lth initial second color feature and the SLth second color feature, performing second color domain image processing prediction on the fused second color feature, and determining the L+1th initial second color prediction feature; until the Sth initial second color prediction feature is determined as the second color prediction feature through the S-1th level second color feature prediction module; L is not less than 1 and less than S.

[0107] For example, the network structure of the third network branch may be as follows: Fig. 9As shown. At least one level of second color feature extraction module includes: a first level second color feature extraction module 90, a second level second color feature extraction module 91, and a third level second color feature extraction module 92. The first level second color feature extraction module 90 performs feature extraction of the second color domain on the first color prediction feature to obtain the first second color feature; the second level second color feature extraction module 91 performs feature extraction of the second color domain on the first color prediction feature to obtain the second second color feature; the third level second color feature extraction module 92 performs feature extraction of the second color domain on the first color prediction feature to obtain the third second color feature. At least one level of second color feature prediction module includes: a first level second color feature prediction module 93, and a second level second color feature prediction module 94. The third second color feature output by the third-level second color feature extraction module 92 is used as the first initial second color prediction feature, the first initial second color prediction feature is upsampled by the first-level second color feature prediction module 93, the upsampled first initial second color prediction feature is fused with the second second color feature output by the second-level second color feature extraction module 91, and the fused feature is subjected to image processing prediction in the second color domain to obtain the second initial second color prediction feature. The second-level second color feature prediction module 94 upsamples the second initial second color prediction feature output by the first-level second color feature prediction module 93, and the upsampled second initial second color prediction feature is fused with the first second color feature output by the first-level second color feature extraction module 90, and the fused feature is subjected to image processing prediction in the second color domain to obtain the third initial second color prediction feature as the second color prediction feature. The upsampling module can be implemented by a convolutional layer, and the downsampling module can be implemented by a deconvolutional layer.

[0108] It can be seen that both the second color feature extraction module and the second color feature prediction module include an RSG module, wherein the RSG module includes a region growing network based on a dual attention mechanism of space and channel. Figure 5 As shown, no further description is given here.

[0109] It can be understood that the multi-level pyramid structure of the third network branch performs multi-level feature extraction and prediction on the first color prediction feature to obtain the second color prediction feature for image reconstruction. The semantic information contained in the feature has richer dimensions and content, which is conducive to improving the final image processing effect. Furthermore, through the dual attention mechanism network in the RSG module, the accuracy of feature extraction and feature prediction can be improved, thereby improving the image processing performance of the third network branch, and then improving the image processing effect of the target image processing network.

[0110] It can be seen that compared with the current method of replacing some functional modules in traditional ISP with deep learning network modules, this technical solution will not be coupled with traditional ISP, and can exclusively use hardware resources such as accelerated processing unit (APU) or neural network processor (NPU), and will not have problems such as resource grabbing and low operating efficiency. In addition, the embodiment of the present application realizes end-to-end image processing through a multi-segment network structure in the target image processing network, and does not require too much manual participation in application. It only needs to input the Bayer domain image (bayer raw) corresponding to the original image to obtain a reconstructed image in the second color domain with better effect, thereby greatly improving the efficiency of image processing.

[0111] The image processing method provided in the embodiment of the present application also includes a training process for the target image processing network. In some embodiments, such as Fig.10 As shown, S10-S15 may also be performed as follows:

[0112] S10. Determine an initial image processing network and at least one set of training data.

[0113] In the embodiment of the present application, the initial image processing network includes an initial first network branch, an initial second network branch, and an initial third network branch. The electronic device obtains at least one set of training data to train the initial first network branch, the initial second network branch, and the initial third network branch to obtain a target image processing network including the initial first network branch, the initial second network branch, and the initial third network branch.

[0114] In an embodiment of the present application, each set of training data in at least one set of training data includes: a training image and at least one sample image corresponding to the training image. Among them, at least one sample image is obtained by performing at least one image signal processing process on the training image through a preset image signal processing algorithm. Exemplarily, at least one sample image may include a Bayer domain sample image, an RGB domain sample image, and a YUV domain sample image obtained by performing image signal processing in the Bayer domain, RGB domain, and YUV domain on the training image through a software ISP algorithm.

[0115] In some embodiments, before determining the initial image processing network and at least one set of training data, the process of determining at least one set of training data can be as follows: Fig.11 The S01-S04 shown in the figure are implemented as follows:

[0116] S01. Using an image sensor, collect at least one original training image in a static state.

[0117] In the embodiment of the present application, at least one raw image is captured by an image sensor in a static state as at least one original training image. Here, the static state refers to a relative static state between the image sensor and the photographed scene, that is, the picture content of at least one original training image is the same.

[0118] S02: Determine a training image from at least one original training image, and perform superposition and denoising on the at least one original training image to determine an initial sample image.

[0119] In an embodiment of the present application, an image is determined from at least one original training image with the same picture content as a training image, and at least one original training image is superimposed and denoised to obtain an initial sample image. In some embodiments, the training image can be randomly selected from at least one original training image.

[0120] S03. Perform Bayer domain image processing on the initial sample image through a preset image processing algorithm module to determine a Bayer domain sample image; and / or, perform first color domain image processing on the Bayer domain sample image through a preset image processing algorithm module to determine a first color domain sample image; and / or, perform second color domain image processing on the first color domain sample image through a preset image processing algorithm module to determine at least one second color domain sample image.

[0121] In the embodiment of the present application, the preset image processing algorithm module is an ISP software algorithm tool. Since the requirements for the quality of training data are much higher than the requirements for the cost of generating data in the training data generation stage, in some embodiments, the preset image processing algorithm module can be implemented by combining a good hardware ISP simulation tool with a complex software ISP algorithm to improve the quality of at least one training data, thereby improving the image processing effect of the target image processing network trained based on at least one training data.

[0122] In the embodiment of the present application, the algorithm processing flow of the preset image processing algorithm module includes Bayer domain image processing, first color domain image processing and second color domain image processing in the image signal processing flow. The Bayer domain sample image can be determined by performing Bayer domain image processing on the initial sample image by the preset image processing algorithm module. The first color domain sample image can be determined by continuing to perform first color domain image processing on the Bayer domain sample image by the preset image processing algorithm module; at least one second color domain sample image can be determined by continuing to perform second color domain image processing on the first color domain sample image by the preset image processing algorithm module. In this way, at least one of the at least one second color domain sample image, the first color domain sample image and the Bayer domain sample image can be determined as at least one sample image.

[0123] It is understandable that since at least one sample image is obtained through a preset image processing algorithm module and has a good image processing effect, it can be used as sample data (ground truth) for supervised training to train and improve the image processing effect of the initial image processing network.

[0124] In some embodiments, since the first color domain image processing, such as RGB domain image processing, involves color preference processing, in order to enable the neural network to predict the effect of higher color preference, when the first color domain image processing of the preset image processing algorithm module is completed, the image determined by performing the first color domain image processing on the Bayer domain sample image through the preset image processing algorithm module can be used as the initial first color domain sample image; and the initial first color domain sample image is then subjected to color preference processing to determine the first color domain sample image.

[0125] Exemplarily, the image output after RGB domain image processing by the preset image processing algorithm module, that is, three-dimensional look up table (3DLUT) color mapping processing, or the color of the image is tuned by image processing software, can be used as an RGB domain sample image.

[0126] S04. Combine at least one of the second color domain sample image, the Bayer domain sample image, and the first color domain sample image with the training image to determine a set of training data, and further determine at least one set of training data.

[0127] In an embodiment of the present application, a training image is compared with at least one of a second color domain sample image, a Bayer domain sample image, and a first color domain sample image to determine a set of training data, thereby determining at least one set of training data. That is, at least one set of training data includes a training image for image processing by a neural network, and at least one sample image for comparing with an image output by the neural network in different image signal processing flows to update network parameters. Exemplarily, after Bayer domain image processing (bayer processing), RGB domain image processing (RGB processing) and YUV domain image processing (YUVprocessing) are performed on the training image input_raw through a preset image processing algorithm module, a Bayer sample image target_raw, an RGB sample image target_rgb, and a first-scale YUV sample image are obtained. Referring again to Fig. 9The pyramid structure of the third network branch in the first scale YUV sample image is downsampled, such as performing two downsampling operations to obtain the second scale YUV sample image target_yuv_ds4 and the third scale YUV sample image target_yuv_ds16. Then a training image input_raw and five sample images target_raw, target_rgb, target_yuv, target_yuv_ds4, and target_yuv_ds16 are used as a set of training data.

[0128] S11. Determine at least one training image prediction feature by extracting and predicting features based on the training images in each set of training data through at least one of the initial first network branch, the initial second network branch, and the initial third network branch.

[0129] In the embodiment of the present application, the network parameters in the initial first network branch, the initial second network branch, and the initial third network branch are initial values, and network training needs to be performed based on training data, and the network parameters are updated according to the results of each network training. Here, at least one of the initial first network branch, the initial second network branch, and the initial third network branch can be used to extract and predict features based on the training images in each set of training data, and determine the Bayer training features corresponding to the initial first network branch, and / or the first color training features corresponding to the initial second network branch, and / or the second color training features corresponding to the initial third network branch as at least one training image prediction feature.

[0130] S12: Perform image reconstruction based on at least one training image prediction feature to determine at least one training reconstructed image.

[0131] In the embodiment of the present application, image reconstruction is performed based on at least one of the Bayer training feature, the first color training feature, and the second color training feature to determine at least one training reconstructed image.

[0132] In some embodiments, the structure of the initial first network branch is Figure 4A or Figure 4B The structure of the first network branch shown is the same. The initial first network branch performs feature extraction and prediction based on the training image, and outputs the Bayer training features through the third-level Bayer feature prediction module 110 in the initial first network branch. Fig. 12A or Fig. 12B As shown, the Bayer training features output by the third-level Bayer feature prediction module 110 can be reconstructed by an image reconstruction module 111 composed of two convolutional layers to obtain a Bayer domain reconstructed image.

[0133] In some embodiments, the structure of the initial second network branch is Fig. 7A or Figure 7B The structure of the second network branch shown is the same. The initial second network branch performs feature extraction and prediction based on the Bayer training features output by the initial first network branch, and outputs the first color training features through the third-level first color feature prediction module in the initial second network branch. Similarly, the first color training features output by the third-level first color feature prediction module can be reconstructed through an image reconstruction module composed of two convolutional layers to obtain a first color domain reconstructed image.

[0134] It should be noted that when the first color domain is the RGB color domain, the image resolution of the output RGB color domain reconstructed image should be twice that of the Bayer domain reconstructed image.

[0135] In some embodiments, the structure of the initial third network branch is Fig. 9 The structure of the third network branch shown is the same. The initial third network branch performs at least one level of downsampling, feature extraction and prediction based on the first color feature output by the initial second network branch, and can obtain at least one second color training feature corresponding to at least one resolution. Fig. 12C As shown, the image reconstruction module 113 composed of three convolutional layers can be used to reconstruct the third second color feature output by the third-level second color feature extraction module, that is, the first second color training feature, to obtain a second color domain reconstructed image of the first resolution. Here, the first second color training feature is the feature output by the third-level second color feature extraction module in the initial third network branch, corresponding to Fig. 9 The first initial second color prediction feature in the image reconstruction module 115 is used to reconstruct the second initial second color prediction feature output by the first-level second color feature prediction module 114 to obtain a second color domain reconstructed image with a second resolution; the third color prediction feature output by the second-level second color feature prediction module 116 is used to reconstruct the second color domain reconstructed image with a third resolution to obtain a second color domain reconstructed image with a third resolution. Thus, at least one second color domain reconstructed image corresponding to at least one resolution is obtained.

[0136] In some embodiments, at least one of the at least one second color domain reconstructed image, the Bayer domain reconstructed image, and the first color domain reconstructed image may be used as at least one training reconstructed image.

[0137] For example, the training and application process of the image processing network of the embodiment of the present application can be as follows: Fig.13As shown, the original image is input into the first network branch, and the first network branch resamples the original image to obtain a Bayer domain image. The feature encoder 1 is used to extract the Bayer domain image and determine the Bayer feature; the feature decoder 1 is used to predict the Bayer feature and determine the Bayer prediction feature. If it is the network training stage, a reconstruction module 81 can be set at the output of the first network branch to reconstruct the image according to the output Bayer prediction feature and train the first network branch. The Bayer prediction feature is input into the second network branch, and the feature encoder 2 of the second network branch is used to extract the Bayer prediction feature and determine the first color feature; the feature decoder 2 is used to predict the first color feature and determine the first color prediction feature. If it is the network training stage, a reconstruction module 82 can be set at the output of the second network branch to reconstruct the image according to the output first color prediction feature and train the second network branch. The first color prediction feature is input into the third network branch, and the feature encoder 3 of the third network branch is used to extract the first color prediction feature and determine the second color feature; the feature decoder 3 is used to predict the second color feature and determine the second color prediction feature. If it is the network training stage, a reconstruction module 3 can be set at the output of the third network branch to reconstruct the image according to the output second color prediction feature and train the third network branch. Exemplarily, the first network branch is used to complete the Bayer domain processing and reconstruction of the image; the second network branch completes the RGB domain processing and reconstruction; the third network branch is used to perform YUV domain processing. That is, the corresponding image signal processing process is implemented through the segmented network structure.

[0138] S13. According to at least one difference between at least one training reconstructed image and at least one sample image, iteratively update the network parameters of at least one of the initial first network branch, the initial second network branch, and the initial third network branch until a preset training target is reached, and determine at least one of the first network branch, the second network branch, and the third network branch.

[0139] In an embodiment of the present application, for the training process of the initial first network branch, the network parameters of the initial first network branch can be iteratively updated according to the difference between the Bayer domain reconstructed image and the Bayer domain sample image in at least one training reconstructed image, until the preset training target corresponding to the initial first network branch is reached, and the first network branch is determined. And / or, for the training process of the initial second network branch, the network parameters of the initial second network branch can be iteratively updated according to the difference between the first color domain reconstructed image and the first color domain sample image in at least one training reconstructed image, until the preset training target corresponding to the initial second network branch is reached, and the second network branch is determined. And / or, for the training process of the initial third network branch, the network parameters of the initial third network branch can be iteratively updated according to the difference between the second color domain reconstructed image and the second color domain sample image in at least one training reconstructed image, until the preset training target corresponding to the initial third network branch is reached, and the third network branch is determined.

[0140] In the embodiment of the present application, since different network branches correspond to different image signal processing processes, it is necessary to use different loss functions specifically for different network branches, so as to determine the loss of each round of training according to the loss function and update the network parameters according to the loss.

[0141] In some embodiments, the first network branch is mainly used for Bayer domain image processing in the ISP processing flow, and its main function is to correct the input image, for example, such as Bad Pixel Correction (BPC), Lens Shading Correction (LSC), White Balance (WB), and some Bayer noise processing. Therefore, the training loss of the initial first network branch can be determined by combining the norm loss and the LSC loss.

[0142] In some embodiments, at least one training reconstructed image includes: a Bayer domain reconstructed image; the Bayer domain reconstructed image is determined by performing image reconstruction on the Bayer training features; the Bayer training features are determined by extracting and predicting features based on the training image by the initial first network branch. Determining the first training loss corresponding to the initial first network branch may include: determining the Bayer domain norm loss corresponding to each pixel position according to the Bayer domain reconstructed pixel corresponding to each pixel position in the Bayer domain reconstructed image and the Bayer domain sample pixel corresponding to each pixel position in the Bayer domain sample image. Determine at least one correction gain corresponding to each pixel position according to a preset correction coefficient corresponding to the image sensor; at least one correction gain represents a gain value corresponding to the correction processing of at least one Bayer domain image. Determine the Bayer domain training loss corresponding to each pixel position according to the product of at least one correction gain value and the Bayer domain norm loss, and determine the first training loss corresponding to the initial first network branch according to the Bayer domain training loss corresponding to each pixel position.

[0143] Exemplarily, the preset correction coefficients may include: an LSC table and a white balance correction gain. The white balance gain value may be determined by a preset configuration parameter in the image sensor. At least one correction gain may include: a lens occlusion correction gain and a white balance correction gain. The first training loss may be determined by formula (1), as follows:

[0144]

[0145] Among them, y p represents the pixel value of the Bayer domain reconstructed pixel at pixel position p in the Bayer domain reconstructed image, represents the pixel value of the Bayer domain sample pixel at pixel position p in the Bayer domain sample image, represents the Bayer domain norm loss at pixel position p, N 1 is the number of pixels in the Bayer domain image. LSCG represents the lens occlusion correction gain (LSC Gain value) at pixel position p calculated according to the LSC table of the image sensor, and WBG represents the white balance correction gain (WB Gain value) at pixel position p. loss_raw represents the first training loss.

[0146] In this way, based on the first training loss, the network parameters of the initial first network branch are updated, the next round of training is performed based on the updated network parameters, and then the network parameters are updated according to the first training loss of the next round of training. The update is iterated until the preset training target corresponding to the initial first network branch is reached, and the first network branch is determined.

[0147] It can be understood that by combining the norm loss and the correction gain to determine the training loss of the initial first network branch, the supervision constraint on the area in the image can be amplified, thereby better guiding the learning prediction direction of the network, improving the training effect, and further improving the effect of image processing by the first network branch obtained through training.

[0148] In some embodiments, the second network branch is mainly used to perform image processing in the first color domain, such as the RGB domain, in the ISP processing flow. For example, the color and hue of the image are processed, such as color correction matrix (CCM) processing, gamma correction processing, GTM, and LTM. In addition, the second network branch is also used to convert the image color domain, such as converting the color domain of the Bayer domain image from linear to standard RGB (sRGB) domain; it also involves the processing of factors such as preferred colors. Therefore, the training loss of the initial second network branch can be determined by combining the norm loss and the perceptual loss.

[0149] In some embodiments, at least one training reconstructed image includes: a first color domain reconstructed image; the first color domain reconstructed image is determined by performing image reconstruction on a first color training feature; the first color training feature is determined by extracting and predicting features based on the Bayer training feature by the initial second network branch. The process of determining the second training loss of the initial second network branch may include: determining the first color domain norm loss between the first color domain reconstructed image and the first color domain sample image; performing perceptual prediction on the first color domain reconstructed image and the first color domain sample image according to a preset perceptual network, respectively, to determine the perceptual loss between the first color domain reconstructed image and the first color domain sample image; weighted merging the first color domain norm loss and the perceptual loss to determine the second training loss corresponding to the initial second network branch.

[0150] Exemplarily, the second training loss can be determined by formula (2), as follows:

[0151]

[0152] Where y' represents the first color domain reconstructed image, Represents the sample image in the first color domain. G indicates that the prediction is performed through the VGG network and the perceptual loss is calculated in the feature domain. is the perceptual loss. p represents the pixel value of the first color domain reconstructed pixel at the pixel position p in the first color domain reconstructed image, represents the pixel value of the first color domain sample pixel at the pixel position p in the first color domain sample image; N2 is the number of pixels of the first color domain image. is the first color domain norm loss. 1 , 2 Represent the weights of perceptual loss and first color domain norm loss respectively. loss rgb represents the second training loss.

[0153] In an embodiment of the present application, based on the second training loss, the network parameters of the initial second network branch are updated, the next round of training is performed based on the updated network parameters, and then the network parameters are updated according to the second training loss of the next round of training, and the update is iterated until the preset training target corresponding to the initial second network branch is reached, and the second network branch is determined.

[0154] In some embodiments, the third network branch is mainly used to process the noise and clarity of the image, and combined with the network structure of the third network branch in the embodiment of the present application, it includes a process of obtaining features of different resolutions by downsampling, so the losses corresponding to different resolutions can also be weighted. Therefore, the training loss of the initial third network branch can be determined by combining the norm loss with the loss related to the clarity evaluation, such as the structural similarity loss (SSIM Loss) and / or the edge loss.

[0155] In some embodiments, at least one training reconstructed image includes: at least one second color domain reconstructed image; at least one second color domain reconstructed image is determined by image reconstruction of at least one second color training feature corresponding to at least one resolution; at least one second color training feature is determined by performing at least one level of downsampling, feature extraction and prediction on the first color training feature by the initial third network branch. At least one sample image includes: at least one second color domain sample image corresponding to at least one resolution. The process of determining the third training loss corresponding to the initial third network branch may include: for each resolution of at least one resolution, determining the second color domain norm loss between the second color domain reconstructed image corresponding to each resolution and the second color domain sample image; determining at least one of the structural similarity loss and the edge loss between the second color domain reconstructed image corresponding to each resolution and the second color domain sample image; based on at least one of the structural similarity loss and the edge loss, weighted merging with the second color domain norm loss to determine the second color domain training loss corresponding to each resolution, and then determining at least one second color domain training loss corresponding to at least one resolution; weighted merging of at least one second color domain training loss to determine the third training loss corresponding to the initial third network branch.

[0156] Exemplarily, the second color domain training loss corresponding to each resolution can be determined by formula (3), as follows:

[0157]

[0158] Wherein, y" represents a second color domain reconstructed image corresponding to a resolution, A sample image of the second color domain representing the resolution; Represents the structural similarity loss corresponding to this resolution; Represents the edge loss corresponding to the resolution; Represents the second color domain norm loss corresponding to the resolution. 3 , 4 and λ 5 Represents the weights of structural similarity loss, edge loss and second color domain norm loss respectively. loss' yuv Represents the second color domain training loss corresponding to this resolution.

[0159] According to formula (3), at least one second color domain training loss corresponding to at least one resolution is determined. Exemplarily, at least one second color domain training loss may include loss' yuv 、loss' yuv4 and loss' yuv16 The third training loss corresponding to the initial third network branch can be calculated according to formula (4).

[0160] loss yuv =α*loss' yuv +β*loss' yuv4 +γ*loss' yuv16 (4)

[0161] Among them, α, β and γ are the weights of the second color domain training loss corresponding to each resolution, loss yuv is the third training loss.

[0162] Based on the third training loss, the network parameters of the initial third network branch are updated, the next round of training is performed based on the updated network parameters, and then the network parameters are updated according to the second training loss of the next round of training. The update is repeated until the preset training target corresponding to the initial third network branch is reached, and the third network branch is determined.

[0163] S14. Determine a target image processing network based on at least one of the first network branch, the second network branch, and the third network branch.

[0164] In an embodiment of the present application, the initial first network branch, the initial second network branch, and the initial third network branch in the initial image processing network can be trained by the above-mentioned method for generating at least one set of training data, combined with the method for determining the first training loss, the second training loss, and the third training loss, that is, three-stage supervised training of the initial image processing network is implemented to obtain a target image processing network including the first network branch, the second network branch, and the third network branch, so that the Bayer domain image is processed by the target image processing network obtained by the three-stage supervised training to determine the reconstructed image.

[0165] In the embodiment of the present application, the above-mentioned method for generating at least one set of training data can also be combined with one or two of the methods for determining the first training loss, the second training loss, and the third training loss; supervised training is performed on one or two of the first network branch, the second network branch, and the third network branch to obtain one or two of the first network branch, the second network branch, and the third network branch, and the other network branches are trained by other network training methods to obtain the trained first network branch, the second network branch, and the third network branch as the target image processing network. The specific selection is made according to the actual situation, and the embodiment of the present application is not limited.

[0166] In some embodiments, the initial first network branch, the initial second network branch, and the initial third network branch can be subjected to segmented supervised training. Exemplarily, the initial first network branch, the initial second network branch, and the initial third network branch are trained in sequence, and the currently trained network branch reaches the training target. If the training converges, the network parameters of the network branch are fixed, and the gradient update is no longer performed, and the training of the next network branch is continued until the initial first network branch, the initial second network branch, and the initial third network branch are all trained to obtain the target image processing network.

[0167] In some embodiments, the initial first network branch, the initial second network branch, and the initial third network branch may also be supervised and trained simultaneously. Exemplarily, in a reverse iteration of network training, the weights of all branches are updated simultaneously. At this time, the loss function may be set as the training loss of the three network branches, i.e., the weighted sum of the first training loss, the second training loss, and the third training loss, as shown in formula (5):

[0168] loss isp =α 1 *loss raw +α 2 *loss rgb +α 3 *loss yuv (5)

[0169] In formula (5), α1 , α 2 With α 3 They are the first training loss loss raw , second training loss loss rgb And the third training loss loss yuv The corresponding weighting coefficient can be set according to the characteristics of the image sensor in practical applications, and is not limited in the present embodiment. isp is the total training loss for each training when the initial first network branch, the initial second network branch, and the initial third network branch are simultaneously supervised. isp The network parameters of the initial first network branch, the initial second network branch and the initial third network branch are updated simultaneously until the loss isp When the training goal is achieved, such as when the training converges, the target image processing network is obtained.

[0170] It can be understood that compared with the current direct end-to-end convolutional neural network learning, the embodiment of the present application designs a corresponding training loss determination method according to the characteristics of image processing tasks of different network branches, and fits the processing of different domain processes in ISP processing, which can better supervise and guide the learning direction of the network and improve the image processing effect based on the trained target image processing network.

[0171] The present application embodiment provides an image processing device 1, which is applied to a target image processing network, such as Fig.14 As shown, it includes: a network prediction module 11 and a reconstruction module 12. Among them:

[0172] The network prediction module 11 is used to resample the original image collected by the image sensor into a target Bayer format through a first network branch to obtain a Bayer domain image, perform feature extraction and prediction based on the Bayer domain image, and determine the Bayer prediction feature; perform feature extraction and prediction of a first color domain on the Bayer prediction feature through a second network branch to determine the first color prediction feature; perform feature extraction and prediction of a second color domain on the first color prediction feature through a third network branch to determine the second color prediction feature;

[0173] The target image processing network is obtained by performing supervised training on at least one of the initial first network branch, the initial second network branch and the initial third network branch;

[0174] The reconstruction module 12 is used to perform image reconstruction based on the second color prediction feature and determine a reconstructed image corresponding to the original image.

[0175] In some embodiments, the first network branch includes: at least one level of Bayer feature extraction module and at least one level of Bayer feature prediction module; the network prediction module 11 is also used to perform at least one level of feature extraction based on the Bayer domain image through the at least one level of Bayer feature extraction module to determine the Bayer feature; perform at least one level of Bayer domain image processing prediction based on the Bayer feature through the at least one level of Bayer feature prediction module to determine the Bayer prediction feature.

[0176] In some embodiments, the network prediction module 11 is further used to perform feature extraction on the Bayer domain image through a first-level Bayer feature extraction module in the at least one-level Bayer feature extraction module to determine a first initial Bayer feature; downsample and extract features on the i-1th initial Bayer feature corresponding to the i-1th level Bayer feature extraction module through an i-th level Bayer feature extraction module in the at least one-level Bayer feature extraction module to determine the i-th initial Bayer feature, until the Mth initial Bayer feature is determined through an M-th level Bayer feature extraction module as the Bayer feature; M is the number of levels of the at least one level Bayer feature extraction module; i is not less than 2 and not greater than M.

[0177] In some embodiments, the Mth level Bayer feature extraction module is connected to the first level Bayer feature prediction module in the at least one level Bayer feature prediction module; the network prediction module 11 is also used to perform Bayer domain image processing prediction on the Bayer feature through the first level Bayer feature prediction module to determine a first initial Bayer prediction feature; upsample and perform Bayer domain image processing prediction on the i-1th initial Bayer prediction feature corresponding to the i-1th level Bayer feature prediction module through the i-th level Bayer feature prediction module in the at least one level Bayer feature prediction module to determine the i-th initial Bayer prediction feature, until the Mth initial Bayer prediction feature is determined through the Mth level Bayer feature prediction module as the Bayer prediction feature; the number of levels of the at least one level Bayer feature prediction module is the same as the number of levels of the at least one level Bayer feature extraction module.

[0178] In some embodiments, the Mth-level Bayer feature extraction module is connected to the first-level Bayer feature prediction module in the at least one-level Bayer feature prediction module; the M-(i-1)th-level Bayer feature extraction module is connected to the i-th-level Bayer feature prediction module; the network prediction module 11 is also used to perform Bayer domain image processing prediction on the Bayer feature through the first-level Bayer feature prediction module to determine a first initial Bayer prediction feature; upsample the i-1th initial Bayer prediction feature corresponding to the i-1th-level Bayer feature prediction module through the i-th-level prediction module in the at least one-level Bayer feature prediction module, perform feature fusion on the upsampled i-1th initial Bayer prediction feature and the M-(i-1)th initial Bayer prediction feature, and perform Bayer domain image processing prediction on the fused feature to determine the i-th initial Bayer prediction feature, until the Mth initial Bayer prediction feature is determined by the M-level Bayer feature prediction module as the Bayer prediction feature.

[0179] In some embodiments, the second network branch includes: at least one first color feature extraction module and at least one first color feature prediction module; the network prediction module 11 is also used to perform at least one first color feature extraction of the first color domain based on the Bayer prediction feature through the at least one first color feature extraction module to determine the first color feature; perform at least one first color domain image processing prediction based on the first color feature through the at least one first color feature prediction module to determine the first color prediction feature.

[0180] In some embodiments, the network prediction module 11 is also used to perform first color feature extraction on the Bayer prediction feature through the first-level first color feature extraction module in the at least one-level first color feature extraction module to determine the first initial first color feature; downsample and extract features of the j-1th initial first color feature corresponding to the j-1th level first color feature extraction module through the j-th level first color feature extraction module in the at least one-level first color feature extraction module to determine the j-th initial first color feature, until the K-th initial first color feature is determined through the K-th level first color feature extraction module as the first color feature; K is the number of the at least one-level first color feature extraction module; j is not less than 2 and not greater than K.

[0181] In some embodiments, the K-th level first color feature extraction module is connected to the first-level first color feature prediction module in the at least one level first color feature prediction module; the network prediction module 11 is also used to perform first color domain image processing prediction on the first color feature through the first-level first color feature prediction module to determine the first initial first color prediction feature; through the j-th level first color feature prediction module in the at least one level first color feature prediction module, upsample and perform first color domain image processing prediction on the j-1th initial first color prediction feature corresponding to the j-1th level first color feature prediction module to determine the jth initial first color prediction feature, until the K-th first color prediction feature is determined through the K-th level first color feature prediction module as the first color prediction feature; the number of levels of the at least one level first color feature extraction module is the same as the number of levels of the at least one level first color feature prediction module.

[0182] In some embodiments, the K-th level first color feature extraction module is connected to the first-level first color feature prediction module in the at least one level first color feature prediction module; the K-(j-1)-th level first color feature extraction module is connected to the j-th level first color feature prediction module; the network prediction module 11 is also used to perform first color domain image processing prediction on the first color feature through the first-level first color feature prediction module to determine a first initial first color prediction feature; upsample the j-1th initial first color prediction feature corresponding to the j-1-th level first color feature prediction module through the j-th level first color feature prediction module in the at least one level first color feature prediction module, perform feature fusion on the upsampled j-1th initial first color prediction feature and perform first color domain image processing prediction on the fused feature to determine the j-th initial first color prediction feature, until the K-th initial first color prediction feature is determined by the K-th level first color feature prediction module as the first color prediction feature.

[0183] In some embodiments, the third network branch includes: at least one level of second color feature extraction module and at least one level of second color feature prediction module; the network prediction module 11 is also used to perform feature extraction of the second color domain on the first color prediction feature through each level of second color feature extraction module in the at least one level of second color feature extraction module, determine the second color feature corresponding to each level of second color feature extraction module, and thus determine at least one second color feature; through the at least one level of second color feature prediction module, perform image processing prediction based on the at least one second color feature to determine the second color prediction feature.

[0184] In some embodiments, the level of the at least one level second color feature prediction module is S-1 level; the network prediction module 11 is also used to use the Sth second color feature among the at least one second color feature as the first initial second color prediction feature; S is the level of the at least one level second color feature extraction module; through the Lth level second color feature prediction module among the at least one level second color feature prediction module, the Lth initial second color prediction feature is upsampled, the upsampled Lth initial second color feature is fused with the SLth second color feature, and the fused second color feature is predicted by the second color domain image processing to determine the L+1th initial second color prediction feature; until the Sth initial second color prediction feature is determined as the second color prediction feature through the S-1th level second color feature prediction module; L is not less than 1 and less than S.

[0185] In some embodiments, the Bayer feature extraction module, the Bayer feature prediction module, the first color feature extraction module, the first color feature prediction module, the second color feature extraction module and the second color feature prediction module include: a region growing network implementation based on a dual attention mechanism of space and channel.

[0186] In some embodiments, the image processing device also includes a network training module; the network training module is used to determine an initial image processing network and at least one set of training data; the initial image processing network includes the initial first network branch, the initial second network branch and the initial third network branch; each set of training data in the at least one set of training data includes: a training image and at least one sample image corresponding to the training image; through at least one branch of the initial first network branch, the initial second network branch and the initial third network branch, feature extraction and prediction are performed based on the training image in each set of training data to determine at least one training image prediction feature; image reconstruction is performed based on the at least one training image prediction feature to determine at least one training reconstructed image; according to at least one difference between the at least one training reconstructed image and the at least one sample image, the network parameters of at least one branch of the initial first network branch, the initial second network branch and the initial third network branch are iteratively updated until the preset training target is reached, and at least one branch of the first network branch, the second network branch and the third network branch is determined; based on at least one branch of the first network branch, the second network branch and the third network branch, the target image processing network is determined.

[0187] In some embodiments, the at least one training reconstructed image includes: a Bayer domain reconstructed image; the Bayer domain reconstructed image is determined by performing image reconstruction on the Bayer training features; the Bayer training features are determined by extracting and predicting features based on the training image by the initial first network branch; the at least one sample image includes: a Bayer domain sample image; the network training module is further used to determine the Bayer domain norm loss corresponding to each pixel position according to the Bayer domain reconstructed pixel corresponding to each pixel position in the Bayer domain reconstructed image and the Bayer domain sample pixel corresponding to each pixel position in the Bayer domain sample image; determine at least one correction gain corresponding to each pixel position according to a preset correction coefficient corresponding to the image sensor; the at least one correction gain represents a gain value corresponding to the correction processing of at least one Bayer domain image; determine the Bayer domain training loss corresponding to each pixel position according to the product of the at least one correction gain value and the Bayer domain norm loss, and determine the first training loss corresponding to the initial first network branch according to the Bayer domain training loss corresponding to each pixel position; based on the first training loss, iteratively update the network parameters of the initial first network branch until the preset training target corresponding to the initial first network branch is reached, and the first network branch is determined.

[0188] In some embodiments, the at least one training reconstructed image includes: a first color domain reconstructed image; the first color domain reconstructed image is determined by performing image reconstruction on a first color training feature; the first color training feature is determined by extracting and predicting features based on the Bayer training feature by the initial second network branch; the at least one sample image includes: a first color domain sample image; the network training module is also used to determine a first color domain norm loss between the first color domain reconstructed image and the first color domain sample image; according to a preset perception network, perceptual prediction is performed on the first color domain reconstructed image and the first color domain sample image respectively to determine the perceptual loss between the first color domain reconstructed image and the first color domain sample image; the first color domain norm loss and the perceptual loss are weightedly merged to determine a second training loss corresponding to the initial second network branch; based on the second training loss, the network parameters of the initial second network branch are iteratively updated until the preset training target corresponding to the initial second network branch is reached, and the second network branch is determined.

[0189] In some embodiments, the at least one training reconstructed image includes: at least one second color domain reconstructed image; the at least one second color domain reconstructed image is determined by image reconstruction of at least one second color training feature corresponding to at least one resolution; the at least one second color training feature is determined by performing at least one level of downsampling, feature extraction and prediction based on the first color training feature by the initial third network branch; the at least one sample image includes: at least one second color domain sample image corresponding to the at least one resolution; the network training module is further used to determine, for each resolution of the at least one resolution, the second color domain norm loss between the second color domain reconstructed image corresponding to each resolution and the second color domain sample image; determine At least one of the structural similarity loss and the edge loss between the second color domain reconstructed image corresponding to each resolution and the second color domain sample image; based on at least one of the structural similarity loss and the edge loss, weighted merging with the second color domain norm loss to determine the second color domain training loss corresponding to each resolution, and then determining at least one second color domain training loss corresponding to the at least one resolution; weighted merging of the at least one second color domain training loss to determine the third training loss corresponding to the initial third network branch; based on the third training loss, iteratively updating the network parameters of the initial third network branch until the preset training target corresponding to the initial third network branch is reached, and determining the third network branch.

[0190] In some embodiments, the network training module is also used to, before determining the initial image processing network and at least one set of training data, collect at least one original training image in a stationary state through the image sensor; determine the training image from the at least one original training image, and perform superposition and denoising on the at least one original training image to determine the initial sample image; perform Bayer domain image processing on the initial sample image through a preset image processing algorithm module to determine the Bayer domain sample image; and / or, perform first color domain image processing on the Bayer domain sample image through the preset image processing algorithm module to determine the first color domain sample image; and / or, perform second color domain image processing on the first color domain sample image through the preset image processing algorithm module to determine the at least one second color domain sample image; combine at least one of the at least one second color domain sample image, the Bayer domain sample image and the first color domain sample image with the training image to determine a set of training data, and then determine the at least one set of training data.

[0191] In some embodiments, the network training module is further used to perform first color domain image processing on the Bayer domain sample image through a preset image processing algorithm module to determine an initial first color domain sample image; and perform color preference processing on the initial first color domain sample image to determine the first color domain sample image.

[0192] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0193] In some embodiments, the present application also provides an electronic device, Fig.15 This is an optional structural diagram of an electronic device provided in an embodiment of the present application. Fig.15 As shown, the electronic device 2 includes: a memory 22 and a processor 23. The memory 22 and the processor 23 are connected via a communication bus 24; the memory 22 is used to store executable instructions; the processor 23 is used to execute the executable instructions stored in the memory 22 to implement the image processing method provided in the embodiment of the present application.

[0194] The embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will be caused to execute the image processing method provided by the embodiment of the present application. In some embodiments, the computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it can be various devices including one or any combination of the above memories.

[0195] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0196] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0197] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0198] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0199] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0200] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0201] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0202] The above is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An image processing method, applied to a target image processing network, characterized in that: include: Resampling the original image acquired by the image sensor into a target Bayer format to obtain a Bayer domain image through the first network branch, extracting and predicting features based on the Bayer domain image, and determining Bayer prediction features; Extracting and predicting features in a first color domain on the Bayer prediction features through a second network branch to determine a first color prediction feature; By using a third network branch, extracting and predicting features of a second color domain on the first color prediction feature, to determine a second color prediction feature; Performing image reconstruction based on the second color prediction feature to determine a reconstructed image corresponding to the original image; The target image processing network is obtained by performing supervised training on at least one of the initial first network branch, the initial second network branch and the initial third network branch.

2. The method according to claim 1, characterized in that The first network branch includes: at least one level of Bayer feature extraction module and at least one level of Bayer feature prediction module; the feature extraction and prediction based on the Bayer domain image to determine the Bayer prediction feature includes: Performing at least one level of feature extraction based on the Bayer domain image by the at least one level of Bayer feature extraction module to determine the Bayer feature; By means of the at least one level Bayer feature prediction module, at least one level Bayer domain image processing prediction is performed based on the Bayer feature to determine the Bayer prediction feature.

3. The method according to claim 2, characterized in that The step of performing at least one level of feature extraction based on the Bayer domain image by the at least one level of Bayer feature extraction module to determine the Bayer feature includes: Performing feature extraction on the Bayer domain image by a first-level Bayer feature extraction module in the at least one-level Bayer feature extraction module to determine a first initial Bayer feature; The i-th initial Bayer feature corresponding to the i-th level Bayer feature extraction module is downsampled and feature extracted by the i-th level Bayer feature extraction module in the at least one level Bayer feature extraction module to determine the i-th initial Bayer feature, until the M-th initial Bayer feature is determined by the M-th level Bayer feature extraction module as the Bayer feature; M is the number of the at least one level Bayer feature extraction module; i is not less than 2 and not greater than M.

4. The method according to claim 2 or 3, characterized in that: The M-th level Bayer feature extraction module is connected to the first level Bayer feature prediction module in the at least one level Bayer feature prediction module; performing at least one level Bayer domain image processing prediction based on the Bayer feature through the at least one level Bayer feature prediction module to determine the Bayer prediction feature, including: Performing Bayer domain image processing prediction on the Bayer feature through the first-level Bayer feature prediction module to determine a first initial Bayer prediction feature; The i-th level Bayer feature prediction module in the at least one level Bayer feature prediction module performs upsampling and Bayer domain image processing prediction on the i-1th initial Bayer prediction feature corresponding to the i-1th level Bayer feature prediction module to determine the i-th initial Bayer prediction feature, until the Mth initial Bayer prediction feature is determined by the Mth level Bayer feature prediction module as the Bayer prediction feature; the number of levels of the at least one level Bayer feature prediction module is the same as the number of levels of the at least one level Bayer feature extraction module.

5. The method according to claim 2 or 3, characterized in that: The M-th level Bayer feature extraction module is connected to the first level Bayer feature prediction module in the at least one level Bayer feature prediction module; the M-(i-1)-th level Bayer feature extraction module is connected to the i-th level Bayer feature prediction module; and the at least one level Bayer feature prediction module is used to perform at least one level Bayer domain image processing prediction based on the Bayer feature to determine the Bayer prediction feature, including: Performing Bayer domain image processing prediction on the Bayer feature through the first-level Bayer feature prediction module to determine a first initial Bayer prediction feature; The i-1th initial Bayer prediction feature corresponding to the i-1th level Bayer feature prediction module is upsampled through the i-th level prediction module in the at least one level Bayer feature prediction module, the upsampled i-1th initial Bayer prediction feature is fused with the M-(i-1)th initial Bayer prediction feature, and the fused feature is subjected to Bayer domain image processing prediction to determine the i-th initial Bayer prediction feature, until the M-th initial Bayer prediction feature is determined by the M-th level Bayer feature prediction module as the Bayer prediction feature.

6. The method according to claim 1, characterized in that The second network branch includes: at least one level first color feature extraction module and at least one level first color feature prediction module; the extracting and predicting the first color domain feature of the Bayer prediction feature to determine the first color prediction feature includes: Performing at least one level of feature extraction of a first color domain based on the Bayer prediction feature by the at least one level first color feature extraction module to determine a first color feature; By using the at least one first color feature prediction module, at least one first color domain image processing prediction is performed based on the first color feature to determine the first color prediction feature.

7. The method according to claim 6, characterized in that The step of extracting at least one level of features of the first color domain based on the Bayer prediction feature by using the at least one level first color feature extraction module to determine the first color feature includes: Extracting the first color feature of the Bayer prediction feature by using a first-level first color feature extraction module in the at least one-level first color feature extraction module to determine a first initial first color feature; The j-1th initial first color feature corresponding to the j-1th level first color feature extraction module is downsampled and feature extracted through the j-th level first color feature extraction module in the at least one level first color feature extraction module to determine the j-1th initial first color feature, until the Kth initial first color feature is determined through the K-th level first color feature extraction module as the first color feature; K is the level of the at least one level first color feature extraction module; j is not less than 2 and not greater than K.

8. The method according to claim 6 or 7, characterized in that: The K-th level first color feature extraction module is connected to the first level first color feature prediction module in the at least one level first color feature prediction module; the at least one level first color feature prediction module performs at least one level first color domain image processing prediction based on the first color feature to determine the first color prediction feature, including: Performing first color domain image processing prediction on the first color feature through the first-level first color feature prediction module to determine a first initial first color prediction feature; The j-1th initial first color prediction feature corresponding to the j-1th level first color feature prediction module is upsampled and predicted by the first color domain image processing through the j-th level first color feature prediction module in the at least one level first color feature prediction module to determine the j-1th initial first color prediction feature, until the K-th first color prediction feature is determined through the K-th level first color feature prediction module as the first color prediction feature; the number of levels of the at least one level first color feature extraction module is the same as the number of levels of the at least one level first color feature prediction module.

9. The method according to claim 6 or 7, characterized in that: The K-th level first color feature extraction module is connected to the first level first color feature prediction module in the at least one level first color feature prediction module; the K-(j-1)-th level first color feature extraction module is connected to the j-th level first color feature prediction module; and the at least one level first color feature prediction module is used to perform at least one level first color domain image processing prediction based on the first color feature to determine the first color prediction feature, including: Performing first color domain image processing prediction on the first color feature through the first-level first color feature prediction module to determine a first initial first color prediction feature; The j-1th initial first color prediction feature corresponding to the j-1th first color feature prediction module is upsampled through the j-th first color feature prediction module in the at least one first color feature prediction module, the upsampled j-1th initial first color prediction feature is fused with the K-(j-1)th initial first color feature, and the first color domain image processing prediction is performed on the fused feature to determine the j-th initial first color prediction feature, until the K-th initial first color prediction feature is determined by the K-th first color feature prediction module as the first color prediction feature.

10. The method according to claim 1, characterized in that The third network branch includes: at least one level second color feature extraction module and at least one level second color feature prediction module; the extraction and prediction of the first color prediction feature in the second color domain through the third network branch to determine the second color prediction feature includes: By using each level of the at least one level of the second color feature extraction module, extracting the feature of the second color domain for the first color prediction feature, determining the second color feature corresponding to each level of the second color feature extraction module, thereby determining at least one second color feature; The at least one level second color feature prediction module performs image processing prediction based on the at least one second color feature to determine the second color prediction feature.

11. The method according to claim 10, characterized in that The level of the at least one level second color feature prediction module is S-1 level; the at least one level second color feature prediction module is used to perform image processing prediction based on the at least one second color feature to determine the second color prediction feature, including: The Sth second color feature among the at least one second color feature is used as the first initial second color prediction feature; S is the number of the at least one level second color feature extraction module; The Lth initial second color prediction feature is upsampled through the Lth level second color feature prediction module among the at least one level second color feature prediction module, the upsampled Lth initial second color feature is fused with the SLth second color feature, the fused second color feature is subjected to second color domain image processing prediction, and the L+1th initial second color prediction feature is determined; until the Sth initial second color prediction feature is determined through the S-1th level second color feature prediction module as the second color prediction feature; L is not less than 1 and less than S.

12. The method according to any one of claims 2, 3, 6, 7, 10 or 11, characterized in that The Bayer feature extraction module, the Bayer feature prediction module, the first color feature extraction module, the first color feature prediction module, the second color feature extraction module and the second color feature prediction module include: a regional growing network implementation based on a dual attention mechanism of space and channel.

13. The method according to any one of claims 1, 2, 3, 6, 7, 10 or 11, characterized in that The method further comprises: Determine an initial image processing network and at least one set of training data; the initial image processing network includes the initial first network branch, the initial second network branch and the initial third network branch; each set of training data in the at least one set of training data includes: a training image and at least one sample image corresponding to the training image; Determine at least one training image prediction feature by extracting and predicting features based on the training images in each set of training data through at least one of the initial first network branch, the initial second network branch, and the initial third network branch; Perform image reconstruction based on the at least one training image prediction feature to determine at least one training reconstructed image; Iteratively updating network parameters of at least one of the initial first network branch, the initial second network branch, and the initial third network branch according to at least one difference between the at least one training reconstructed image and the at least one sample image until a preset training target is reached, and determining at least one of the first network branch, the second network branch, and the third network branch; The target image processing network is determined based on at least one of the first network branch, the second network branch, and the third network branch.

14. The method according to claim 13, characterized in that The at least one training reconstructed image includes: a Bayer domain reconstructed image; the Bayer domain reconstructed image is determined by performing image reconstruction on Bayer training features; the Bayer training features are determined by extracting and predicting features based on the training image by the initial first network branch; the at least one sample image includes: a Bayer domain sample image; the network parameters of at least one of the initial first network branch, the initial second network branch and the initial third network branch are iteratively updated according to at least one difference between the at least one training reconstructed image and the at least one sample image until a preset training target is reached, and determining at least one of the first network branch, the second network branch and the third network branch, including: Determine the Bayer domain norm loss corresponding to each pixel position according to the Bayer domain reconstructed pixel corresponding to each pixel position in the Bayer domain reconstructed image and the Bayer domain sample pixel corresponding to each pixel position in the Bayer domain sample image; Determining at least one correction gain corresponding to each pixel position according to a preset correction coefficient corresponding to the image sensor; the at least one correction gain represents a gain value corresponding to at least one Bayer domain image correction process; Determine the Bayer domain training loss corresponding to each pixel position according to the product of the at least one correction gain value and the Bayer domain norm loss, and determine the first training loss corresponding to the initial first network branch according to the Bayer domain training loss corresponding to each pixel position; Based on the first training loss, the network parameters of the initial first network branch are iteratively updated until a preset training target corresponding to the initial first network branch is reached, and the first network branch is determined.

15. The method according to claim 13, characterized in that The at least one training reconstructed image includes: a first color domain reconstructed image; the first color domain reconstructed image is determined by performing image reconstruction on a first color training feature; the first color training feature is determined by extracting and predicting features based on the Bayer training feature by the initial second network branch; the at least one sample image includes: a first color domain sample image; according to at least one difference between the at least one training reconstructed image and the at least one sample image, the network parameters of at least one of the initial first network branch, the initial second network branch and the initial third network branch are iteratively updated until a preset training target is reached, and at least one of the first network branch, the second network branch and the third network branch is determined, including: determining a first color domain norm loss between the first color domain reconstructed image and the first color domain sample image; According to a preset perception network, performing perceptual prediction on the first color domain reconstructed image and the first color domain sample image respectively, and determining a perceptual loss between the first color domain reconstructed image and the first color domain sample image; Performing a weighted combination on the first color domain norm loss and the perceptual loss to determine a second training loss corresponding to the initial second network branch; Based on the second training loss, the network parameters of the initial second network branch are iteratively updated until a preset training target corresponding to the initial second network branch is reached, and the second network branch is determined.

16. The method according to claim 13, characterized in that The at least one training reconstructed image includes: at least one second color domain reconstructed image; the at least one second color domain reconstructed image is determined by image reconstruction of at least one second color training feature corresponding to at least one resolution; the at least one second color training feature is determined by performing at least one level of downsampling, feature extraction and prediction by the initial third network branch based on the first color training feature; the at least one sample image includes: at least one second color domain sample image corresponding to the at least one resolution; according to at least one difference between the at least one training reconstructed image and the at least one sample image, the network parameters of at least one of the initial first network branch, the initial second network branch and the initial third network branch are iteratively updated until a preset training target is reached, and at least one of the first network branch, the second network branch and the third network branch is determined, including: For each of the at least one resolution, determining a second color domain norm loss between a second color domain reconstructed image and a second color domain sample image corresponding to each resolution; Determine at least one of a structural similarity loss and an edge loss between the second color domain reconstructed image corresponding to each resolution and the second color domain sample image; Based on at least one of the structural similarity loss and the edge loss, a weighted combination is performed with the second color domain norm loss to determine the second color domain training loss corresponding to each resolution, and then determine at least one second color domain training loss corresponding to the at least one resolution; Performing weighted merging on the at least one second color domain training loss to determine a third training loss corresponding to the initial third network branch; Based on the third training loss, the network parameters of the initial third network branch are iteratively updated until a preset training target corresponding to the initial third network branch is reached, and the third network branch is determined.

17. The method according to any one of claims 14 to 16, characterized in that: Before determining the initial image processing network and at least one set of training data, the method further includes: Using the image sensor, collecting at least one original training image in a static state; Determine a training image from the at least one original training image, and perform superposition and denoising on the at least one original training image to determine an initial sample image; By using a preset image processing algorithm module, the initial sample image is subjected to Bayer domain image processing to determine the Bayer domain sample image; and / or, By using the preset image processing algorithm module, performing first color domain image processing on the Bayer domain sample image to determine the first color domain sample image; and / or, Performing second color domain image processing on the first color domain sample image through a preset image processing algorithm module to determine at least one second color domain sample image; At least one of the second color domain sample image, the Bayer domain sample image, and the first color domain sample image is combined with the training image to determine a set of training data, thereby determining the at least one set of training data.

18. The method according to claim 17, characterized in that The step of performing first color domain image processing on the Bayer domain sample image by using the preset image processing algorithm module to determine the first color domain sample image includes: Performing first color domain image processing on the Bayer domain sample image through a preset image processing algorithm module to determine an initial first color domain sample image; The initial first color domain sample image is processed according to color preference to determine the first color domain sample image.

19. An image processing device, applied to a target image processing network, characterized in that: include: A network prediction module, configured to resample the original image acquired by the image sensor into a target Bayer format through a first network branch to obtain a Bayer domain image, extract and predict features based on the Bayer domain image, and determine a Bayer prediction feature; Through the second network branch, extract and predict the features of the first color domain on the Bayer prediction feature to determine the first color prediction feature; through the third network branch, extract and predict the features of the second color domain on the first color prediction feature to determine the second color prediction feature; The target image processing network is obtained by performing supervised training on at least one of the initial first network branch, the initial second network branch and the initial third network branch; A reconstruction module is used to perform image reconstruction based on the second color prediction feature and determine a reconstructed image corresponding to the original image.

20. An electronic device, characterized in that: include: A memory for storing executable instructions; A processor, configured to implement the method according to any one of claims 1 to 18 when executing the executable instructions stored in the memory.

21. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to cause a processor to execute and implement the method described in any one of claims 1 to 18.

Images