Image processing method, model training method and electronic equipment

Abstract

The invention discloses an image processing method, a model training method and electronic equipment, and belongs to the technical field of image processing, and the training method comprises the steps: obtaining training data which comprises a first exposure image, a second exposure image, a third exposure image and a reference image; carrying out portrait feature extraction on the reference image to obtain portrait features; performing global feature extraction on the exposure image set to obtain global features; carrying out feature fusion on the portrait features and the global features to obtain fusion features; a prediction image is obtained based on fusion feature training; a network loss is calculated based on the standard image and the predicted image. Therefore, information fusion can be carried out based on the exposure images corresponding to different exposure durations, dynamic range expansion is realized, and image reconstruction quality of different night scenes is improved. And on the other hand, the portrait features and the global features are fused for training, so that the detail recovery of the portrait area and the recovery of the overall effect of the night scene can be realized.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to an image processing method, a model training method and an electronic device. Background Art

[0002] With the continuous development of mobile phone and other terminal shooting applications, users have higher and higher requirements for mobile phone and other terminals. For example, users require mobile phones to meet different shooting scenes and capture high-quality images in each shooting scene.

[0003] However, due to the poor ambient light conditions, the night scene shooting scene causes the captured images to have problems such as blur, color cast, and high noise. As a special scene among the night scene shooting scenes, the night scene portrait shooting scene also causes the captured images to have problems such as blurry portraits and unnatural textures due to the poor ambient light conditions. Summary of the invention

[0004] The present application provides an image processing method, a model training method and an electronic device, which can improve the image quality of night scene shooting scenes.

[0005] In a first aspect, the present application provides a training method for an image processing model, the image processing model comprising a first network and a second network, the method comprising: obtaining training data, the training data comprising an exposure image set and a standard image corresponding to the exposure image set; wherein the exposure image set comprises a first exposure image, a second exposure image, a third exposure image and a reference image, wherein an exposure time of the first exposure image is greater than an exposure time of the second exposure image, an exposure time of the second exposure image is greater than an exposure time of the third exposure image, and the reference image is an image with the best image quality among the second exposure images; using the first network, performing portrait feature extraction on the reference image to obtain portrait features; using the second network, performing global feature extraction on the exposure image set to obtain global features; using the second network, performing feature fusion on the portrait features and the global features to obtain fused features; using the second network, training based on the fused features to obtain a predicted image; based on the standard image and the predicted image, training the image processing model to obtain a trained image processing model.

[0006] In this way, on the one hand, since the exposure image set includes exposure images corresponding to different exposure times, the second network can perform information fusion based on the exposure images corresponding to different exposure times to achieve dynamic range expansion and improve the image reconstruction quality of different night scenes. On the other hand, by fusing portrait features and global features for training, it is possible to restore the details of the portrait area and the overall effect of the night scene.

[0007] In one implementable manner, the portrait features include intermediate portrait features of multiple scales, and the global features include intermediate global features corresponding one-to-one to each scale of the multiple scales; and using the second network to perform feature fusion on the portrait features and the global features to obtain fused features includes: performing feature fusion on the intermediate portrait features of the same scale and the intermediate global features respectively to obtain the fused features.

[0008] In this way, since features of different scales focus on different details, the intermediate portrait features and intermediate global features of different scales are fused separately to learn information of different granularities to obtain a more comprehensive feature representation.

[0009] In an implementable manner, the fusing the first intermediate portrait features and the first intermediate global features of the same scale respectively includes: using a first GCA module to fusing the first intermediate portrait features and the first intermediate global features of the same scale respectively.

[0010] In this way, since GCA has the advantage of simple calculation, it is suitable for models configured in terminal devices such as mobile phones, so that terminal devices such as mobile phones can quickly process the collected images and improve the image output speed.

[0011] In one implementable manner, the reference image is a RAW image, and the using the first network to extract portrait features from the reference image to obtain portrait features includes: de-mosaicing the reference image to obtain a reference image in RGB format; identifying the portrait area in the reference image in RGB format to obtain a human body semantic segmentation map; splicing the reference image in RGB format with the human body semantic segmentation map to obtain a spliced ​​feature map; and using the first network to extract portrait features from the spliced ​​feature map to obtain portrait features.

[0012] In this way, the obtained portrait features may include the color features of the three RGB channels of the image, as well as the position information of the portrait in the image.

[0013] In one implementable manner, the identifying of the portrait area in the reference image in the RGB format to obtain a human body semantic segmentation map includes: using a trained human body mask extraction network to perform human body mask extraction processing on the reference image in the RGB format to obtain the human body semantic segmentation map.

[0014] In this way, the trained human body mask extraction network can be used to extract the human body semantic segmentation map, which can ensure the accuracy of the human body semantic segmentation map.

[0015] In one implementable manner, the first network includes a first encoder and a first decoder; the using the first network to extract portrait features from the spliced ​​feature map to obtain portrait features includes: using the first encoder to encode the spliced ​​feature map to obtain first intermediate portrait features of multiple scales; using the first decoder to respectively decode the first intermediate portrait features of the multiple scales to obtain second intermediate portrait features of multiple scales; wherein each scale of the second intermediate portrait feature corresponds one-to-one to each scale of the first intermediate portrait feature; wherein the portrait features include the first intermediate portrait features of the multiple scales and the second intermediate portrait features of the multiple scales.

[0016] In this way, the first network can use the UNet network to extract intermediate portrait features of multiple scales.

[0017] In one implementable manner, the second network includes a second encoder and a second decoder; the second network is used to perform global feature extraction on the set of exposed images to obtain global features, including: using the second encoder to encode the set of exposed images to obtain first intermediate global features of multiple scales; using the second decoder to decode the first intermediate global features of the multiple scales to obtain second intermediate global features of multiple scales; wherein each scale of the second intermediate global feature corresponds one-to-one to each scale of the first intermediate global feature; wherein the global features include the first intermediate global features of the multiple scales and the second intermediate global features of the multiple scales.

[0018] In this way, the second network can use the UNet network to extract the intermediate global features that correspond one-to-one to each intermediate portrait feature, so as to further perform feature fusion processing on the intermediate portrait features and the intermediate global features of the same size.

[0019] In one implementable manner, the encoding and processing of the exposure image set by the second encoder to obtain first intermediate global features of multiple scales includes: performing convolution calculation on the exposure image set by the first convolution layer in the second encoder to obtain a first feature map; performing downsampling processing based on the first feature map by the first downsampling layer in the second encoder to obtain a second feature map of the first scale; performing fusion processing on the global features of different exposure images in the second feature map by the first feature fusion layer in the second encoder to obtain a first intermediate global feature of the first scale; performing downsampling processing based on the first intermediate global features of the first scale by the second downsampling layer in the second encoder to obtain a third feature map of the second scale; performing fusion processing on the global features of different exposure images in the third feature map by the second feature fusion layer in the second encoder to obtain a first intermediate global feature of the second scale; the first scale is different from the second scale.

[0020] In one implementable manner, the first feature fusion layer includes a standardization module, a residual module and a second GCA module, and the first feature fusion layer in the second encoder is used to fuse the global features of the different exposure images in the second feature map to obtain the first intermediate global feature of the first scale, including: using the standardization module to standardize the second feature map; using the residual module to perform residual processing on the second feature map after standardization; using the second GCA module to fuse the global features of the different exposure images in the second feature map after residual processing; and splicing the fused second feature map and the second feature map to obtain the first intermediate global feature of the first scale.

[0021] In one implementable manner, the image processing model is trained based on the standard image and the predicted image to obtain a trained image processing model, including: calculating a first loss based on the standard image and the predicted image; calculating a second loss corresponding to a portrait area based on the standard image, the predicted image and the human semantic segmentation map; determining a third loss based on the first loss and the second loss; and when the third loss is less than a loss threshold, ending the training of the image processing model to obtain a trained image processing model.

[0022] In this way, the global quality of image processing can be controlled through the first loss, and the detail quality of the portrait area of ​​the image processing can be controlled through the second loss, thereby achieving the effect of restoring both the image details and the image as a whole.

[0023] In one achievable manner, the training data includes a set of exposure images corresponding to a night scene portrait photography scene and a standard image corresponding to the set of exposure images.

[0024] In a second aspect, the present application provides an image processing method, which should be used in an electronic device, wherein the electronic device is configured with an image processing model trained by the method described in any one of the first aspects, and the method includes: obtaining an image to be processed; using a first network in the image processing model to extract portrait features of the image to be processed to obtain portrait features; using a second network in the image processing model to extract global features of the image to be processed to obtain global features; using the second network to fuse the portrait features with the global features to obtain fused features; using the second network to obtain a target image based on the fused features.

[0025] In this way, the image processing model obtained through the above training can be configured in electronic devices such as mobile phones. In this way, when users use mobile phones or other electronic devices configured with the above image processing model to take images in night scenes or night portrait scenes, they can output images with clear textures and good details in the portrait area, thereby improving the user experience.

[0026] In a third aspect, the present application provides an electronic device, comprising a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program code, the computer program code comprises computer instructions, and when the processor executes the computer instructions, the electronic device executes a method as described in any one of the first aspect or the second aspect.

[0027] In a fourth aspect, the present application provides a chip system, comprising a processor; the processor is coupled to a memory, the memory is used to store computer program code, the computer program code comprises computer instructions, and when the processor executes the computer instructions, the method as described in any one of the first aspect or the second aspect is executed.

[0028] In a fifth aspect, the present application provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed on a computer, the computer executes the method as described in any one of the first aspect or the second aspect.

[0029] In a sixth aspect, the present application provides a computer program product, comprising: a computer program or instructions, which, when executed on a computer, causes the computer to execute a method as described in any one of the first aspect or the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A flowchart of a method for training an image processing model provided in an embodiment of the present application;

[0031] Figure 2 A schematic diagram of an algorithm structure of an image processing model provided in an embodiment of the present application;

[0032] Figure 3 A flowchart of another method for training an image processing model provided in an embodiment of the present application;

[0033] Figure 4 A flowchart of extracting portrait features from a reference image provided in an embodiment of the present application;

[0034] Figure 5 A schematic diagram of an algorithm structure of an image processing model provided in an embodiment of the present application;

[0035] Figure 6 A schematic diagram of the structure of a feature fusion layer provided in an embodiment of the present application;

[0036] Figure 7 A schematic diagram of the structure of a residual module provided in an embodiment of the present application;

[0037] Figure 8 A schematic diagram of the structure of a GCA module provided in an embodiment of the present application;

[0038] Fig. 9 A flow chart of loss calculation provided in an embodiment of the present application;

[0039] Fig.10 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0040] Fig.11 is a software structure block diagram of an electronic device according to an embodiment of the present application;

[0041] Fig.12 A workflow diagram of an image processing method provided in an embodiment of the present application;

[0042] Fig.13 A structural block diagram of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] With the continuous development of mobile phone and other terminal shooting applications, users have higher and higher requirements for mobile phone and other terminals. For example, users require mobile phones to meet different shooting scenes and capture high-quality images in each shooting scene.

[0044] However, due to the poor ambient light conditions, the night scene shooting scene causes the captured images to have problems such as blur, color cast, and high noise. As a special scene among the night scene shooting scenes, the night scene portrait shooting scene also causes the captured images to have problems such as blurry portraits and unnatural textures due to the poor ambient light conditions.

[0045] Based on this, an embodiment of the present application provides an image processing model, which includes a first network and a second network. In the training stage, the first network can be used to learn the portrait features of the portrait area, and the second network can be used to learn the global features of images taken with different exposure times. Then, the portrait features are fused with the global features to achieve the restoration of the details of the portrait area and the overall effect of the night scene. In this way, by using the image processing model obtained by the above training to process the collected night scene or night scene portrait image, it is possible to restore the details of the portrait area and ensure the restoration of the overall effect of the night scene, thereby improving the quality of the output image.

[0046] The following describes a method for training an image processing model provided in an embodiment of the present application.

[0047] The training method of the image processing model provided in the embodiment of the present application can be implemented by deploying a neural network model and computer program code in software form in a hardware computing environment. Available hardware computing environments include: personal computers, servers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, programmable consumer electronic devices, cloud servers, server instances, supercomputers, etc.

[0048] Figure 1 A flowchart of a method for training an image processing model provided in an embodiment of the present application, Figure 2 The following is a schematic diagram of the algorithm structure of an image processing model provided in an embodiment of the present application. Figure 1 and 2 As shown, the training method of the image processing model provided in the embodiment of the present application may include the following steps:

[0049] S101, obtaining training data.

[0050] In the embodiment of the present application, the training data may include multiple sets of training samples. Each set of training samples includes a pair of exposure image sets and a standard image corresponding to the exposure image set. The exposure image set may include multiple exposure images corresponding to different exposure times. The standard image is a high-definition image after image enhancement processing.

[0051] For example, the exposure image set includes a first exposure image corresponding to a short exposure time, a second exposure image corresponding to a medium exposure time, a third exposure image corresponding to a long exposure time, and a reference image under the same night portrait scene. The short exposure time is less than the medium exposure time, and the medium exposure time is less than the long exposure time. The reference image is a frame with the best image quality among the images corresponding to the medium exposure time. The first exposure image, the second exposure image, the third exposure image, and the reference image may also be referred to as a short frame, a medium frame, a long frame, and a reference frame.

[0052] When shooting night scenes, the second exposure image is usually darker and details are lost. In this way, the third exposure image can be used to make up for the details of the darker areas in the second exposure image. For brighter night scenes including light billboards, the brighter areas such as light billboards in the third exposure image may be overexposed. In this case, the first exposure image can be used to make up for the details of the overexposed areas. Reference images can better guide the network to pay attention to important scene content and structural information, and optimize the output results.

[0053] It can be seen that in the training data provided by the embodiment of the present application, the first exposure image can retain scene details, the third exposure image can provide dynamic range, and the reference image can provide overall information. In this way, diversified training data can improve the generalization ability of the model and improve the reconstruction quality of different night scene portrait scenes.

[0054] In some embodiments, the images in the exposure image set may all be original images, also referred to as RAW images. A RAW image refers to the original unprocessed image data collected by an image collector of an electronic device. That is, the embodiment of the present application may train an image processing model based on a RAW image.

[0055] like Figure 2 As shown, the image processing model provided in the embodiment of the present application includes two network branches, namely a first network and a second network. The reference image in the training data is used as the input of the first network for training, and the exposure image set in the training data is used as the input of the second network for training. For the specific training process of the first network and the second network, please refer to the description of steps S102 to S106.

[0056] S102, using the first network to extract portrait features from the reference image to obtain portrait features.

[0057] In the embodiment of the present application, the reference image in the training data is used as the input of the first network to extract the portrait features and obtain the portrait features.

[0058] like Figure 3 As shown, the first network may include a preprocessing module, a human mask extraction network and a portrait extraction module.

[0059] In some embodiments, Figure 4 As shown, the portrait feature extraction of the reference image can be achieved by the following steps:

[0060] S1021, using a preprocessing module to perform a demosaicing process on a reference image in an RGB Bayer format to obtain a reference image in an RGB format.

[0061] Since the color space of a RAW image is usually in RGB Bayer format, in the embodiment of the present application, if the reference image is a RAW image, demosaicing can be performed on the reference image first to convert the reference image in RGB Bayer format into a reference image in RGB format.

[0062] In one possible implementation, Figure 5 As shown, the preprocessing module may include two convolutional layers (CONV1 and CONV2) and three residual modules (resblock1, resblock2 and resblock3) between the two convolutional layers. In this way, de-mosaicing of the reference image can be implemented in the following manner: using two convolutional layers and three residual modules between the two convolutional layers, the reference image in RGB Bayer format is preprocessed and a reference image in RGB format is output. The reference image in RGB format is a three-channel feature map.

[0063] S1022, using a human mask extraction network to identify a human portrait region in a reference image in RGB format, and obtaining a human semantic segmentation map.

[0064] The human mask extraction network may be a trained human mask extraction network. In one achievable manner, the trained human mask extraction network may perform binary separation of the portrait region and the background region in the reference image in RGB format. Thus, after the reference image in RGB format is input into the trained human mask extraction network, an accurate human semantic segmentation map may be output.

[0065] For example, Figure 5 As shown in FIG. 1 , the human body semantic segmentation map includes a background area and a portrait area. The black area represents the background area, and the white area represents the portrait area. It can be seen that the human body semantic segmentation map can represent the position of the portrait area.

[0066] It should be noted that if the reference image in RGB format does not include a portrait area, a human body semantic segmentation map that is entirely background, that is, a human body semantic segmentation map that is completely black, can be obtained.

[0067] It should also be noted that the embodiments of the present application are only illustrative of the use of a human mask extraction network to obtain a human semantic segmentation map as an example, and do not limit the method of obtaining the position information of the portrait area. For example, in some embodiments, other tools that can extract the position information of the portrait area can also be used to obtain the position information of the portrait area. Among them, the position information of the portrait area is not limited to the method of the human semantic segmentation map. For example, the position information of the portrait area can also be represented by the correspondence between pixels and human semantic labels. For example, the human semantic label corresponding to the pixels in the portrait area is portrait, and the human semantic label corresponding to the pixels in other areas except the portrait area is background.

[0068] S1023, splicing the reference image in RGB format with the human body semantic segmentation map to obtain a spliced ​​feature map.

[0069] Since the reference image in RGB format is a three-channel feature map, and the human semantic segmentation map is a one-channel feature map, after splicing the reference image in RGB format and the human semantic segmentation map in the channel size, a four-channel spliced ​​feature map is obtained. The four-channel spliced ​​feature map obtained in this way includes both the RGB color features of the reference image and the portrait position information of the reference image.

[0070] S1024, using a portrait extraction module to extract portrait features from the spliced ​​feature map to obtain portrait features.

[0071] After the four-channel splicing feature map is obtained, the four-channel splicing feature map can be input into the portrait extraction module. Then, the portrait extraction module is used to extract the portrait feature from the splicing feature map to obtain the portrait feature.

[0072] The embodiments of the present application do not limit the specific structure of the portrait extraction module. For example, the portrait extraction module can adopt network architectures such as Attention-to-Scale, JPPNet, and UNet. The Attention-to-Scale network architecture is a network architecture that uses pyramid pooling to fuse features of different scales. The JPPNet network architecture is a network architecture that jointly utilizes global human body information and local part information. The UNet network architecture is an encoder-decoder network architecture.

[0073] The following takes the UNet network architecture used in the portrait extraction module as an example to illustrate the process of extracting portrait features from the spliced ​​feature map using the portrait extraction module to obtain portrait features.

[0074] like Figure 5As shown, the portrait extraction module includes a first encoder and a first decoder. The first encoder can gradually downsample the input spliced ​​feature map to obtain first intermediate portrait features of different scales. The first decoder can gradually upsample the first intermediate portrait features of different scales to obtain second intermediate portrait features corresponding to the scales of the first intermediate portrait features, thereby restoring the feature space size and forming a symmetrical encoding-decoding structure.

[0075] In this way, since the input splicing feature map includes both the RGB color features of the reference image and the portrait position information of the reference image, the portrait extraction module can output multiple layers of feature maps of different scales to express different levels of information such as details and textures in the image.

[0076] like Figure 5 As shown, the first encoder may include a convolutional layer (CONV3), three downsampling layers, and three residual modules (resblock4, resblock5, and resblock6). The first decoder may include a convolutional layer (CONV4), three upsampling layers, and three residual modules (resblock7, resblock8, and resblock9).

[0077] Thus, in one implementable method, the first encoder is used to encode the spliced ​​feature map to obtain the first intermediate portrait features of multiple scales, which can be implemented in the following manner: the convolution layer CONV3 is used to perform convolution calculation on the spliced ​​feature map to obtain the feature map M0. The feature map M0 is downsampled by the downsampling layer to obtain the feature map M1 of the first scale. Afterwards, the residual module resblock4 is used to perform residual processing on the feature map M1 to obtain the feature map M2 of the first scale, wherein the residual module can accelerate the feature learning of the feature map M1. Afterwards, the feature map M2 of the first scale is downsampled and residual processed again as the basis to obtain the feature map M3 of the second scale. Similarly, the feature map M3 of the second scale is downsampled and residual processed again as the basis to obtain the feature map M4 of the third scale.

[0078] The feature map M2 of the first scale, the feature map M3 of the second scale, and the feature map M4 of the third scale constitute the first intermediate portrait feature. The first scale, the second scale, and the third scale are all different. For example, the first scale is 128×128×32, the second scale is 64×64×64, and the third scale is 32×32×128.

[0079] In one implementable manner, the first decoder is used to decode the first intermediate portrait features of multiple scales respectively to obtain the second intermediate portrait features of multiple scales. This can be implemented in the following manner: the residual module resblock7 is used to perform residual processing on the feature map M4 output by the encoder to obtain a feature map M5 of the third size. Then, the feature map M5 of the third size is upsampled and residual processed to obtain a feature map M6 of the second size. After that, based on the feature map M6 of the second size, upsampling and residual processing are performed again to obtain a feature map M7 of the first size. Finally, the feature map M7 of the first size is input into the convolution layer CONV4 for deconvolution calculation to obtain a feature map M8 of the same size as the input image.

[0080] Among them, the feature map M5 of the third scale, the feature map M6 of the second scale, and the feature map M7 of the first scale constitute the second intermediate portrait feature.

[0081] In summary, the embodiment of the present application can use the first network to extract portrait features from the reference image to obtain first intermediate portrait features of multiple different scales and second intermediate portrait features of multiple different scales.

[0082] S103, using the second network, extracting global features from the set of exposed images to obtain global features.

[0083] In the embodiment of the present application, the exposure image set is used as the input of the second network, and global features are extracted from the exposure image set. Since the exposure image set includes exposure images corresponding to different exposure times, the second network can perform dynamic range expansion based on the exposure images corresponding to different exposure times, fuse the information of the exposure images corresponding to different exposure times, and improve the quality of the night scene output.

[0084] The embodiment of the present application does not limit the specific structure of the second network. For example, the second network can adopt a network architecture such as Attention-to-Scale, JPPNet, and UNet.

[0085] The following takes the UNet network architecture used in the second network as an example to illustrate the process of using the second network to extract global features from the exposure image set and obtain global features.

[0086] like Figure 5 As shown, the second network may include a second encoder and a second decoder. The second encoder may encode the exposure image set to obtain first intermediate global features of multiple scales. The second decoder may decode the first intermediate global features of multiple scales to obtain second intermediate global features of multiple scales.

[0087] In this way, since the input exposure image set includes images with different exposure times and reference images, the second network can be used to fuse images with different exposure times, repair images under different exposure conditions, and obtain images with both details and high dynamic range. The model trained in this way can perform image reconstruction under different exposure conditions, enhance the robustness of the model, and avoid over-reliance on a certain exposure condition.

[0088] like Figure 5 As shown, the second encoder may include a convolution layer (CONV5), three downsampling layers (for example, the three sampling layers are the first downsampling layer, the second downsampling layer and the third downsampling layer) and three feature fusion layers (K1, K2 and K3). The second decoder may include a convolution layer (CONV6), three upsampling layers and three feature fusion layers (K4, K5 and K6).

[0089] Thus, in one implementable manner, the second encoder is used to encode the set of exposure images to obtain the first intermediate global features of multiple scales, which can be implemented in the following manner: first, the convolution layer CONV5 (CONV5 can also be called the first convolution layer) is used to perform convolution calculation on the input set of exposure images to obtain the feature map N0 (the feature map N0 can also be called the first feature map). The feature map N0 is downsampled by the downsampling layer to obtain the feature map N1 of the first scale (the feature map N1 can also be called the second feature map). Afterwards, the global features of different exposure images in the feature map N1 are fused by the first feature fusion layer K1 to obtain the feature map N2 of the first scale. Afterwards, based on the feature map N2 of the first scale, the second downsampling layer and the second feature fusion layer K2 are used to perform downsampling and fusion processing again to obtain the feature map N3 of the second scale. Similarly, the feature map N3 of the second scale is continued to be downsampled and fused again to obtain the feature map N4 of the third scale.

[0090] Among them, the feature map N2 of the first scale, the feature map N3 of the second scale, and the feature map N4 of the third scale constitute the first intermediate global feature.

[0091] In one possible implementation, the second decoder is used to decode the first intermediate global features of multiple scales respectively to obtain the second intermediate global features of multiple scales. This can be implemented in the following manner: the feature fusion layer K4 is used to perform feature fusion processing on the feature map N4 output by the encoder to obtain a feature map N5 of the third size. Then, the feature map N5 of the third size is upsampled and feature fused to obtain a feature map N6 of the second size. After that, based on the feature map N6 of the second size, upsampling and feature fusion are performed again to obtain a feature map N7 of the first size. Finally, the feature map N7 of the first size is input into the deconvolution layer for deconvolution calculation to obtain a feature map N8 of the same size as the input image.

[0092] In the embodiment of the present application, the feature fusion layer is used to fuse the global features of images with different exposures. The present application does not limit the specific implementation method of the feature fusion layer.

[0093] In one possible implementation, Figure 6 As shown, the feature fusion layer may include a normalization module (LayerNorm), a residual module (resblock) and a global context attention (GCA) module. In this way, the feature fusion layer is used to fuse the global features of images with different exposures, which can be achieved in the following way: the feature map N1 is normalized by the normalization module; the residual module is used to perform residual processing on the normalized feature map N1; the GCA module is used to fuse the global features of images with different exposures in the residual-processed feature map N1. Finally, the fused feature map N1 and the unfused feature map N1 are concatenated to obtain the feature map N2 of the first scale.

[0094] Among them, the residual module in the feature fusion layer can perform local feature learning, and the GCA module can enhance the influence of global features on local features by changing the size of features, thereby performing global feature learning. The combination of the two is conducive to feature recovery.

[0095] For example, Figure 7 As shown in the figure, the residual module can be composed of two convolutional layers, so that the features of the input residual module can be calculated by convolution twice to perform local feature learning. After each convolution calculation, a nonlinear factor can be introduced through the activation function ReLU to better mine relevant features, fit the training data, and keep the convergence speed of the network in a stable state.

[0096] It should be noted that the structures of all residual modules in the embodiments of the present application can refer to Figure 7The structure shown in FIG. 1 , for example, the residual module mentioned in the preprocessing module and the portrait extraction module can refer to Figure 7 The structure shown will not be described in detail later.

[0097] Among them, the GCA module is based on an attention learning mechanism, which has the advantage of simple calculation and is suitable for models configured in terminal devices such as mobile phones.

[0098] For example, Figure 8 As shown, the GCA module may include multiple convolutional layers, a normalized exponential function layer (softmax), and an activation function layer (Leaky ReLU).

[0099] Taking the input feature of the GCA module with a scale of b×f×H×W as an example, it first undergoes a convolution calculation and outputs a feature map with a scale of b×1×1×W. Then, the feature map with a scale of b×1×1×W is input into softmax, and the feature map after softmax normalization is matrix multiplied with the feature map with a scale of b×f×H×W. After that, the features output by the multiplication operation are sequentially input into two convolutional layers for convolution calculation, and a feature map with a scale of b×f / r×1×1 and a feature map with a scale of b×f×1×1 are output respectively. Among them, the feature map with a scale of b×f / r×1×1 can be processed by a Leaky ReLU layer before being input into the convolutional layer to better mine relevant features and fit the training data. Finally, the feature map with a scale of b×f×1×1 and the feature map with a scale of b×f×H×W can be added to obtain the output of the GCA module.

[0100] Where b represents the number of images processed simultaneously, that is, the number of images in a batch. f represents the number of channels of the feature, where different convolutional layers have different numbers of output channels. H represents the height of the feature map, and H represents the width of the feature map. r represents the reduction ratio of the channel space.

[0101] In this way, the GCA module can enhance the ability to communicate and integrate information.

[0102] Similarly, the above fusion processing method can be used to obtain feature maps N3, N4, N5, N6 and N7. Please refer to the above description for details, which will not be repeated here.

[0103] In summary, the embodiment of the present application can use the second network to perform global feature learning on the input exposure image set to obtain global features.

[0104] like Figure 2As shown, in the embodiment of the present application, the second network includes two inputs, one input is a set of exposure images, and the other input is the portrait features output by the first network. The input set of exposure images can be used for the second network to learn global features of different exposure conditions.

[0105] In this way, the embodiment of the present application can obtain a model that can restore the details of the portrait area and ensure the restoration of the overall effect of the night scene by inputting the portrait features into the second network and fusing them with the global features learned by the second network. For details, please refer to the description of step S104 below.

[0106] S104: Using the second network, the portrait feature and the global feature are fused to obtain a fused feature.

[0107] like Figure 5 As shown, the portrait features extracted by the portrait extraction module may include intermediate portrait features including portrait information of multiple different scales. Correspondingly, the global features extracted by the second network also include intermediate global features including global information of multiple different scales. Among them, the intermediate portrait features and the intermediate global features have a one-to-one correspondence in each scale. For example, the intermediate portrait features include a feature map M2 of the first scale, a feature map M3 of the second scale, and a feature map M4 of the third scale. The intermediate global features include a feature map N2 of the first scale, a feature map N3 of the second scale, and a feature map N4 of the third scale. In this way, the embodiments of the present application can respectively fuse the intermediate portrait features of the same scale with the intermediate global features to obtain fused features.

[0108] Exemplarily, the second network can be used to perform feature fusion on the feature map M2 of the first scale and the feature map N2 of the first scale, perform feature fusion on the feature map M3 of the second scale and the feature map N3 of the second scale, and perform feature fusion on the feature map M4 of the third scale and the feature map N4 of the third scale.

[0109] Similarly, the feature map M5 of the third scale can be fused with the feature map N5 of the third scale, the feature map M6 of the second scale can be fused with the feature map N6 of the second scale, and the feature map M7 of the first scale can be fused with the feature map N7 of the first scale through the second network.

[0110] In one possible implementation, in order to better learn the portrait features and better fuse the portrait features with the global features, each intermediate portrait feature output by the portrait extraction module is processed once by GCA and then passed to the second network, and the intermediate portrait features are fused with the intermediate global features by the GCA module.

[0111] The structure of the GCA module can be referred to Figure 8The GCA module shown will not be described in detail here.

[0112] It should be noted that, for the sake of distinction, the GCA module used in the present application to fuse the portrait feature with the global feature can be referred to as the first GCA module. The GCA module used in the feature fusion layer can be referred to as the second GCA module.

[0113] S105, using the second network to obtain a predicted image based on fusion feature training.

[0114] For example, Figure 5 As shown, the feature map N8 output by the convolutional layer CONV6 is the predicted image.

[0115] S106, training the image processing model based on the standard image and the predicted image to obtain a trained image processing model.

[0116] The embodiment of the present application can calculate the network loss based on the standard image and the predicted image, so as to make the predicted image as close to the standard image as possible, so as to achieve the effect of image restoration.

[0117] The training process of the image processing model is an iterative process. Through multiple iterations, the network parameters inside the image processing model are continuously optimized and updated to make the network loss continue to converge. When the network loss is less than the preset threshold, the training ends and a trained image processing model is obtained.

[0118] In one possible implementation, Fig. 9 As shown, based on the standard image and the predicted image, the network loss is calculated, which can be achieved in the following way:

[0119] S1061, calculate the first loss based on the standard image and the predicted image.

[0120] In one implementation, the first loss may be a mean square error between a standard image and a predicted image.

[0121] Exemplarily, the first loss L1=MSE(output-GT), where output represents the predicted image and GT represents the standard image.

[0122] In this way, the global quality of image processing can be controlled through the above-mentioned first loss.

[0123] S1062, based on the standard image, the predicted image and the human body semantic segmentation map, calculate the second loss corresponding to the portrait area.

[0124] In one achievable manner, the second loss may be a perceptual loss based on a VGG network, and the second loss may be used to compare the similarity between the standard image and the predicted image in the feature space. Specifically, the embodiment of the present application may focus on the quality of the portrait area in the image by multiplying the standard image and the predicted image with the human body semantic segmentation map.

[0125] Exemplarily, the second loss L2 = VGG (output × bodymask - GT × bodymask). Where bodymask represents a human body semantic segmentation map. Thus, output × bodymask represents the portrait area in the predicted image, and GT × bodymask represents the portrait area in the standard image.

[0126] In this way, the detail quality of the portrait area in the image processing can be controlled through the second loss.

[0127] S1063: Determine a third loss based on the first loss and the second loss.

[0128] In some embodiments, the third loss may be the sum of the first loss and the second loss. In this way, the third loss can control both the global restoration of the image and the restoration of details in the portrait area. Exemplarily, the third loss L=L1+L2.

[0129] In some embodiments, the first loss and the second loss may be configured with different weight values ​​to adjust the ratio of the first loss to the second loss. In this way, the third loss may be expressed as: L = W1·L1+W2·L2. Where W1 represents the weight value corresponding to the first loss, and W2 represents the weight value corresponding to the second loss.

[0130] It should be noted that the weight values ​​W1 and W2 in the embodiment of the present application can be obtained through training, or can be configured based on experience during the training process, and the embodiment of the present application is not limited to this.

[0131] S1064: When the third loss is less than the loss threshold, end the training of the image processing model to obtain a trained image processing model.

[0132] When the third loss is less than the loss threshold, it indicates that the image processing model has been trained to convergence. In this case, the training of the image processing model can be terminated to obtain the trained image processing model.

[0133] When the third loss is greater than or equal to the loss threshold, it indicates that the image processing model has not been trained to convergence. In this case, back propagation can be used to continue to optimize the network parameters of the image processing model until the image processing model is trained to convergence to obtain a trained image processing model.

[0134] It should be noted that the embodiments of the present application are only illustrative of the training of the image processing model based on network loss, and do not limit the image processing model training method. For example, when the number of training times of the image processing model training reaches a preset number, the training can be terminated to obtain a trained image processing model.

[0135] In summary, the training method of the image processing model provided in the embodiment of the present application uses the first network to learn the portrait features of the portrait area. The second network is used to learn the global features of images taken with different exposure times. Then, the portrait features are fused with the global features to achieve the restoration of the details of the portrait area and the overall effect of the night scene. In this way, by using the image processing model obtained by the above training to process the collected night scene or night scene portrait image, it is possible to achieve the restoration of the details of the portrait area and ensure the restoration of the overall effect of the night scene, thereby improving the image quality.

[0136] In this way, the image processing model trained by the above embodiment can be configured in electronic devices such as mobile phones. In this way, when users use mobile phones or other electronic devices configured with the above image processing model to take images in night scenes or night portrait scenes, they can output images with clear textures and good details in the portrait area, thereby improving the user experience.

[0137] The structure of the electronic device in the embodiment of the present application is described below by taking a mobile phone as an example.

[0138] Fig.10 The hardware structure diagram of an electronic device provided in an embodiment of the present application is shown in FIG. Fig.10As shown, the electronic device 200 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc. The sensor module 280 may include a pressure sensor 280A, a gyroscope sensor 280B, an air pressure sensor 280C, a magnetic sensor 280D, an acceleration sensor 280E, a distance sensor 280F, a proximity light sensor 280G, a fingerprint sensor 280H, a temperature sensor 280J, a touch sensor 280K, an ambient light sensor 280L, a bone conduction sensor 280M, etc.

[0139] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0140] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0141] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0142] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0143] In some embodiments, the processor 210 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface 230, etc.

[0144] The wireless communication function of the electronic device 200 can be implemented through the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modem processor and the baseband processor.

[0145] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 200 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0146] The mobile communication module 250 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 200. The mobile communication module 250 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 250 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 250 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 250 can be set in the processor 210. In some embodiments, at least some of the functional modules of the mobile communication module 250 can be set in the same device as at least some of the modules of the processor 210.

[0147] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 270A, a receiver 270B, etc.), or displays an image or video through a display screen 294. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 210 and be set in the same device as the mobile communication module 250 or other functional modules.

[0148] The wireless communication module 260 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 200. The wireless communication module 260 can be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 210. The wireless communication module 260 can also receive the signal to be sent from the processor 210, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0149] In some embodiments, the antenna 1 of the electronic device 200 is coupled to the mobile communication module 250, and the antenna 2 is coupled to the wireless communication module 260, so that the electronic device 200 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0150] The electronic device 200 implements the display function through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or change display information.

[0151] The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 200 may include 1 or N display screens 294, where N is a positive integer greater than 1.

[0152] The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 200. The external memory card communicates with the processor 210 through the external memory interface 220 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0153] The internal memory 221 can be used to store computer executable program codes, which include instructions. The internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 200 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 210 executes various functional applications and data processing of the electronic device 200 by running instructions stored in the internal memory 221 and / or instructions stored in a memory provided in the processor.

[0154] The electronic device 200 can implement audio functions such as music playing and recording through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the headphone jack 270D, and the application processor.

[0155] The key 290 includes a power key, a volume key, etc. The key 290 can be a mechanical key or a touch key. The electronic device 200 can receive key input and generate key signal input related to user settings and function control of the electronic device 200.

[0156] Motor 291 can generate vibration prompts. Motor 291 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 294, motor 291 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0157] Indicator 292 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0158] The SIM card interface 295 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 200 by inserting it into the SIM card interface 295 or pulling it out from the SIM card interface 295. The electronic device 200 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 295 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 295 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 295 can also be compatible with different types of SIM cards. The SIM card interface 295 can also be compatible with external memory cards. The electronic device 200 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 200 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 200 and cannot be separated from the electronic device 200.

[0159] The software system of the electronic device 200 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system of the layered architecture is taken as an example to exemplify the software structure of the electronic device 200.

[0160] Fig.11 It is a software structure block diagram of the electronic device 200 according to an embodiment of the present application.

[0161] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0162] The application layer can include a series of application packages.

[0163] like Fig.11 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0164] Among them, the camera application can include night scene shooting mode and night scene portrait shooting mode.

[0165] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0166] like Fig.11 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0167] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0168] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0169] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0170] The phone manager is used to provide communication functions of the electronic device 200, such as management of call status (including connecting, hanging up, etc.).

[0171] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0172] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0173] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0174] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0175] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0176] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0177] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0178] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0179] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0180] A 2D graphics engine is a drawing engine for 2D drawings.

[0181] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0182] Fig.12 The following is a workflow diagram of an image processing method provided in an embodiment of the present application. Fig.12 As shown, the method may include the following steps:

[0183] S301, obtaining an image to be processed.

[0184] Exemplarily, in response to the user triggering the night scene shooting mode or the night scene portrait shooting mode, the image collector of the electronic device collects an image of the target object to obtain an image to be processed, wherein the image to be processed may be a RAW image.

[0185] S302, using the first network in the image processing model to extract portrait features from the image to be processed to obtain portrait features.

[0186] S303, using the second network in the image processing model, extracting global features from the image to be processed to obtain global features.

[0187] S304: Using the second network, fuse the portrait feature with the global feature to obtain a fused feature.

[0188] S305, using the second network to obtain a target image based on the fusion features.

[0189] Among them, steps S302 to S305 can refer to the description of steps S102 to S105, which will not be repeated here.

[0190] In this way, the image to be processed is used as the input of the image processing model, and after the image to be processed is subjected to image restoration processing, a target image is output. The target image may be an image displayed by an electronic device.

[0191] It should be noted that the image processing method provided in the embodiment of the present application can be used not only to perform image restoration processing on the to-be-processed images collected in the night scene shooting mode or the night scene portrait shooting mode, but also to perform image restoration processing on the to-be-processed images collected in other shooting modes using the image processing model, and the present application does not limit this.

[0192] In this way, by using an electronic device equipped with the image processing model trained by the embodiment of the present application, the output image can not only ensure the quality of the night scene, but also restore the portrait area to obtain an image with clear texture and clear portrait.

[0193] The various method embodiments described in this document may be independent solutions or may be combined according to internal logic, and these solutions all fall within the protection scope of this application.

[0194] It can be understood that, in the above-mentioned various method embodiments, the methods and operations implemented by the electronic device can also be implemented by components (such as chips or circuits) that can be used in the electronic device.

[0195] The above embodiments introduce the training method and image processing method of the image processing model provided by the present application. It is understandable that, in order to realize the above functions, the electronic device includes a hardware structure and / or software module corresponding to each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0196] The embodiment of the present application further provides a processing device, which includes at least one processor and a communication interface. The communication interface is used to provide information input and / or output to the at least one processor, and the at least one processor is used to execute the method in the above method embodiment.

[0197] It should be understood that the above-mentioned processing device can be a chip. Fig.13 , Fig.13 A structural block diagram of a chip provided in an embodiment of the present application. Fig.13 The chip shown may be a general-purpose processor or a dedicated processor. The chip 400 may include at least one processor 401. The at least one processor 401 may be used to support execution Figures 1 to 12 The technical solution shown in any one of the embodiments.

[0198] Optionally, the chip 400 may further include a transceiver 402, which is used to accept the control of the processor 401 and to support the execution Figures 1 to 12 The technical solution shown in any one of the embodiments. Optionally, Fig.13 The chip 400 shown may further include a storage medium 403. Specifically, the transceiver 402 may be replaced by a communication interface, and the communication interface provides information input and / or output for the at least one processor 401.

[0199] It should be noted that Fig.13The chip 400 shown can be implemented using the following circuits or devices: one or more field programmable gate arrays (FPGA), programmable logic devices (PLD), application specific integrated circuits (ASIC), system on chip (SoC), central processor unit (CPU), network processor (NP), digital signal processor (DSP), microcontroller unit (MCU), controller, state machine, gate logic, discrete hardware components, any other suitable circuits, or any combination of circuits capable of performing the various functions described throughout this application.

[0200] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in a processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0201] It should be noted that the processor in the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor can be combined and performed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0202] It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0203] According to the method provided in the embodiments of the present application, the embodiments of the present application also provide a computer program product, which includes: a computer program or instructions, when the computer program or instructions are run on a computer, the computer executes the method of any one of the embodiments of the method.

[0204] According to the method provided by the embodiments of the present application, the embodiments of the present application also provide a computer storage medium, which stores a computer program or instructions. When the computer program or instructions are run on a computer, the computer executes the method of any one of the embodiments of the method.

[0205] According to the method provided by an embodiment of the present application, an embodiment of the present application also provides an electronic device, including a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the method of any one of the embodiments of the method embodiment.

[0206] According to the method provided in the embodiment of the present application, the embodiment of the present application also provides a chip system, which includes a processor, the processor is coupled to a memory, and is used to execute a computer program or instruction stored in the memory. When the computer program or instruction is executed, the chip system can implement all or part of the steps in the method embodiment. The chip system can be composed of a chip, or it can include a chip and other discrete devices.

[0207] Those of ordinary skill in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0208] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0209] The computer storage medium, computer program product, and electronic device provided in the above-mentioned embodiments of the present application are all used to execute the method provided above. Therefore, the beneficial effects that can be achieved can refer to the corresponding beneficial effects of the method provided above, and will not be repeated here.

[0210] It should be understood that in each embodiment of the present application, the execution order of each step should be determined by its function and internal logic, and the size of the sequence number of each step does not mean the order of execution and does not limit the implementation process of the embodiment.

[0211] Each part of this specification is described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the device, computer storage medium, computer program product, and electronic device, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0212] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0213] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application.

Claims

1. A training method for an image processing model, characterized in that: The image processing model includes a first network and a second network, and the method includes: Acquire training data, the training data comprising an exposure image set and a standard image corresponding to the exposure image set; wherein the exposure image set comprises a first exposure image, a second exposure image, a third exposure image and a reference image, wherein the exposure time of the first exposure image is greater than the exposure time of the second exposure image, the exposure time of the second exposure image is greater than the exposure time of the third exposure image, and the reference image is an image with the best image quality among the second exposure images; Using the first network, extracting portrait features from the reference image to obtain portrait features; Using the second network, extracting global features from the set of exposure images to obtain global features; Using the second network, fusing the portrait feature with the global feature to obtain a fused feature; Using the second network, training based on the fusion features to obtain a predicted image; Based on the standard image and the predicted image, the image processing model is trained to obtain a trained image processing model.

2. The method according to claim 1, characterized in that The portrait features include intermediate portrait features of multiple scales, and the global features include intermediate global features corresponding to each scale in the multiple scales; The using the second network to fuse the portrait feature with the global feature to obtain a fused feature includes: The intermediate portrait features and the intermediate global features of the same scale are respectively fused to obtain the fused features.

3. The method according to claim 2, characterized in that The fusing the first intermediate portrait features and the first intermediate global features of the same scale respectively includes: The first intermediate portrait features and the first intermediate global features of the same scale are respectively fused by using the first GCA module.

4. The method according to claim 1, characterized in that: The reference image is a RAW image, and the first network is used to extract portrait features from the reference image to obtain portrait features, including: Performing a demosaicing process on the reference image to obtain a reference image in RGB format; Identify the human portrait area in the reference image in the RGB format to obtain a human body semantic segmentation map; The reference image in RGB format is spliced ​​with the human body semantic segmentation map to obtain a spliced ​​feature map; The first network is used to extract portrait features from the spliced ​​feature map to obtain portrait features.

5. The method according to claim 4, characterized in that The step of identifying the portrait area in the reference image in the RGB format to obtain a human body semantic segmentation map includes: The trained human mask extraction network is used to perform human mask extraction processing on the reference image in RGB format to obtain the human semantic segmentation map.

6. The method according to claim 4, characterized in that The first network includes a first encoder and a first decoder; and the step of extracting portrait features from the spliced ​​feature map using the first network to obtain portrait features includes: Using the first encoder to encode the spliced ​​feature map to obtain first intermediate portrait features of multiple scales; Using the first decoder to decode the first intermediate portrait features of the multiple scales respectively, to obtain second intermediate portrait features of the multiple scales; wherein each scale of the second intermediate portrait feature corresponds to each scale of the first intermediate portrait feature one by one; The portrait features include first intermediate portrait features of the multiple scales and second intermediate portrait features of the multiple scales.

7. The method according to claim 1, characterized in that The second network includes a second encoder and a second decoder; using the second network, extracting global features from the set of exposed images to obtain global features, including: Using the second encoder to encode the set of exposure images to obtain first intermediate global features at multiple scales; Using the second decoder to decode the first intermediate global features of the multiple scales respectively, to obtain second intermediate global features of the multiple scales; wherein each scale of the second intermediate global feature corresponds to each scale of the first intermediate global feature one by one; The global features include first intermediate global features of the multiple scales and second intermediate global features of the multiple scales.

8. The method according to claim 7, characterized in that The encoding process of the exposure image set by using the second encoder to obtain first intermediate global features of multiple scales includes: Using the first convolutional layer in the second encoder to perform convolution calculation on the set of exposure images to obtain a first feature map; Performing downsampling processing based on the first feature map using the first downsampling layer in the second encoder to obtain a second feature map of a first scale; Using a first feature fusion layer in the second encoder to fuse the global features of the different exposure images in the second feature map, to obtain a first intermediate global feature of a first scale; Using a second downsampling layer in the second encoder to perform downsampling processing based on the first intermediate global feature of the first scale, to obtain a third feature map of the second scale; The global features of the different exposure images in the third feature map are fused using a second feature fusion layer in the second encoder to obtain a first intermediate global feature of a second scale; the first scale is different from the second scale.

9. The method according to claim 8, characterized in that The first feature fusion layer includes a normalization module, a residual module, and a second GCA module. The first feature fusion layer in the second encoder is used to fuse the global features of the different exposure images in the second feature map to obtain the first intermediate global features of the first scale, including: Using the standardization module to perform standardization processing on the second feature map; Using the residual module to perform residual processing on the second feature map after the standardization processing; Using the second GCA module to fuse the global features of the different exposure images in the second feature map after the residual processing; The fused second feature map and the second feature map are spliced ​​together to obtain a first intermediate global feature of a first scale.

10. The method according to claim 4, characterized in that The step of training the image processing model based on the standard image and the predicted image to obtain a trained image processing model comprises: Calculating a first loss based on the standard image and the predicted image; Calculating a second loss corresponding to a portrait region based on the standard image, the predicted image, and the human body semantic segmentation map; determining a third loss based on the first loss and the second loss; When the third loss is less than the loss threshold, the training of the image processing model is terminated to obtain a trained image processing model.

11. The method according to claim 1, characterized in that: The training data includes an exposure image set corresponding to a night scene portrait photography scene and a standard image corresponding to the exposure image set.

12. An image processing method, characterized in that: The invention relates to an electronic device, wherein the electronic device is configured with an image processing model trained by the method according to any one of claims 1 to 11, wherein the method comprises: Get the image to be processed; Using the first network in the image processing model, extracting portrait features from the image to be processed to obtain portrait features; Using the second network in the image processing model, extracting global features of the image to be processed to obtain global features; Using the second network, fusing the portrait feature with the global feature to obtain a fused feature; The target image is obtained by utilizing the second network based on the fusion features.

13. An electronic device, characterized in that: It comprises a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program code, the computer program code comprises computer instructions, and when the processor executes the computer instructions, the electronic device executes the method as described in any one of claims 1-12.

14. A chip system, characterized in that: The chip system includes a processor; the processor is coupled to a memory, the memory is used to store computer program code, the computer program code includes computer instructions, and when the processor executes the computer instructions, the method as described in any one of claims 1-12 is executed.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program or instruction. When the computer program or instruction is executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 12.

16. A computer program product, characterized in that The computer program product comprises: a computer program or instructions, and when the computer program or instructions are executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 12.

Images