Photographing method and apparatus, terminal, and storage medium

Abstract

The present disclosure relates to a photographing method and device, a terminal and a storage medium. The method comprises: continuously collecting multiple frames of images when a photographing instruction is detected; obtaining multiple frames of images after ghosting is removed based on the multiple frames of images; inputting the multiple frames of images after ghosting is removed into an image fusion model to obtain a fused image; wherein the noise of the fused image is lower than the noise of any frame of image in the multiple frames of images, and the resolution of the fused image is higher than the resolution of any frame of image in the multiple frames of images; performing image enhancement processing on the fused image to obtain a target image and outputting the target image. The method has the characteristics of high integration and good processing effect.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of electronic devices, and in particular to a photographing method and device, a terminal, and a storage medium. BACKGROUND

[0002] The camera system of a mobile phone has evolved from a single camera to four or even five cameras today in order to meet different photographing needs of users. Super-resolution algorithms can achieve continuous zooming and high-magnification digital zooming on the basis of optical zooming. Figure 1 An example of an optical zoom lens can achieve 5x optical zooming, and in combination with a super-resolution algorithm, a maximum of 50x continuous digital zooming can be achieved. Therefore, the super-resolution algorithm is very important for improving the zooming capability of a terminal.

[0003] Super-resolution algorithms can be simply divided into single-frame super-resolution algorithms and multi-frame super-resolution algorithms. At present, multi-frame super-resolution algorithms are widely used in mobile phone zooming systems. That is, after a photographing button is pressed, a mobile phone will continuously capture multiple frames of images, and a high-quality image is output by image fusion of the multiple frames of images by an algorithm. The multi-frame super-resolution algorithm has obvious advantages over the single-frame super-resolution algorithm in terms of noise and resolution. SUMMARY

[0004] The present disclosure provides a photographing method and device, a terminal, and a storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, a photographing method is provided, comprising:

[0006] continuously capturing multiple frames of images when a photographing instruction is detected;

[0007] obtaining multiple frames of images after ghosting is removed based on the multiple frames of images;

[0008] inputting the multiple frames of images after ghosting is removed into an image fusion model to obtain a fused image; wherein the noise of the fused image is lower than the noise of any frame of image in the multiple frames of images, and the resolution of the fused image is higher than the resolution of any frame of image in the multiple frames of images;

[0009] performing image enhancement processing on the fused image to obtain a target image and output the target image.

[0010] Optionally, the image fusion model comprises multiple super-resolution models; wherein the super-resolution models correspond to image types one by one.

[0011] The inputting the multiple frames of images after ghosting is removed into an image fusion model to obtain a fused image comprises:

[0012] perform image segmentation on each single frame image in the multiple frame images after the ghosting is removed to obtain multiple region images;

[0013] input multiple region images belonging to the same image type in the multiple region images into a super-resolution model corresponding to the same image type to obtain a region fusion image of the same image type; wherein the noise of the region fusion image is lower than the noise of any region image of the multiple region images of the same image type, and the resolution of the region fusion image is higher than the resolution of any region image of the multiple region images of the same image type;

[0014] fuse region fusion images of different image types to obtain the fusion image.

[0015] Optionally, the performing image segmentation on each single frame image in the multiple frame images after the ghosting is removed to obtain multiple region images comprises:

[0016] for each single frame image in the multiple frame images after the ghosting is removed, determining the type of each pixel of the single frame image;

[0017] determining the image type to which each pixel belongs based on the type of each pixel;

[0018] dividing pixels belonging to the same image type in the same single frame image into the same region image to obtain the multiple region images.

[0019] Optionally, the fusing region fusion images of different image types to obtain the fusion image comprises:

[0020] fusing the region fusion images of different image types based on the weight of pixels in the region images corresponding to the region fusion images of different image types to obtain the fusion image.

[0021] Optionally, the super-resolution model comprises: multiple modules, each module comprising: a convolution layer, a normalization layer and an activation layer;

[0022] the inputting multiple region images belonging to the same image type in the multiple region images into a super-resolution model corresponding to the same image type to obtain a region fusion image of the same image type comprises:

[0023] inputting multiple region images belonging to the same image type in the multiple region images into a convolution layer of the super-resolution model corresponding to the same image type to obtain a convolution result;

[0024] inputting the convolution result into a normalization layer to obtain a normalization result;

[0025] obtain a region fusion image of the same image type based on the normalization result and the activation layer, wherein an input of the activation layer is the normalization result.

[0026] Optionally, the inputting the multiple region images of the same image type into the convolution layer of the super-resolution model corresponding to the same image type to obtain a convolution result comprises:

[0027] converting the M rows and L columns of convolution kernels of the convolution layer in the super-resolution model corresponding to the same image type into an M row and P×L column two-dimensional convolution matrix, wherein the M, P and L are positive integers;

[0028] obtaining multiple two-dimensional image matrices to be convolved based on the size of the convolution kernel, a preset convolution sliding step, a preset padding size and the multiple region images, wherein each column of the two-dimensional image matrix to be convolved has P×L data, one column of data is data to be intercepted when the convolution kernel performs one convolution operation on a padded region image, and the number of columns of the two-dimensional image matrix to be convolved is a maximum convolution number supported by the padded region image;

[0029] multiplying the two-dimensional convolution matrix and each two-dimensional image matrix to be convolved to obtain the convolution result.

[0030] Optionally, the image type comprises at least one of the following:

[0031] a texture feature image;

[0032] a color feature image;

[0033] a shape feature image;

[0034] a spatial relationship image.

[0035] Optionally, the obtaining the multiple images after removing ghosting based on the multiple images comprises:

[0036] aligning the multiple images;

[0037] performing ghosting removal processing on the aligned multiple images to obtain the multiple images after removing ghosting.

[0038] Optionally, the performing image enhancement processing on the fusion image to obtain a target image and outputting the target image comprises:

[0039] performing sharpening processing on the fusion image to obtain the target image and output the target image.

[0040] According to a second aspect of the embodiments of the present disclosure, a photographing device is provided, comprising:

[0041] The acquisition module is configured to continuously acquire multiple frames of images when a photographing instruction is detected.

[0042] The removal module is configured to obtain multiple frames of images after ghosting is removed based on the multiple frames of images.

[0043] The fusion module is configured to input the multiple frames of images after ghosting is removed into an image fusion model to obtain a fusion image; wherein noise of the fusion image is lower than noise of any frame of image in the multiple frames of images, and resolution of the fusion image is higher than resolution of any frame of image in the multiple frames of images.

[0044] The enhancement module is configured to perform image enhancement processing on the fusion image to obtain a target image and output the target image.

[0045] Optionally, the image fusion model comprises multiple super-resolution models; wherein the super-resolution model corresponds to an image type one by one.

[0046] The fusion module is specifically configured to perform image segmentation on each single frame of image in the multiple frames of images after ghosting is removed to obtain multiple region images; input multiple region images belonging to the same image type in the multiple region images into a super-resolution model corresponding to the same image type to obtain a region fusion image of the same image type; wherein noise of the region fusion image is lower than noise of any region image of the same image type, and resolution of the region fusion image is higher than resolution of any region image of the same image type; and fuse region fusion images of different image types to obtain the fusion image.

[0047] Optionally, the fusion module is specifically configured to determine a type of each pixel of each single frame of image in the multiple frames of images after ghosting is removed based on the type of each pixel to determine an image type to which the pixel belongs; and divide pixels belonging to the same image type in the same single frame of image into the same region image to obtain multiple region images.

[0048] Optionally, the fusion module is specifically configured to fuse region fusion images of different image types based on weights of pixels in region images corresponding to the region fusion images to obtain the fusion image.

[0049] Optionally, the super-resolution model comprises multiple modules, and each module comprises a convolution layer, a normalization layer and an activation layer.

[0050] The fusion module is specifically configured to input a plurality of regional images belonging to a same image type in the plurality of regional images into a convolution layer in a super-resolution model corresponding to the same image type, to obtain a convolution result; input the convolution result into a normalization layer, to obtain a normalization result; and based on the normalization result and the activation layer, obtain a regional fusion image of the same image type; wherein an input of the activation layer is the normalization result.

[0051] Optionally, the fusion module is specifically configured to convert M rows and L columns of convolution kernels of the convolution layer in the super-resolution model corresponding to the same image type into an M row and P x L column two-dimensional convolution matrix; wherein M, P and L are positive integers; based on a size of the convolution kernel, a preset convolution sliding step, a preset padding size and the plurality of regional images, obtain a plurality of two-dimensional image matrices to be convolved; wherein each column in the two-dimensional image matrix to be convolved has P x L data, one column of data is data to be intercepted when the convolution kernel performs a convolution operation on one regional image, and a column number of the two-dimensional image matrix to be convolved is a maximum convolution number supported by one regional image; multiply the two-dimensional convolution matrix and each two-dimensional image matrix to be convolved to obtain the convolution result.

[0052] Optionally, the image type includes at least one of the following:

[0053] an image of a texture feature;

[0054] an image of a color feature;

[0055] an image of a shape feature;

[0056] an image of a spatial relationship.

[0057] Optionally, the removal module is specifically configured to align the plurality of frames of images; and perform ghosting removal processing on the aligned plurality of frames of images to obtain the plurality of frames of images after ghosting removal.

[0058] Optionally, the enhancement module is specifically configured to perform sharpening processing on the fusion image to obtain the target image and output the target image.

[0059] According to a third aspect of the embodiments of the present disclosure, a terminal is provided, including:

[0060] a processor;

[0061] a memory for storing processor-executable instructions;

[0062] The processor is configured to perform the photographing method as described in the first aspect above.

[0063] According to a fourth aspect of the present disclosure, a storage medium is provided, comprising:

[0064] When the instructions in the storage medium are executed by the terminal's processor, the terminal is able to perform the photographing method as described in the first aspect above.

[0065] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0066] In the embodiments of this disclosure, a multi-frame super-resolution algorithm is used. The multi-frame images after removing ghosting are input into a pre-trained image fusion model, which can simultaneously obtain a fused image with reduced noise and improved resolution. The fused image is then further processed to obtain a target image with improved image quality and output it. This is done without the need to obtain a super-resolution image through a separate noise reduction module or super-resolution module. It has the characteristics of high integration and good processing effect.

[0067] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0068] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0069] Figure 1 This is an example image of an optical zoom lens.

[0070] Figure 2 This is a flowchart illustrating a photographing method according to an embodiment of the present disclosure.

[0071] Figure 3 This is an example image of a ghost image disclosed herein.

[0072] Figure 4 This is a text example image disclosed herein.

[0073] Figure 5 This is an example diagram of a convolution disclosed herein.

[0074] Figure 6 This is a flowchart illustrating a super-resolution algorithm applied to photography.

[0075] Figure 7 This is a diagram illustrating a photographing device according to an exemplary embodiment.

[0076] Figure 8 This is a block diagram of a terminal shown in an embodiment of this disclosure. Detailed Implementation

[0077] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0078] Figure 2 This is a flowchart illustrating a photographing method according to an embodiment of the present disclosure, such as... Figure 2 As shown, the photo-taking method applied to the terminal includes the following steps:

[0079] S11. Upon detecting a photo capture command, continuously capture multiple frames of images;

[0080] S12. Based on the multi-frame images, obtain multi-frame images after removing ghosting;

[0081] S13. Input the multi-frame images after removing ghosting into the image fusion model to obtain a fused image; wherein the noise of the fused image is lower than the noise of any frame in the multi-frame images, and the resolution of the fused image is higher than the resolution of any frame in the multi-frame images;

[0082] S14. Perform image enhancement processing on the fused image to obtain the target image and output it.

[0083] In embodiments of this disclosure, the terminal includes a smartphone, camera, tablet computer, or wearable electronic device, etc. The terminal is equipped with an image acquisition component, which enables the mobile terminal to take photos. Taking a smartphone as an example, the image acquisition component includes a front-facing camera and a rear-facing camera.

[0084] In step S11, when the terminal detects a photo-taking command, it continuously acquires multiple frames of images. These multiple frames can be low dynamic range images with different exposure levels captured from the same shooting scene, such as three frames: short exposure, normal exposure, and long exposure.

[0085] In step S12, the terminal obtains multiple frames of images after removing ghosting based on the detected multiple frames. Because handheld devices inevitably experience shaking or movement of objects during shooting, image shifts or distortions can occur, resulting in residual images, or ghosting, in the acquired images. Figure 3 This is an example image of a ghosting phenomenon disclosed herein, such as... Figure 3 As shown, the ghost image on the glass on the left side of the image may be due to hand tremors during the shooting process.

[0086] To address this, this disclosure removes ghosting from multi-frame images. In one embodiment of this disclosure, to improve the ghosting removal effect, the multi-frame images can be aligned beforehand before ghosting removal. The difficulty of multi-frame alignment lies in the fact that its accuracy needs to be at the sub-pixel level, and it needs to support extremely low light conditions.

[0087] In one embodiment, step S12 includes:

[0088] Align the multi-frame images;

[0089] The aligned multi-frame images are then subjected to ghosting removal processing to obtain the ghost-removed multi-frame images.

[0090] In this embodiment, when aligning multiple frames of images, the median threshold bitmap (MTB) method can be used. Alternatively, any frame from the multiple frames can be used as a reference image, and the remaining non-reference images can be aligned to the reference image. For example, feature points can be extracted from the reference image and the non-reference image respectively, and the non-reference images and the reference image can be registered based on the feature points to achieve alignment.

[0091] In step S13, after the terminal obtains the multi-frame images after removing ghosting, it inputs the multi-frame images after removing ghosting into the image fusion model. In particular, the image fusion model has both noise reduction and super-resolution functions, so that the fused image has lower noise and higher resolution than any frame of the multi-frame images after removing ghosting.

[0092] In one embodiment of this disclosure, the image fusion model can be a model trained using a support vector machine. In another embodiment of this disclosure, the image fusion model is a model trained using a deep learning network.

[0093] It should be noted that, in the embodiments of this disclosure, this disclosure is not limited to using support vector machines or deep learning networks to train and obtain image fusion models. Furthermore, when training the image fusion model, the input images in the training set are noisy, low-resolution images, while the label images (GT images) are low-noise, high-resolution images. Using (x, y) to represent a pair of training data, the image fusion model fits the correspondence between x and y, y = f(x), through a support vector machine network or deep learning network. The function f(x) simultaneously possesses noise reduction and super-resolution capabilities, where x represents multiple frames of input images, and y represents the output image.

[0094] In step 14, after obtaining the fused image with reduced noise and improved resolution, the terminal will perform image enhancement processing on the fused image to obtain a target image with further improved image quality and output it.

[0095] In the embodiments of this disclosure, the output target image is the image that is finally presented to the user based on the photo-taking command.

[0096] In one embodiment, step S14 includes:

[0097] The fused image is sharpened to obtain the target image, which is then output.

[0098] In this embodiment, the image enhancement processing includes sharpening processing. Sharpening processing can compensate for the contours of the image, enhance the edges and grayscale transitions of the image, making the image clearer. As a result, the target image obtained has better image quality and can improve the user experience.

[0099] However, in the embodiments of this disclosure, image enhancement processing is not limited to sharpening processing, but can also be a combination of sharpening processing and other enhancement processing methods, or at least one of other image enhancement methods: contrast enhancement, color correction, etc.

[0100] In super-resolution algorithms, noise, detail, and artifacts are the main factors considered in evaluating the quality of super-resolution images. The goal of super-resolution algorithms is to improve image quality from these three aspects. In this disclosure, a multi-frame super-resolution algorithm is used. Multiple frames of images after ghosting removal are input into a pre-trained image fusion model, which simultaneously obtains a fused image with reduced noise and improved resolution. Further image enhancement processing is performed on the fused image to obtain the target image with improved image quality, and then output. This eliminates the need for a separate noise reduction module or super-resolution module to obtain the super-resolution image, exhibiting high integration and good processing performance.

[0101] In one embodiment, the image fusion model includes multiple super-resolution models; wherein, the super-resolution models correspond one-to-one with image types; step S13 includes:

[0102] Each single frame in the multi-frame image after ghosting removal is segmented to obtain multiple region images;

[0103] Multiple region images belonging to the same image type are input into a super-resolution model corresponding to the same image type to obtain a region fusion image of the same image type; wherein, the noise of the region fusion image is lower than the noise of any region image among the multiple region images of the same image type, and the resolution of the region fusion image is higher than the resolution of any region image among the multiple region images of the same image type;

[0104] The fused image is obtained by fusing regions of different image types.

[0105] In the embodiments of this disclosure, since different types of images give people different subjective feelings, this disclosure trains super-resolution models for different image types. The image fusion model includes super-resolution models corresponding to different image types, and the super-resolution model can be a model trained based on a deep learning network.

[0106] In one embodiment, the image type includes at least one of the following:

[0107] Images with texture features;

[0108] Images with color characteristics;

[0109] Images of shape features;

[0110] Images of spatial relationships.

[0111] For example, for areas with more random textures such as road surfaces, tree patterns, and grass, the overall structure may be relatively weak. If the model can retain more noise, it can give people a more detailed "subjective illusion." However, for images such as text, buildings, and faces, which have specific spatial relationships, people may focus on higher-level semantic cognition and pay less attention to textures, that is, they are more concerned with the overall structure. Therefore, the model needs to have stronger noise reduction and better structural details. Figure 4 This is a sample diagram of text disclosed herein, from which one may focus on the structural features of each character.

[0112] For example, the sky is blue, trees are green or yellow, rivers are nearly transparent, and human skin is yellow or black. Based on specific objects, there are predominantly certain colors, so the model might need to focus more on color features. Or, for another example, the sun is round, and the moon is crescent-shaped or round. Based on specific objects, there are predominantly certain shapes, so the model might need to focus more on shape features.

[0113] Therefore, this disclosure provides super-resolution models corresponding to different image types. It should be noted that when training the image fusion model, this disclosure can select a large number of images with high noise and low resolution, but also primarily based on texture characteristics, as input, while using the corresponding images with low noise and high resolution as label images to train the super-resolution model. Similarly, super-resolution models primarily based on structural characteristics, color features, or spatial relationships can also be trained, which will not be detailed in this disclosure.

[0114] However, images typically captured are not images of a single object; for example, the captured image may contain buildings, trees, or sky. Different objects have different characteristics. As mentioned earlier, the focus may be more on the overall structure of buildings, on the texture of trees, and on the color of the sky. Therefore, this disclosure allows for pre-segmentation of individual frames within a multi-frame image after ghosting removal, resulting in individual frames containing images of different types of regions. For example, image segmentation can divide the aforementioned image into building regions, tree regions, and sky regions.

[0115] It should be noted that because multiple frames are captured continuously, the content of the images is highly similar. Therefore, image segmentation of different ghost-removed single-frame images usually yields image regions of the same type. Of course, some ghost-removed single-frame images may be missing image regions of a certain type after image segmentation. In this disclosure, when such missing regions exist, image regions of that type in the single-frame image are ignored.

[0116] In one embodiment of this disclosure, when performing image segmentation, the image segmentation result can be obtained based on a pre-trained neural network model. The image segmentation result includes the segmented region image and the image type corresponding to the region image.

[0117] In another embodiment, the step of performing image segmentation on each single frame of the multi-frame images after ghosting removal to obtain multiple region images includes:

[0118] For each single frame in the multi-frame image after ghosting removal, determine the type of each pixel in the single frame image;

[0119] The image type to which each pixel belongs is determined based on the type of each pixel;

[0120] Pixels of the same image type belonging to the same single frame image are divided into the same region image to obtain multiple region images.

[0121] In this embodiment, the type of each pixel in a single-frame image after ghosting removal can be statistically analyzed. For example, using a region growing method, starting from a single pixel, pixels belonging to the same image type in the same single-frame image are gradually merged into the same region, thereby obtaining multiple segmented region images. After obtaining region images of different image types included in the single-frame image after ghosting removal, multi-frame region images belonging to the same salience type from different multi-frame images can be input into the super-resolution model corresponding to the same image type to obtain a region fusion image of the same image type. For example, if the current multi-frame region images of the same image type are texture types, then the multi-frame region images can be input into the super-resolution model corresponding to the texture type to obtain a region fusion image that retains more texture details while reducing other noise and improving resolution.

[0122] After obtaining fused images of different regions, the fused images can be stitched together into a complete fused image based on the regional location information of each type of image during image segmentation. Since the noise of each fused image is reduced and its resolution is improved, the resulting fused image is also an image with reduced noise and improved resolution.

[0123] In one embodiment, fusing regions of different image types to obtain the fused image includes:

[0124] Based on the pixel weights in the corresponding regional images of different types of region fusion images, the region fusion images of different image types are fused to obtain the fused image.

[0125] In the embodiments of this disclosure, when stitching region fusion images into a complete fusion image based on the regional location information of each type of image, the region fusion images can be merged into a fusion image based on the weights of different types of region images. For example, if a texture feature region image is located in the central region of the image, the weight of each pixel in the texture feature region image can be set to be larger to highlight the central region in the fusion image.

[0126] It should be noted that, in the embodiments of this disclosure, the input image in the image fusion model may also be a local region image from an image. This local region image is also obtained by performing image segmentation on the training image, which will not be described in detail in this disclosure.

[0127] If deep learning networks are used to train the model, due to the special nature of convolutional neural networks, related techniques typically operate on the entire image without considering the characteristics of different regions within the image. Consequently, the image quality of the resulting super-resolution image cannot be significantly improved. In contrast, this disclosure considers the complexity of the shooting scene, trains super-resolution models for different image types, then performs image segmentation on each single frame image after ghosting removal, and then inputs the super-resolution model corresponding to the region image type to obtain the region fusion result, which can greatly improve the image quality of the super-resolution image. Furthermore, this disclosure inputs the super-resolution model corresponding to the image region, without operating the model on the entire image, which can greatly reduce the amount of computation.

[0128] In one embodiment, the super-resolution model includes: multiple modules, each module including: a convolutional layer, a normalization layer, and an activation layer;

[0129] The step of inputting multiple region images belonging to the same image type from the multiple region images into the super-resolution model corresponding to the same image type to obtain a region fusion image of the same image type includes:

[0130] Multiple region images belonging to the same image type from the multiple region images are input into the convolutional layer of the super-resolution model corresponding to the same image type to obtain the convolution result;

[0131] The convolution result is input into the normalization layer to obtain the normalization result;

[0132] Based on the normalization result and the activation layer, a region fusion image of the same image type is obtained; wherein, the input of the activation layer is the normalization result.

[0133] As previously mentioned, the super-resolution model can be a model trained using a deep learning network. In the embodiments of this disclosure, the super-resolution model includes multiple modules, for example, up to 16 modules, each module including a convolutional layer, a normalization layer, and an activation layer. When using the super-resolution model, multiple region images of the same image type can be input into the convolutional layers of the corresponding modules of the super-resolution model to obtain convolution results. Then, the convolution results are passed through a normalization layer to obtain a normalized result with values ​​within a predetermined range for subsequent calculations. Further, when obtaining a region fusion image of the same image type based on the normalization result and the activation layer, the normalization result can be input into the activation layer to improve the model's expressive power. Then, the outputs of the activation layers in each module are input into the pooling layer, and the output of the pooling layer is used as the input to the fully connected layer to obtain a region fusion image of the same image type.

[0134] As mentioned earlier, this disclosure uses image regions as input for corresponding super-resolution models, without requiring the models to operate on the entire image, which greatly reduces computational cost. Specifically, each super-resolution model only performs convolution operations on the regions corresponding to its type, while other regions are not convolutionally processed. This computational cost is independent of the number of models, thus significantly saving computational resources.

[0135] In one embodiment, the step of inputting multiple region images belonging to the same image type from the multiple region images into the convolutional layer of the super-resolution model corresponding to the same image type to obtain the convolution result includes:

[0136] The M convolutional kernels of the convolutional layer in the super-resolution model corresponding to the same image type are transformed into a two-dimensional convolutional matrix with M rows and P×L columns; wherein M, P and L are all positive integers.

[0137] Based on the size of the convolution kernel, the preset convolution stride, and the multiple region images, multiple two-dimensional image matrices to be convolved are obtained; wherein, each column of the two-dimensional image matrix to be convolved contains P*L data, and one column of data is the data to be truncated when the convolution kernel performs a convolution operation on a region image; the number of columns of the two-dimensional image matrix to be convolved is: the maximum number of convolutions supported by a region image.

[0138] The convolution result is obtained by multiplying the two-dimensional convolution matrix with each of the two-dimensional image matrices to be convolved.

[0139] In embodiments of this disclosure, the super-resolution model may include M P×L convolutional kernels, which are transformed into a two-dimensional convolutional matrix with M rows and P×L columns. That is, one two-dimensional convolutional kernel is transformed into a one-dimensional convolutional kernel, and multiple transformed one-dimensional convolutional kernels form a two-dimensional convolutional matrix. Each row of the transformed two-dimensional convolutional matrix is ​​the weight value in a two-dimensional convolutional kernel.

[0140] For example, M is the number of convolution kernels, the size of the convolution kernel is P×L and is k×k. Each row in the transformed two-dimensional convolution matrix is ​​a convolution kernel vector, and each row has k×k weight values. There are a total of M rows, representing M convolution kernels.

[0141] Furthermore, this disclosure obtains a two-dimensional image matrix to be convolved based on the size of the convolution kernel, a preset convolution stride, and a single-frame region image.

[0142] It should be noted that, in the embodiments of this disclosure, in order to ensure that the size of the matrix after convolution is consistent with the size of the region image of the input convolutional layer, when the size of the convolutional kernel, the convolutional stride and the size of the region image to be convolved are insufficient to meet the above requirements, the area around a region image can be filled to obtain a filled region image, and the filled region image can be processed.

[0143] For example, the convolution kernel is a k×k two-dimensional matrix; the preset convolution stride is stride_h in the row direction and stride_w in the column direction; the preset padding size is pad_h in the row direction and pad_w in the column direction; the size of the single-frame region image is image_h×image_w; then, the obtained two-dimensional image matrix to be convolved has k×k rows (i.e., P×L rows) and N columns, where N can be represented by the following formula (1):

[0144] N=((image_h+2×pad_h–k) / stride_h+1)×((image_w+2×pad_w–k) / stride_w+1) (1)

[0145] Each column of the two-dimensional image matrix to be convolved contains k×k data points. One column represents the data extracted by the convolution kernel during a single convolution operation on the padded single-frame region image. The pixel values ​​of each pixel in the padded single-frame region image are covered by N columns of data, where N is the maximum number of convolutions supported by the padded single-frame region image. Padded single-frame region images can be achieved by filling the perimeter of the single-frame region image with a ring of zero-valued data satisfying a preset fill size.

[0146] If the region image does not require padding, the value of the number of columns N of the two-dimensional image matrix to be convolved is the value corresponding to pad_h and pad_w being both 0, which is also the maximum number of convolutions supported by the region image.

[0147] Figure 5 This is an example image of a convolution disclosed herein, such as... Figure 5 As shown, if the size of the P×L two-dimensional convolution kernel is denoted as K, then the above convolution process is the product of the M×K two-dimensional convolution matrix and the K×N two-dimensional image matrix to be convolved, resulting in an M×N convolution result matrix, where each row of the matrix is ​​a feature map.

[0148] It is understood that in the embodiments of this disclosure, considering that the segmented region images may not be regular rectangular regions during image segmentation, and that the region images may have different shapes and sizes, this disclosure specifically modifies the original regular rectangular convolution operation into an irregular sliding operation, without having to fill each irregular region image to meet the convolution requirements. Furthermore, the final computational complexity is not related to the number of region images, but only to the total area of ​​a single frame image, thus reducing the algorithm complexity and greatly improving the convolution efficiency.

[0149] Figure 6 This is a flowchart illustrating a super-resolution algorithm applied to photography, as disclosed herein. Figure 6 As shown, the super-resolution algorithm includes the following steps:

[0150] S21, Multi-frame alignment.

[0151] In this embodiment, multi-frame alignment means aligning multiple frames of images acquired continuously based on the shooting command.

[0152] S22, Remove Ghost Shadows.

[0153] In this embodiment, the aligned multi-frame images are subjected to a ghosting removal operation to obtain multi-frame images after ghosting removal.

[0154] S23, Deep Learning Multi-Frame Fusion.

[0155] In this embodiment, the multi-frame images after ghosting removal are input into a pre-trained image fusion model using a deep learning network. This image fusion model has the functions of both noise reduction and resolution improvement. Therefore, after the multi-frame images after ghosting removal are input into the image fusion model, a fused image with reduced noise and improved resolution can be obtained.

[0156] S24, Sharpen.

[0157] In this embodiment, after obtaining the fused image with reduced noise and improved resolution through step S23, the fused image can be sharpened to obtain a target image with improved image quality and display it to the user.

[0158] Immediately, this disclosure employs a multi-frame super-resolution algorithm, which inputs the aligned and ghost-removed multi-frame images into a pre-trained image fusion model based on a deep learning network. This allows for the simultaneous acquisition of a fused image with reduced noise and improved resolution. Furthermore, the fused image is sharpened to obtain a target image with enhanced image quality, which is then output without the need for a separate noise reduction module or super-resolution module to obtain the super-resolution image. This approach features high integration and excellent processing performance.

[0159] Figure 7 This is a diagram illustrating a photographing device according to an exemplary embodiment. (Refer to...) Figure 7The camera device includes:

[0160] The acquisition module 101 is configured to continuously acquire multiple frames of images when a photo capture command is detected;

[0161] The ghost removal module 102 is configured to obtain a multi-frame image after ghost removal based on the multi-frame image;

[0162] The fusion module 103 is configured to input the ghost-removed multi-frame images into an image fusion model to obtain a fused image; wherein the noise of the fused image is lower than the noise of any frame in the multi-frame images, and the resolution of the fused image is higher than the resolution of any frame in the multi-frame images.

[0163] The enhancement module 104 is configured to perform image enhancement processing on the fused image to obtain and output the target image.

[0164] Optionally, the image fusion model includes multiple super-resolution models; wherein, each super-resolution model corresponds one-to-one with an image type;

[0165] The fusion module 103 is specifically configured to perform image segmentation on each single frame image in the multi-frame images after ghosting removal to obtain multiple region images; input multiple region images belonging to the same image type into the super-resolution model corresponding to the same image type to obtain a region fusion image of the same image type; wherein the noise of the region fusion image is lower than the noise of any region image among the multiple region images of the same image type, and the resolution of the region fusion image is higher than the resolution of any region image among the multiple region images of the same image type; and fuse the region fusion images of different image types to obtain the fused image.

[0166] Optionally, the fusion module 103 is specifically configured to, for each single frame image in the multi-frame images after ghosting removal, determine the type of each pixel in the single frame image; determine the image type to which the pixel belongs based on the type of each pixel; and divide pixels of the same image type in the same single frame image into the same region image to obtain multiple region images.

[0167] Optionally, the fusion module 103 is specifically configured to fuse the region fusion images of different image types based on the pixel weights in the region images corresponding to the region fusion images of different types, to obtain the fused image.

[0168] Optionally, the super-resolution model includes: multiple modules, each module including: a convolutional layer, a normalization layer and an activation layer;

[0169] The fusion module 103 is specifically configured to input multiple region images belonging to the same image type from the multiple region images into the convolutional layer of the super-resolution model corresponding to the same image type to obtain a convolution result; input the convolution result into a normalization layer to obtain a normalization result; and obtain a region fusion image of the same image type based on the normalization result and the activation layer; wherein the input of the activation layer is the normalization result.

[0170] Optionally, the fusion module 103 is specifically configured to convert the M P-row, L-column convolution kernels of the convolutional layer in the super-resolution model corresponding to the same image type into a two-dimensional convolution matrix with M rows and P×L columns; wherein M, P, and L are all positive integers; based on the size of the convolution kernel, a preset convolution stride, and the multiple region images, multiple two-dimensional image matrices to be convolved are obtained; wherein each column of the two-dimensional image matrix to be convolved contains P×L data, and one column of data is the data to be truncated when the convolution kernel performs a convolution operation on a region image, and the number of columns of the two-dimensional image matrix to be convolved is: the maximum number of convolutions supported by a region image; the two-dimensional convolution matrix and the two-dimensional image matrix to be convolved are multiplied to obtain the convolution result.

[0171] Optionally, the image type includes at least one of the following:

[0172] Images with texture features;

[0173] Images with color characteristics;

[0174] Images of shape features;

[0175] Images of spatial relationships.

[0176] Optionally, the removal module 102 is specifically configured to align the multi-frame images; and perform ghosting removal processing on the aligned multi-frame images to obtain the ghosting-removed multi-frame images.

[0177] Optionally, the enhancement module 104 is specifically configured to sharpen the fused image to obtain the target image and output it.

[0178] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0179] Figure 8 This is a block diagram illustrating a mobile terminal device 800 according to an exemplary embodiment. For example, device 800 may be a mobile phone, a mobile computer, etc.

[0180] Reference Figure 8The device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0181] Processing component 802 typically controls the overall operation of device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0182] Memory 804 is configured to store various types of data to support the operation of device 800. Examples of this data include instructions for any application or method operating on device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0183] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 800.

[0184] Multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0185] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0186] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0187] Sensor assembly 814 includes one or more sensors for providing status assessments of various aspects of device 800. For example, sensor assembly 814 may detect the on / off state of device 800, the relative positioning of components such as the display and keypad of device 800, changes in the position of device 800 or a component of device 800, the presence or absence of user contact with device 800, the orientation or acceleration / deceleration of device 800, and temperature changes of device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0188] Communication component 816 is configured to facilitate wired or wireless communication between device 800 and other devices. Device 800 can access wireless networks based on communication standards, such as Wi-Fi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0189] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0190] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of the device 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0191] A non-transitory computer-readable storage medium, wherein when instructions in the storage medium are executed by a terminal's processor, the terminal is enabled to perform a photographing method, the method comprising:

[0192] When a photo capture command is detected, multiple frames of images are captured continuously;

[0193] Based on the aforementioned multi-frame images, obtain multi-frame images after removing ghosting;

[0194] The multi-frame images after ghosting removal are input into an image fusion model to obtain a fused image; wherein the noise of the fused image is lower than the noise of any single frame in the multi-frame images, and the resolution of the fused image is higher than the resolution of any single frame in the multi-frame images;

[0195] The fused image is then enhanced to obtain the target image, which is then output.

[0196] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0197] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for taking photos, characterized in that, The method includes: When a photo capture command is detected, multiple frames of images are captured continuously; Based on the aforementioned multi-frame images, a multi-frame image after removing ghosting is obtained; wherein, ghosting refers to residual images on the image; The multi-frame images after ghosting removal are input into an image fusion model to obtain a fused image; wherein the noise of the fused image is lower than the noise of any single frame in the multi-frame images, and the resolution of the fused image is higher than the resolution of any single frame in the multi-frame images; The fused image is then subjected to image enhancement processing to obtain the target image and output it. The image fusion model includes multiple super-resolution models; wherein, each super-resolution model corresponds one-to-one with an image type. The step of inputting the ghost-removed multi-frame images into the image fusion model to obtain the fused image includes: Each single frame in the multi-frame image after ghosting removal is segmented to obtain multiple region images; Multiple region images belonging to the same image type are input into a super-resolution model corresponding to the same image type to obtain a region fusion image of the same image type; wherein, the noise of the region fusion image is lower than the noise of any region image among the multiple region images of the same image type, and the resolution of the region fusion image is higher than the resolution of any region image among the multiple region images of the same image type; The fused image is obtained by fusing regions of different image types.

2. The method according to claim 1, characterized in that, The process of segmenting each single frame in the multi-frame image after ghost removal to obtain multiple region images includes: For each single frame in the multi-frame image after ghosting removal, determine the type of each pixel in the single frame image; The image type to which each pixel belongs is determined based on the type of each pixel; Pixels of the same image type belonging to the same single frame image are divided into the same region image to obtain multiple region images.

3. The method according to claim 1, characterized in that, The process of fusing regions of different image types to obtain the fused image includes: Based on the pixel weights in the corresponding regional images of different types of region fusion images, the region fusion images of different image types are fused to obtain the fused image.

4. The method according to claim 1, characterized in that, The super-resolution model includes multiple modules, each module including a convolutional layer, a normalization layer, and an activation layer; The step of inputting multiple region images belonging to the same image type from the multiple region images into the super-resolution model corresponding to the same image type to obtain a region fusion image of the same image type includes: Multiple region images belonging to the same image type from the multiple region images are input into the convolutional layer of the super-resolution model corresponding to the same image type to obtain the convolution result; The convolution result is input into the normalization layer to obtain the normalization result; Based on the normalization result and the activation layer, a region fusion image of the same image type is obtained; wherein, the input of the activation layer is the normalization result.

5. The method according to claim 1, characterized in that, The step of inputting multiple region images belonging to the same image type from the multiple region images into the convolutional layer of the super-resolution model corresponding to the same image type to obtain the convolution result includes: The M convolutional kernels of the convolutional layer in the super-resolution model corresponding to the same image type are transformed into a two-dimensional convolutional matrix with M rows and P×L columns; wherein M, P and L are all positive integers. Based on the size of the convolution kernel, the preset convolution stride, and the multiple region images, multiple two-dimensional image matrices to be convolved are obtained; wherein, each column of the two-dimensional image matrix to be convolved contains P×L data, and one column of data is the data to be truncated when the convolution kernel performs a convolution operation on a region image; the number of columns of the two-dimensional image matrix to be convolved is: the maximum number of convolutions supported by a region image. The convolution result is obtained by multiplying the two-dimensional convolution matrix with each of the two-dimensional image matrices to be convolved.

6. The method according to claim 1, characterized in that, The image type includes at least one of the following: Images with texture features; Images with color characteristics; Images of shape features; Images of spatial relationships.

7. The method according to any one of claims 1 to 6, characterized in that, The process of obtaining the ghost-free multi-frame images based on the multi-frame images includes: Align the multi-frame images; The aligned multi-frame images are then subjected to ghosting removal processing to obtain the ghost-removed multi-frame images.

8. The method according to claim 1, characterized in that, The step of performing image enhancement processing on the fused image to obtain and output the target image includes: The fused image is sharpened to obtain the target image, which is then output.

9. A photographing device, characterized in that, The device includes: The acquisition module is configured to continuously acquire multiple frames of images when a photo capture command is detected; The ghosting module is configured to obtain multiple frames of images after ghosting removal based on the multiple frames of images; wherein, ghosting refers to residual images on the images; The fusion module is configured to input the ghost-removed multi-frame images into an image fusion model to obtain a fused image; wherein the noise of the fused image is lower than the noise of any single frame in the multi-frame images, and the resolution of the fused image is higher than the resolution of any single frame in the multi-frame images; the image fusion model includes multiple super-resolution models; wherein each super-resolution model corresponds one-to-one with an image type; the image fusion model performs the following processing: image segmentation is performed on each single frame in the ghost-removed multi-frame images to obtain multiple region images; multiple region images belonging to the same image type are input into the super-resolution model corresponding to the same image type to obtain a region fusion image of the same image type; the noise of the region fusion image is lower than the noise of any single region image in the multiple region images of the same image type, and the resolution of the region fusion image is higher than the resolution of any single region image in the multiple region images of the same image type; the region fusion images of different image types are fused to obtain the fused image; The enhancement module is configured to perform image enhancement processing on the fused image to obtain and output the target image.

10. A terminal, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to perform the photographing method as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the terminal, the terminal is able to perform the photographing method as described in any one of claims 1 to 8.

Images