Image processing method and device, computer readable storage medium, and terminal

Abstract

An image processing method and device, a computer readable storage medium and a terminal, the method comprising: acquiring F frames of first images, F being an integer greater than 1; inputting the F frames of first images into a neural network model trained in advance to obtain fusion parameters output by the neural network model, the fusion parameters comprising enhancement parameters and weight parameters, wherein the enhancement parameters comprise a mapping vector corresponding to each pixel point of each frame of first image, and the weight parameters comprise a weight value corresponding to each pixel point of each frame of first image; performing enhancement processing on the F frames of first images according to at least the enhancement parameters to obtain F frames of second images; and performing fusion processing on the F frames of second images according to the weight parameters to obtain a target image. Through the scheme provided in the present application, multiple frames of images can be fused efficiently to obtain a high-quality image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method and apparatus, a computer-readable storage medium, and a terminal. Background Technology

[0002] Using multiple frames for dynamic range fusion is currently the mainstream approach in the industry for acquiring High Dynamic Range (HDR) images. However, it is difficult to balance image quality and processing efficiency when fusing multiple frames. Existing solutions often employ complex algorithms with high computational cost and low processing efficiency in order to improve the quality of the fused image. Alternatively, simpler fusion methods may be used to improve processing efficiency, but these usually fail to achieve good image quality. Summary of the Invention

[0003] One of the technical objectives of this application is to provide an image processing method that can efficiently fuse multiple frames of images and obtain high-quality images.

[0004] To address the aforementioned technical problems, this application provides an image processing method, comprising: acquiring a first image of F frames, where F is an integer greater than 1; inputting the first image of F frames into a pre-trained neural network model to obtain fusion parameters, the fusion parameters including enhancement parameters and weight parameters, wherein the enhancement parameters include mapping vectors corresponding to each pixel of each frame of the first image, and the weight parameters include weight values ​​corresponding to each pixel of each frame of the first image; enhancing the first image of F frames at least according to the enhancement parameters to obtain a second image of F frames; and fusing the second image of F frames according to the weight parameters to obtain a target image.

[0005] Optionally, the processing steps of the neural network model for the first image of frame F include: downsampling the first image of frame F to obtain downsampled data; performing a first convolution on the downsampled data to obtain intermediate enhanced data; and performing an upsampling operation on the intermediate enhanced data to obtain the enhancement parameters.

[0006] Optionally, the fusion parameters further include: guiding parameters, the guiding parameters including a scene vector corresponding to each pixel, the scene vector including the association weight between the pixel and each scene, and enhancing the first image of the F-frame at least according to the enhancement parameters including: enhancing the first image of the F-frame according to the guiding parameters and the enhancement parameters to obtain the second image of the F-frame.

[0007] Optionally, the processing steps of the neural network model for the first image of frame F include: performing a second convolution process on the first image of frame F to obtain the guiding parameters.

[0008] Optionally, the processing steps of the neural network model for the first image of frame F include: performing a third convolution process on the first image of frame F to obtain the weight parameters.

[0009] Optionally, the step of obtaining the neural network model includes: obtaining training data, the training data including: F-frame sample input images and single-frame sample target images; and training the neural network model using the training data until the neural network model converges.

[0010] Optionally, obtaining the first image of F-frame includes: obtaining the original image of F-frame; if the original image of F-frame is captured by an HDR camera, then the original image of F-frame is used as the first image of F-frame; otherwise, the first image of F-frame is aligned to obtain the first image of F-frame.

[0011] To address the aforementioned technical problems, this application also provides an image processing apparatus, comprising: an acquisition module for acquiring an F-frame first image, where F is an integer greater than 1; a parameter calculation module for inputting the F-frame first image into a pre-trained neural network model to obtain fusion parameters, the fusion parameters including enhancement parameters and weight parameters, wherein the enhancement parameters include mapping vectors corresponding to each pixel of each frame of the first image, and the weight parameters include weight values ​​corresponding to each pixel of each frame of the first image; an enhancement module for enhancing the F-frame first image at least according to the enhancement parameters to obtain an F-frame second image; and a fusion module for fusing the F-frame second image according to the weight parameters to obtain a target image.

[0012] This application also provides a computer-readable storage medium storing a computer program thereon, which, when run by a processor, performs the steps of the above-described image processing method.

[0013] This application embodiment also provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the steps of the above-described image processing method when running the computer program.

[0014] Compared with the prior art, the technical solution of this application embodiment has the following beneficial effects:

[0015] In this embodiment of the application, an F-frame first image is acquired and input into a pre-trained neural network model to calculate fusion parameters, which include enhancement parameters and weight parameters. The enhancement parameters include the mapping vector corresponding to each pixel in each frame of the first image, and the weight parameters include the weight value corresponding to each pixel in each frame of the first image. Further, the F-frame first image is enhanced at least according to the enhancement parameters to obtain an F-frame second image, and then fused according to the weight parameters to obtain the target image.

[0016] Compared to existing technologies, the solution in this application provides an end-to-end image processing method. Specifically, a neural network model is used to calculate fusion parameters for image fusion, thus efficiently obtaining the fusion parameters and improving the overall efficiency of the fusion algorithm. Furthermore, since the enhancement parameters include the mapping vectors corresponding to each pixel in each frame of the first image, each pixel in each frame of the first image can be enhanced according to the enhancement parameters to enhance the color and / or brightness of each frame, thereby improving the quality of the final image. Additionally, since the weight parameters include the weight values ​​corresponding to each pixel in each frame of the first image, fusion processing of the F-frame second image according to the weight parameters can be performed based on the weights of each pixel in each frame, effectively suppressing ghosting. Therefore, the solution provided in this application can efficiently obtain high dynamic range images of good quality.

[0017] Furthermore, in the scheme of this application embodiment, the first image of frame F is first downsampled, and then subjected to a first convolution process to obtain intermediate enhancement data. Then, the intermediate enhancement data is upsampled to obtain enhancement parameters. Adopting the above scheme helps to reduce the computational load of calculating enhancement parameters, improves the efficiency of calculating enhancement parameters, and thus improves the overall efficiency of the fusion algorithm.

[0018] Furthermore, in the scheme of this application embodiment, the fusion parameters also include guiding parameters, and the first image of frame F is enhanced according to the guiding parameters and enhancement parameters to obtain the second image of frame F. Since the guiding parameters include the scene vector corresponding to each pixel, and the scene vector includes the association weight between the pixel and each scene, the scheme of this application embodiment expands the scene channel and considers the degree of association between the pixel and the scene during image enhancement. Therefore, the overall algorithm has strong adaptability and can be applied to different shooting scenarios, achieving good fusion quality in different shooting scenarios. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating an image processing method according to an embodiment of this application;

[0020] Figure 2This is a schematic diagram of the structure of a neural network model in an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of the structure of an image processing device according to an embodiment of this application. Detailed Implementation

[0022] As described in the background section, existing multi-frame image fusion schemes have difficulty balancing the quality of the fused image with the fusion efficiency.

[0023] Existing multi-frame image fusion methods include ghost detection, ghost mask morphological processing, Fourier transform, spectral fusion, and inverse Fourier transform. Another existing fusion method divides the image into textured regions and flat regions, processes them separately, and then fuses the two regions separately. These methods require adjusting a large number of parameters, are computationally intensive, and time-consuming. Computationally intensive methods are generally impractical for low-end devices without acceleration hardware. For example, devices without acceleration hardware such as Neural-network Processing Units (NPUs), Graphics Processing Units (GPUs), or Digital Signal Processors (DSPs) may not be able to run computationally intensive multi-frame image fusion methods. Alternatively, in scenarios with high real-time requirements, computationally intensive multi-frame image fusion methods cannot meet the real-time requirements due to their high processing time. Furthermore, a few simple fusion methods often suffer from poor image quality after fusion.

[0024] To address the aforementioned technical problems, this application provides an image processing method. In this embodiment, an F-frame first image is acquired and input into a pre-trained neural network model to obtain fusion parameters, which include enhancement and weight parameters. The enhancement parameters include the mapping vector corresponding to each pixel in each frame of the first image, and the weight parameters include the weight value corresponding to each pixel in each frame of the first image. Further, the F-frame first image is enhanced based on at least the enhancement parameters to obtain an F-frame second image, and then fused based on the weight parameters to obtain the target image.

[0025] Compared to existing technologies, the solution in this application provides an end-to-end image processing method. Specifically, a neural network model is used to calculate fusion parameters for image fusion, thus efficiently obtaining the fusion parameters and improving the overall efficiency of the fusion algorithm. Furthermore, since the enhancement parameters include the mapping vectors corresponding to each pixel in each frame of the first image, each pixel in each frame of the first image can be enhanced according to the enhancement parameters to enhance the color and / or brightness of each frame, thereby improving the quality of the final image. Additionally, since the weight parameters include the weight values ​​corresponding to each pixel in each frame of the first image, fusion processing of the F-frame second image according to the weight parameters can be performed based on the weights of each pixel in each frame, effectively suppressing ghosting.

[0026] To make the above-mentioned objectives, features and beneficial effects of this application more apparent and understandable, the specific embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0027] Reference Figure 1 , Figure 1 This is a flowchart illustrating an image processing method according to an embodiment of this application. The method can be executed by a terminal, such as a mobile phone, computer, tablet, IoT device, wearable device, vehicle terminal, server, etc., but is not limited thereto. The terminal can be equipped with a camera to perform functions such as shooting and recording. Alternatively, the terminal can be an edge device, such as a camera with data processing capabilities, like a camera used for security monitoring. Alternatively, the terminal can be a device without a camera, acquiring multiple frames of images for processing from an external source; this embodiment does not impose any limitations on this. Figure 1 The image processing method shown may include the following steps:

[0028] Step S11: Obtain the first image of frame F, where F is an integer greater than 1;

[0029] Step S12: Input the first image of frame F into the pre-trained neural network model to obtain fusion parameters. The fusion parameters include enhancement parameters and weight parameters. The enhancement parameters include the mapping vector corresponding to each pixel of the first image of each frame, and the weight parameters include the weight value corresponding to each pixel of the first image of each frame.

[0030] Step S13: Enhance the first image of frame F according to at least the enhancement parameters to obtain the second image of frame F;

[0031] Step S14: Perform fusion processing on the second image of frame F according to the weight parameters to obtain the target image.

[0032] It is understandable that, in specific implementation, the above method can be implemented using software programs that run in the processor integrated inside the chip or chip module; or, the method can be implemented using hardware or a combination of hardware and software, such as using a dedicated chip or chip module, or using a dedicated chip or chip module combined with software programs; or, the method can also be implemented using hardware.

[0033] In the specific implementation of step S11, the first image of frame F can be obtained, where F is an integer greater than 1.

[0034] Specifically, the first F-frame image can be a series of consecutive frames captured by a camera. These F-frame images can be captured using the same exposure value or different exposure values.

[0035] It should be noted that this embodiment does not limit the format of the first image; the first image can be a Bayer image. The Bayer image can be captured by a camera based on a Bayer pattern array, and the format of the Bayer image can be RGGB, BGGR, etc. Alternatively, the first image can also be a multi-channel image obtained by processing a Bayer image; for example, it can be an RGB image or a YUV image.

[0036] It should also be noted that this embodiment does not limit the value of F, that is, it does not limit the number of images used for fusion. In a non-limiting example, F = 3. In practical applications, F can also take other values ​​greater than 1.

[0037] In a specific example, the first image of the F-frame can be captured by an HDR camera. Specifically, the first image of the F-frame can be obtained by the HDR camera performing a single shooting operation.

[0038] In practice, the HDR camera performs a single shooting operation to obtain the F-frame raw image. Since the F-frame raw image is obtained from the HDR camera's single shooting operation, the F-frame raw images are captured almost simultaneously. Therefore, the F-frame raw images are already aligned, and in this case, no alignment processing is required. Thus, the F-frame raw image can be directly used as the first F-frame image.

[0039] In another specific example, the first image of the F-frame can be obtained by aligning the original image of the F-frame, which may have been captured by a non-HDR camera.

[0040] In practice, the non-HDR camera performs multiple consecutive shooting operations to obtain the original F-frame image. Further, the original F-frame image can be aligned to obtain the first F-frame image.

[0041] The alignment process for the original F-frame image can include: selecting one frame from the original F-frame image as a reference frame, and then aligning all other frames in the original F-frame image except the reference frame to the reference frame to obtain the first F-frame image. More specifically, if the exposure values ​​of the F-frame images are different, the frame with the middle exposure value can be selected as the reference frame. If the exposure values ​​of the F-frame images are the same, the frame with the middle exposure value can be selected as the reference frame. In specific implementations, various appropriate existing alignment methods can be used to align other frames to the reference frame, such as optical flow-based image alignment methods, keypoint-based image alignment methods, and image patch-based alignment methods, but are not limited to these.

[0042] In the specific implementation of step S12, the first image of frame F can be input into a pre-trained neural network model to obtain fusion parameters. That is, the fusion parameters in this embodiment are calculated by the pre-trained neural network model using the first image of frame F.

[0043] Specifically, the neural network model is pre-built and pre-trained, and the pre-trained neural network model is deployed on the terminal. When it is necessary to fuse multiple frames of images, the multiple frames of images are simply input into the neural network model.

[0044] Reference Figure 2 , Figure 2 This is a schematic diagram of a neural network model according to an embodiment of this application. The following is a combination of... Figure 2 The structure, training process, and processing steps of the neural network model are described in a non-restrictive manner.

[0045] like Figure 2 As shown, the neural network model may include an input module 20, a first convolutional module 21, a second convolutional module 22, and a third convolutional module 23. The first convolutional module 21 can be used to calculate enhancement parameters, the second convolutional module 22 can be used to calculate guidance parameters, and the third convolutional module 23 can be used to calculate weight parameters.

[0046] The input module 20 can be used to acquire F-frame images of the input neural network model. During the model training phase, the F-frame images are F-frame sample input images, and during the model application phase, the F-frame images are the first F-frame images.

[0047] In one example, input module 20 can concatenate the input F-frame images along the channel direction to obtain input data with dimensions H×W×F×C. Here, H represents the image width, W represents the image height, F represents the number of frames, and C represents the number of channels. For example, if the input image is a Bayer image in RGGB format, then C = 4. Another example is a YUV image, where C = 3, and the three channels are the Y, U, and V channels. Yet another example is an RGB image, where C = 3, and the three channels are the R, G, and B channels.

[0048] In another example, input module 20 can also directly transmit the input multi-frame images to subsequent first convolution module 21, second convolution module 22, and third convolution module 23, etc. That is, the input data can also be F-frame images.

[0049] Furthermore, the output of the input module 20 can be connected to the first convolution module 21, the second convolution module 22, and the third convolution module 23, respectively.

[0050] Firstly, the input to the first convolution module 21 can be the input data output by the input module 20, and the output of the first convolution module 21 is the enhancement parameters. The enhancement parameters can be obtained by performing convolution processing on the input data. The convolution processing performed by the first convolution module 21 can be referred to as the first convolution processing. The enhancement parameters can include the mapping vector corresponding to each pixel in each frame of the image. The mapping vector can be used to map each pixel from the original vector space to the enhanced vector space. By performing enhancement processing on each pixel in each frame of the image through the mapping vector, the information of each pixel can be enhanced, thereby obtaining an enhanced image.

[0051] More specifically, the input data of the first convolution module 21 has a dimension of H×W×F×C, and the output of the first convolution module 21 has a dimension of H×W×F×S×P. That is, the dimension of the mapping vector is S×P. In other words, each pixel in each frame of the image corresponds to an S×P mapping vector. Here, S represents the number of scenes. S can be preset. In a non-limiting example, S=4. In other embodiments, S can also take other values. P represents the dimension of the mapping vector, and the value of P can depend on the enhancement purpose. More specifically, if the enhancement purpose is only to enhance the image brightness, then P=1; if the enhancement purpose is to enhance both the color and brightness of the image, then P=C×(C+1).

[0052] like Figure 2 As shown, in one embodiment, the first convolution module 21 may include a downsampling unit 211, a first convolution unit 212, and an upsampling unit 213.

[0053] The downsampling unit 211 can be used to downsample the input data to obtain downsampled data. More specifically, the downsampling unit 211 can downsample each frame of the image and concatenate the downsampled images in the channel direction to obtain downsampled data. It should be noted that this embodiment does not limit the downsampling method; for example, bilinear interpolation can be used for downsampling. In a specific example, the dimension of the downsampled data can be 256×256×F×C.

[0054] Furthermore, the first convolutional unit 212 can be used to perform a first convolution on the downsampled data to obtain intermediate enhancement parameters. The operator used in the first convolutional unit 212 can be any combination of one or more of the following: ordinary convolution operators, separable convolution operators, residual operators, activation function operators, and pooling operators, etc. In a non-limiting example, the convolution operators and activation function operators in the first convolutional unit 212 are concatenated. The convolution operators can include four groups of 3x3 convolution operators, with a stride of 2. The number of convolution kernels in the four groups can be 16, 32, 64, and C', respectively, where C' = F × S × P. The activation function operator can be the ReLU function. The dimension of the intermediate enhancement parameters output by the first convolutional unit 212 can be N × N × C', where N represents the spatial resolution. Optionally, N can be set to 16.

[0055] Furthermore, the upsampling unit 213 can be used to upsample the intermediate enhancement parameters to obtain the enhancement parameters. It should be noted that this embodiment does not limit the upsampling method. For example, bilinear interpolation can be used for upsampling to obtain enhancement parameters with dimensions H×W×F×S×P.

[0056] From the above, Figure 2 The processing steps of the first convolution module 21 for the input image can be represented by the following formula:

[0057] O 1＝ U(B1(G(D(I f ))));

[0058] Where O1 represents the enhancement parameter, I f Let f represent the f-th frame image, 1≤f≤F, where f is a positive integer, D represents downsampling, G represents channel direction connection, B1 represents the first convolution process, and U represents upsampling.

[0059] In other embodiments of this application, the first convolution module 21 may also omit the downsampling unit 211 and the upsampling unit 213. That is, the first convolution process can be directly performed on the input data with dimensions H×W×F×C to obtain the enhancement parameters with dimensions H×W×F×S×P.

[0060] Secondly, the input to the second convolution module 22 is the input data output by the input module 20, and the output of the second convolution module 22 is the guiding parameters. The guiding parameters can be obtained by performing convolution processing on the input data. The convolution processing performed by the second convolution module 22 can be referred to as the second convolution processing. The guiding parameters can include the scene vector corresponding to each pixel. The scene vector includes the association weights between the pixel and each scene, and the sum of the association weights between the pixel and each scene in each scene vector can be 1. The association weights can be used to characterize the degree of association between the pixel and the scene; the higher the degree of association, the larger the association weight. It should be noted that the scene vectors corresponding to pixels at the same position in each frame of the image are the same; that is, the guiding parameters can be used to describe the information of each position in the overall multi-frame image.

[0061] More specifically, the input data of the second convolutional module 22 has a dimension of H×W×F×C, and the output data of the second convolutional module 22 has a dimension of H×W×S. That is, the scene vector has a dimension of 1×S. In other words, each pixel position corresponds to a 1×S scene vector.

[0062] like Figure 2 As shown, in one embodiment, the second convolution module 22 may include a second convolution unit 221 and a first normalization unit 222. The second convolution unit 221 can be used to perform a second convolution process on the input data to obtain intermediate guiding parameters. More specifically, the operator used by the second convolution unit 221 can be any combination of one or more of the following: ordinary convolution operators, separable convolution operators, residual operators, activation function operators, and pooling operators, etc. In a non-limiting example, the convolution operator and the activation function operator in the second convolution unit 221 are concatenated. The convolution operator may include two sets of 3x3 convolution operators, the stride of which can be 1, and the number of convolution kernels in the two sets of convolution operators can be 16 and S, respectively. The activation function operator can be the ReLU function. The dimension of the intermediate guiding parameters output by the second convolution unit 221 can be H×W×S.

[0063] Furthermore, the first normalization unit 222 can be used to normalize the intermediate guidance parameters to obtain the guidance parameters. Specifically, the first normalization unit 222 can use a normalization operator to normalize the intermediate guidance parameters so that the value of the association weight in the scene vector corresponding to each pixel is between [0, 1], thereby obtaining the guidance parameters.

[0064] The normalization operator used by the first normalization unit 222 can be expressed by the following formula:

[0065]

[0066] Where i can represent the index of dimension H, j can represent the index of dimension W, k can represent the index of dimension S, 1≤i≤H, 1≤j≤W, 1≤k≤S, O2 represents the guiding parameter, and O'2 represents the intermediate guiding parameter.

[0067] From the above, Figure 2 The image processing steps of the second convolution module 22 shown can be represented by the following formula:

[0068] O2=N(B2(G(I f )))

[0069] Where O2 represents the bootstrap parameter, I f Let f represent the image of the f-th frame, 1≤f≤F, G represents the connection in the channel direction, B2 represents the second convolution process, and N represents the normalization process.

[0070] It should be noted that in other embodiments, the second convolutional module 22 may not be included. That is, the second convolutional module 22 is optional. It is understood that if the neural network model does not include the second convolutional module 22, the dimension of the enhancement parameters can be H×W×F×P.

[0071] In a specific example, if the first convolutional module 21 does not include the downsampling unit 211 and the upsampling unit 213, then the second convolutional module 22 may not be provided. It should also be noted that if the first convolutional module 21 does not include the downsampling unit 211 and the upsampling unit 213, the second convolutional module 22 may also be provided, and this application does not impose any restrictions on this.

[0072] Thirdly, the input to the third convolution module 23 can be the input data output by the input module 20, and the output of the third convolution module 23 is weight parameters. These weight parameters can be obtained by performing convolution processing on the input data. The convolution processing performed by the third convolution module 23 can be referred to as the third convolution processing. The weight parameters include the weight values ​​corresponding to each pixel in each frame of the image.

[0073] More specifically, the dimension of the input data of the third convolution module 23 is H×W×F×C, and the dimension of the output data of the third convolution module 23 is H×W×F.

[0074] like Figure 2As shown, in one embodiment, the third convolution module 23 may include a third convolution unit 231 and a second normalization unit 232. The third convolution unit 231 performs a third convolution on the input data to obtain intermediate weight parameters. More specifically, the operator used by the third convolution unit 231 can be any combination of one or more of the following: ordinary convolution operators, separable convolution operators, residual operators, activation function operators, and pooling operators. In a non-limiting example, the convolution operator and activation function operator in the third convolution unit 231 are concatenated, where the convolution operator may include two sets of 3x3 convolution operators. The stride of the convolution operator can be 1, and the number of convolution kernels in the two sets of convolution operators can be 16 and F, respectively. The activation function operator can be the ReLU function. The dimension of the intermediate weight parameters output by the third convolution unit 231 can be H×W×F.

[0075] Furthermore, the second normalization unit 232 can be used to normalize the intermediate weight parameters to obtain the weight parameters. Specifically, the second normalization unit 232 can use a normalization operator to normalize the intermediate weight parameters so that the weight value corresponding to each pixel is between [0, 1], thereby obtaining the weight parameters.

[0076] The normalization operator used in the second normalization unit 232 can be expressed by the following formula:

[0077]

[0078] Where i can represent the index of dimension H, j can represent the index of dimension W, l can represent the index of dimension F, 1≤i≤H, 1≤j≤W, 1≤l≤F, O3 represents the guiding parameter, and O3' represents the intermediate guiding parameter.

[0079] From the above, Figure 2 The image processing steps of the third convolution module 23 shown can be represented by the following formula:

[0080] O3=N(B3(G(I f )))

[0081] Where O3 represents the boot parameter, I f Let f represent the f-th frame image, 1≤f≤F, G represents the channel direction connection, B3 represents the third convolution process, and N represents the normalization process.

[0082] Furthermore, the neural network model may also include an output module 26. In a specific example, the output module 26 can be used to output enhancement parameters, guidance parameters, and weight parameters. Specifically, the input of the output module 26 can be connected to the outputs of the first convolutional module 21, the second convolutional module 22, and the third convolutional module 23, respectively, so that the neural network model outputs the calculated enhancement parameters, guidance parameters, and weight parameters.

[0083] In another specific example, output module 26 can be used to output the fused image. During the model training phase, the output of output module 26 can be a single-frame sample target image; during the model application phase, the output of output module 26 is a single-frame target image.

[0084] Specifically, the neural network model may further include an enhancement module 24 and a fusion module 25. The enhancement module 24 can be used to enhance the input data according to enhancement parameters and guidance parameters to obtain an enhanced image, and the fusion module 25 can be used to fuse the enhanced image according to weight parameters to obtain a fused image.

[0085] More specifically, the inputs of enhancement module 24 are connected to the outputs of input module 20, first convolution module 21, and second convolution module 22, respectively. The inputs of fusion module 25 are connected to the outputs of enhancement module 24 and third convolution module 23, respectively. The output of fusion module 25 is connected to output module 26.

[0086] For more information on enhancement module 24 and fusion module 25, please refer to the relevant descriptions of steps S13 and S14 below, which will not be repeated here.

[0087] During the model training phase, training data can be used to train the neural network model until the neural network model converges.

[0088] Specifically, the training data may include: F-frame sample input images and single-frame sample target images. The image content of the sample input images and the sample target images is the same, but their image quality differs, with the image quality of the sample target images being superior to that of the sample input images.

[0089] In a specific example, a single-frame sample target image can be obtained by processing the F-frame sample input image using other existing multi-frame image fusion methods.

[0090] In another specific example, the sample input image and the sample target image can be obtained by capturing the same content in the same shooting scene using different shooting devices. More specifically, the F-frame sample input image can be captured by a first shooting device, and the single-frame sample target image can be captured by a second shooting device, wherein the shooting performance of the second shooting device is superior to that of the first shooting device. For example, the first shooting device can be a mobile phone, and the second shooting device can be a single-lens reflex camera.

[0091] In another specific example, the sample target image can be obtained by capturing images using the second imaging device described above, and the sample target image has high image quality. The sample input image can be obtained by performing an inverse transformation on the sample target image. For example, an inverse transformation of image brightness and image contrast can be performed on the sample target image to obtain the F-frame sample input image.

[0092] Furthermore, the constructed neural network model can be trained using training data. The loss function used during training can be L1 loss, L2 loss, Structural Similarity (SSIM) loss, etc., but is not limited to these. Specifically, the L1 loss function compares differences pixel-by-pixel and then takes the absolute value; the L2 loss function compares differences pixel-by-pixel and then takes the square; the SSIM loss function considers luminance, contrast, and structure metrics, which is more conducive to reflecting image details. The training method can be any existing deep learning algorithm, and this embodiment does not impose any limitations on it.

[0093] Therefore, the solution in this embodiment adopts an end-to-end approach, which can learn the parameters of the neural network model based on training data without the need for manual parameter tuning. The trained neural network model can calculate enhancement parameters, guidance parameters, and weight parameters based on the input image, and can also fuse the image based on the obtained enhancement parameters, guidance parameters, and weight parameters to obtain a fused image.

[0094] In practice, the trained neural network model can be deployed on different hardware units within the terminal. Deploying on different hardware units within the terminal means that different modules of the neural network model can be set up on different hardware units. In practical applications, the neural network model can be deployed on one or more hardware units of the terminal depending on the type of terminal. For example, GPUs and DSPs have good acceleration effects for downsampling, upsampling, and mapping processes, so the first convolutional module 21 can be deployed on a GPU or DSP. As another example, NPUs have good acceleration effects for convolutional neural networks, so the second convolutional module 22 and the third convolutional module 23 can be deployed on the NPU, or the first convolutional module 21 can also be deployed on the NPU.

[0095] Continue to refer to Figure 1 In the specific implementation of step S12, the first image of frame F can be input into the neural network model to obtain fusion parameters for fusing the first image of frame F. The fusion parameters may include enhancement parameters and weight parameters. Further, the fusion parameters may also include guidance parameters. The enhancement parameters, guidance parameters, and weight parameters can be calculated by the pre-trained neural network model using the first image of frame F.

[0096] For details on calculating fusion parameters in neural network models, please refer to the above section. Figure 2 The relevant descriptions will not be repeated here.

[0097] In the specific implementation of step S13, the first image of frame F can be enhanced according to the enhancement parameters to obtain the second image of frame F. More specifically, the mapping vector corresponding to each pixel of the first image of each frame can be used to map the pixel to obtain the second image of frame F.

[0098] Alternatively, the first image of frame F can be enhanced based on enhancement parameters and guidance parameters to obtain the second image of frame F. In one embodiment of this application, the neural network model is... Figure 2 The structure shown has a neural network model output enhancement parameters with dimensions of H×W×F×S×P, and a guiding parameter with dimensions of H×W×S. On one hand, a 1×S scene vector can be used to perform a first enhancement process on the first image of frame F, increasing the information in the scene channels. On the other hand, an S×P mapping vector can be used to perform a second enhancement process on each pixel of the first image of frame F, enhancing the image color and / or brightness, thus obtaining the second image of frame F.

[0099] In a specific example, the enhancement objective is to enhance only the image brightness. As mentioned above, when only image brightness is enhanced, P = 1. Accordingly, the enhancement parameter O1 has a dimension of H×W×F×S×1, and the guiding parameter O2 has a dimension of H×W×S. In other words, each pixel in the first frame of each image corresponds to an S×1 mapping vector, and each pixel corresponds to a 1×S scene vector. Further, step S13 can be represented by the following formula:

[0100]

[0101] Among them, T f Represents the second image in frame f, I f Let f represent the first image in frame f, h represent the index in dimension H, w represent the index in dimension W, and c represent the index in dimension C, where 1 ≤ h ≤ H, 1 ≤ w ≤ W, 1 ≤ c ≤ C, and T. f (h, w, c) can represent the value at index (h, w, c) in the second image of frame f. f (h, w, c) can represent the value at index (h, w, c) in the first image of frame f, where s represents the index in dimension S, 1 ≤ s ≤ S, and f can represent the index in dimension F, 1 ≤ f ≤ F. O1(h, w, f, s, 1) represents the value at index (h, w, f, s, 1) in the mapping parameters, and O2(h, w, s) represents the value at index (h, w, s) in the guiding parameters. h, w, c, f, and s are all positive integers. Therefore, the second image of frame F can be obtained, which has enhanced brightness compared to the first image.

[0102] In another specific example, the enhancement objective is to enhance the color and brightness of the image. As mentioned above, when the enhancement objective is to enhance the color and brightness of the image, P = C × (C + 1). Accordingly, the dimension of the enhancement parameter O1 is H × W × F × S × C × (C + 1), and the dimension of the guiding parameter O2 is H × W × S.

[0103] As an example, the mapping parameter O1 can be decomposed into: a first mapping parameter R1 and a second mapping parameter T1, where the dimension of R1 is H×W×F×S×C×C, and the dimension of T1 is H×W×F×S×1×C.

[0104] Furthermore, step S13 can be expressed by the following formula:

[0105]

[0106] Where f represents the index of dimension F, 1≤f≤F, T f I represents the second image in frame f. fLet f represent the first image in frame f, h represent the index in dimension H, w represent the index in dimension W, 1 ≤ h ≤ H, 1 ≤ w ≤ W, and T. f (h,w) represents the 1×C vector corresponding to the pixel in the h-th row and w-th column of the second image in the f-th frame. f (h,w) represents the 1×C vector corresponding to the pixel in the h-th row and w-th column of the first image in the f-th frame, s represents the index of dimension S, 1≤s≤S, R1(h,w,f,s) represents the C×C vector in R1 with index (h,w,f,s), T1(h,w,f,s) represents the 1×C vector in T1 with index (h,w,f,s), and O2(h,w,s) represents the value of index (h,w,s) in the guiding parameters. h, w, f, and s are all positive integers.

[0107] As described above, by executing step S13, the second image of frame F can be obtained. The second image and the first image are in one-to-one correspondence; each frame's second image is obtained by enhancing the corresponding first image. It should be noted that the above description only exemplifies the calculation process of the enhancement process. Depending on the enhancement parameters and guidance parameters of different dimensions, other calculation methods can also be used to calculate the second image of frame F, and this embodiment is not limited in this regard.

[0108] In the specific implementation of step S14, the second image of frame F can be fused according to the weight parameters. Since the weight parameters include the weight values ​​corresponding to each pixel in each frame of the first image, and the second image and the first image are in one-to-one correspondence, the weight parameters can include the weight values ​​corresponding to each pixel in each frame of the second image. The value of each pixel in the target image can be obtained by weighted summation of the pixel values ​​of the corresponding pixels in each frame of the second image according to the weight parameters.

[0109] Specifically, step S14 can be represented by the following formula:

[0110]

[0111] Where h can represent the index of dimension H, w can represent the index of dimension W, and c can represent the index of dimension C, 1≤h≤H, 1≤w≤W, 1≤c≤C, Y(h,w,c) represents the value of the target image at index (h,w,c), and T f (h,w,c) can represent the value at index (h,w,c) in the second image of frame f, and O3(h,w,f) represents the value at index (h,w,f) in the weight parameter O3.

[0112] Thus, a single frame of the target image can be obtained. Since the weight parameters include the weights corresponding to each pixel in each frame, the fusion process can adapt to the weighted weights of different regions and different frames, effectively suppressing ghosting.

[0113] Therefore, this application provides an end-to-end image processing method. The first image of frame F is input into a pre-trained neural network model to obtain a single-frame target image output by the model. This method fuses multiple frames to obtain a target image with high clarity and high dynamic range. This method can be applied to shooting scenarios with high dynamic range, low brightness, and backlighting, and can solve key problems such as alignment, ghosting removal, image fusion, and image color / brightness enhancement to obtain clearer, higher-quality images.

[0114] Furthermore, the above-described solution involves low computational cost and is time-efficient. In practical applications, the neural network model can be deployed on low-end devices to execute the methods described above. Alternatively, the methods described above can be applied to scenarios with high real-time requirements, such as preview mode and camera mode, balancing processing efficiency and fusion quality. As an example, the above methods can be applied to in-vehicle devices, which have extremely high real-time requirements for algorithms. By executing the above methods, target images can be obtained quickly with high clarity, and areas of different brightness levels can be well exposed, thereby improving the accuracy of target detection or scene analysis.

[0115] Reference Figure 3 , Figure 3 This is a schematic diagram of the structure of an image processing device according to an embodiment of this application. Figure 3 The apparatus shown may include:

[0116] The acquisition module 31 is used to acquire the first image of frame F, where F is an integer greater than 1;

[0117] The parameter calculation module 32 is used to input the first image of frame F into a pre-trained neural network model to obtain fusion parameters. The fusion parameters include enhancement parameters and weight parameters. The enhancement parameters include the mapping vector corresponding to each pixel of the first image of each frame, and the weight parameters include the weight value corresponding to each pixel of the first image of each frame.

[0118] Enhancement module 33 is used to enhance the first image of F frame at least according to the enhancement parameters to obtain the second image of F frame;

[0119] The fusion module 34 is used to perform fusion processing on the second image of the F frame according to the weight parameters to obtain the target image.

[0120] For more information on the working principle, working method, and beneficial effects of the image processing device in the embodiments of this application, please refer to the relevant description of the image processing method above, which will not be repeated here.

[0121] In practice, Figure 3 The image processing device shown may correspond to a chip with image processing function in a terminal, or to a chip with image processing function, such as an image signal processing (ISP) chip; or to a chip module with image processing function, or to a terminal.

[0122] This application also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it performs the steps of the image processing method described above. The storage medium may include ROM, RAM, a magnetic disk, or an optical disk, etc. The storage medium may also include non-volatile memory or non-transitory memory, etc.

[0123] This application also provides a terminal, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor runs the computer program, it performs the steps of the image processing method described above. The terminal includes, but is not limited to, terminal devices such as mobile phones, computers, and tablet computers.

[0124] This application also provides a chip that can be used to perform the image processing method described above. Alternatively, the chip may include... Figure 3 The image processing device is shown. In a specific example, the chip may be an ISP chip.

[0125] This application also provides a chip module that can be used to perform the image processing method described above. Alternatively, the chip module may include... Figure 3 The image processing device shown.

[0126] It should be understood that in the embodiments of this application, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0127] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0128] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means.

[0129] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatus, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and other division methods may exist in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0130] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or in a combination of hardware and software functional units. For example, for various devices or products applied to or integrated into a chip, each module / unit can be implemented using hardware such as circuits, or at least some modules / units can be implemented using software programs running on a processor integrated within the chip, while the remaining (if any) modules / units can be implemented using hardware such as circuits; for various devices or products applied to or integrated into a chip module, each module / unit can be implemented using hardware such as circuits, and different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components of the chip module, or at least some modules / units can be implemented using hardware such as circuits. The components can be implemented using software programs that run on the processor integrated within the chip module. The remaining (if any) modules / units can be implemented using hardware methods such as circuits. For various devices and products applied to or integrated into the terminal, each of its components / units can be implemented using hardware methods such as circuits. Different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or in different components within the terminal. Alternatively, at least some modules / units can be implemented using software programs that run on the processor integrated within the terminal, while the remaining (if any) modules / units can be implemented using hardware methods such as circuits.

[0131] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article indicates that the preceding and following related objects have an "or" relationship.

[0132] In this application's embodiments, "multiple" refers to two or more. The descriptions of "first," "second," etc., appearing in this application's embodiments are merely illustrative and for distinguishing the described objects; they do not indicate any order and do not imply a specific limitation on the number of devices in this application's embodiments, nor do they constitute any limitation on the embodiments of this application. Although this application discloses the above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of this application; therefore, the scope of protection of this application should be determined by the scope defined in the claims.

[0133] While this application discloses the above information, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of this application; therefore, the scope of protection of this application shall be determined by the scope defined in the claims.

Claims

1. An image processing method, characterized in that, The method includes: Get the first image of frame F, where F is an integer greater than 1; The first image of frame F is input into a pre-trained neural network model to obtain fusion parameters. The fusion parameters include enhancement parameters and weight parameters. The enhancement parameters include the mapping vector corresponding to each pixel of the first image of each frame, and the weight parameters include the weight value corresponding to each pixel of the first image of each frame. The first image of frame F is enhanced at least according to the enhancement parameters to obtain the second image of frame F; The second image of frame F is fused according to the weight parameters to obtain the target image; The fusion parameters further include: guiding parameters, which include a scene vector corresponding to each pixel, and the scene vector includes the association weight between the pixel and each scene. The higher the degree of association, the greater the association weight. At least the enhancement processing of the first image of frame F according to the enhancement parameters includes: Based on the guidance parameters and the enhancement parameters, the first image of frame F is enhanced to obtain the second image of frame F.

2. The image processing method according to claim 1, characterized in that, The processing steps of the neural network model for the first image of frame F include: The first image of frame F is downsampled to obtain downsampled data; The downsampled data is subjected to a first convolution process to obtain intermediate augmented data; The intermediate augmented data is upsampled to obtain the augmentation parameters.

3. The image processing method according to claim 1, characterized in that, The processing steps of the neural network model for the first image of frame F include: The first image of frame F is subjected to a second convolution process to obtain the guiding parameters.

4. The image processing method according to claim 1, characterized in that, The processing steps of the neural network model for the first image of frame F include: The first image of frame F is subjected to a third convolution process to obtain the weight parameters.

5. The image processing method according to claim 1, characterized in that, The steps for obtaining the neural network model include: Acquire training data, which includes: F-frame sample input images and single-frame sample target images; The neural network model is trained using the training data until it converges.

6. The image processing method according to claim 1, characterized in that, Obtaining the first image of frame F includes: Obtain the original image of frame F; If the original F-frame image is captured by an HDR camera, then the original F-frame image is used as the first F-frame image; otherwise, the first F-frame image is aligned to obtain the first F-frame image.

7. An image processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the first image of frame F, where F is an integer greater than 1; The parameter calculation module is used to input the first image of frame F into a pre-trained neural network model to obtain fusion parameters. The fusion parameters include enhancement parameters and weight parameters. The enhancement parameters include the mapping vector corresponding to each pixel of the first image of each frame, and the weight parameters include the weight value corresponding to each pixel of the first image of each frame. An enhancement module is configured to enhance the first image of frame F based at least on the enhancement parameters to obtain a second image of frame F; The fusion module is used to perform fusion processing on the second image of frame F according to the weight parameters to obtain the target image; The fusion parameters further include: guiding parameters, which include a scene vector corresponding to each pixel, and the scene vector includes the association weight between the pixel and each scene. The higher the degree of association, the greater the association weight. The enhancement module is used to enhance the first image of the F-frame according to the guiding parameters and the enhancement parameters to obtain the second image of the F-frame.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processor, it performs the steps of the image processing method according to any one of claims 1 to 6.

9. A terminal comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor runs the computer program, it performs the steps of the image processing method according to any one of claims 1 to 6.

Images