Image processing method and device, electronic equipment and computer readable storage medium

Abstract

The invention relates to an image processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring multiple frames of first images and at least one frame of second image; the brightness of the first image is greater than that of the second image; determining a reference image from the multiple frames of first images; performing deblurring processing on the reference image, and fusing the reference image after deblurring processing and the multiple frames of first images to obtain a first fused image; and fusing the first fused image and the second image to obtain a second fused image. By adopting the method, the time consumption is short and the power loss is low under the condition of ensuring the good quality of the second fusion image.

Description

Technical Field

[0001] The present application relates to image processing, and in particular to an image processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] With the development of image technology, in order to improve image quality, electronic devices capture multiple frames of images and fuse them to obtain a new image.

[0003] In traditional solutions, in order to improve the clarity of the image, each frame of the image is deblurred, resulting in serious time-consuming and power-consuming losses. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, device, electronic device, and computer-readable storage medium, which can ensure good image quality with short time consumption and low power consumption loss.

[0005] In a first aspect, the present application provides an image processing method. The method comprises:

[0006] Acquire multiple frames of first images and at least one frame of second image; the brightness of the first image is greater than the brightness of the second image;

[0007] Determine a reference image from the plurality of frames of first images;

[0008] Deblurring the reference image, and fusing the deblurred reference image with the multiple frames of first images to obtain a first fused image;

[0009] The first fused image and the second image are fused to obtain a second fused image.

[0010] In a second aspect, the present application also provides an image processing device. The device comprises:

[0011] An acquisition module, used to acquire multiple frames of first images and at least one frame of second image; the brightness of the first image is greater than the brightness of the second image;

[0012] A screening module, used for determining a reference image from the plurality of frames of first images;

[0013] A first fusion module is used to perform a deblurring process on the reference image, and fuse the deblurred reference image with the multiple frames of first images to obtain a first fused image;

[0014] The second fusion module is used to fuse the first fused image and the second image to obtain a second fused image.

[0015] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0016] Acquire multiple frames of first images and at least one frame of second image; the brightness of the first image is greater than the brightness of the second image;

[0017] Determine a reference image from the plurality of frames of first images;

[0018] Deblurring the reference image, and fusing the deblurred reference image with the multiple frames of first images to obtain a first fused image;

[0019] The first fused image and the second image are fused to obtain a second fused image.

[0020] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0021] Acquire multiple frames of first images and at least one frame of second image; the brightness of the first image is greater than the brightness of the second image;

[0022] Determine a reference image from the plurality of frames of first images;

[0023] Deblurring the reference image, and fusing the deblurred reference image with the multiple frames of first images to obtain a first fused image;

[0024] The first fused image and the second image are fused to obtain a second fused image.

[0025] The above-mentioned image processing method obtains multiple frames of first images and at least one frame of second images; the brightness of the first image is greater than the brightness of the second image, so that the information represented by the first image and the second image in different image regions is different, providing different information dimensions; determining a reference image from the multiple frames of first images; deblurring the reference image, deblurring the reference image, and fusing the deblurred reference image and the multiple frames of first images to obtain a first fused image; thereby, a reference image is selected from multiple frames of first images, and multiple frames of first images are fused based on the deblurred reference image as a reference, without deblurring each frame of the first image, and in this way, a higher image quality is guaranteed, the time consumption is short, and the power consumption loss is low; the first fused image and the second image are fused to obtain a second fused image, and the information represented by the second image in certain regions can not only improve the quality of the first fused image again, but also has the advantages of short time consumption and low power consumption loss. Thus, when the second fused image is of good quality, the time consumption is short and the power consumption loss is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 A diagram showing an application environment of an image processing method in an embodiment;

[0028] Figure 2 is a flowchart of an image processing method in one embodiment;

[0029] Figure 3 A flowchart of a deblurring process in one embodiment;

[0030] Figure 4 is a flowchart of an image processing method in one embodiment;

[0031] Figure 5 is a structural block diagram of an image processing device in one embodiment;

[0032] Figure 6 FIG. 4 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0034] The image processing method provided in the embodiment of the present application is illustrated by an application scenario of an electronic device, which may be a terminal or a server. Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 via a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0035] Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers. This solution can be implemented based on the terminal 102 alone, or it can be implemented based on the communication between the terminal 102 and the server 104.

[0036] In one embodiment, Figure 2 As shown, an image processing method is provided, comprising the following steps:

[0037] Step 202, acquiring multiple frames of first images and at least one frame of second image; the brightness of the first image is greater than the brightness of the second image.

[0038] The first image and the second image are images acquired from the same object under the same environment. The brightness of the first image is greater than that of the second image, which can make the accuracy of different information represented by different areas of the first image and the second image different, thereby laying a foundation for a clearer second fused image. Optionally, due to the brightness difference between the areas in the image, there are highlight areas and dark areas in the image, and the brightness of the highlight areas is greater than that of the dark areas; in this case, the brightness of the first image is greater than that of the second image, which can make the information represented by the highlight areas in the first image more accurate relative to the information represented in the second image, and the information represented by the dark areas in the second image is more accurate relative to the information represented by the highlight areas in the first image, so that the information represented by the first image and the second image complement each other.

[0039] Optionally, the first image may be an image that satisfies a first brightness condition. Optionally, the first brightness condition is an expected brightness, under which the highlight area of ​​the first image can more accurately represent information. Optionally, the second image may be an image that satisfies a second brightness condition.

[0040] The multiple frames of first images are first images captured at different times during the image capture process, and the distribution of highlight areas and dark areas of each frame of the first image may change based on changes in ambient brightness, so that the accuracy of information represented by the first image in different local areas is different. Optionally, the sensitivity of each frame of the first image is the same and the exposure time is the same.

[0041] Optionally, the second brightness condition can be set based on a certain exposure corresponding to the second image, and the exposure is less than the exposure corresponding to the expected brightness; the dark area of ​​the second image can more accurately represent the information. Optionally, the exposure during the acquisition process can be used to determine whether the brightness of the first image meets the first brightness condition; the exposure during the acquisition process can be used to determine whether the brightness of the second image meets the second brightness condition. Optionally, the exposure can be adjusted by sensitivity (ISO) and exposure time, the sensitivity is positively correlated with the size of the aperture, and the exposure time is positively correlated with the length of time the shutter is open.

[0042] For example, when the first brightness condition represents that the expected brightness of the first image is 128, the exposure corresponding to 128 is 0EV, and the corresponding required ISO and exposure time are counted so that the exposure reaches 0EV, so that the brightness of the first image is 128; and the second brightness condition can be represented by -EV. Relative to the brightness of 128, the second brightness condition represents that the expected brightness of the second image is 64, 32 or 16, then the exposure corresponding to 64, 32 or 16 is 0EV, and the corresponding required ISO and exposure time are counted so that the exposure reaches 0EV, so that the brightness of the second image meets the second brightness condition.

[0043] In one embodiment, obtaining multiple frames of first images and at least one frame of second image includes: performing image acquisition based on a first sensitivity and a first exposure time to obtain multiple frames of first images; performing image acquisition based on a second sensitivity and a second exposure time to obtain at least one frame of second image; the exposure represented by the first sensitivity and the first exposure time is less than the exposure represented by the second sensitivity and the second exposure time.

[0044] Step 204: determine a reference image from the multiple frames of first images.

[0045] The reference image is a base frame image in the first image, which is used as a reference for fusing multiple frames of the first image. Optionally, the reference image is a first image selected according to image display requirements. The reference image can be a first image frame displayed by the electronic device at the time when the shutter is activated, or it can be the first image with the best clarity. Among them, the first image frame displayed by the electronic device at the time when the shutter is activated is the image displayed to the user when the user activates the shutter.

[0046] In one embodiment, determining the reference image from the multiple frames of first images includes: selecting a reference image from the multiple frames of first images according to the image attributes of each of the multiple frames of first images. Optionally, the image attribute may be acquisition order or definition.

[0047] Step 206 , deblurring the reference image, and fusing the deblurred reference image with the multiple frames of first images to obtain a first fused image.

[0048] The deblurred reference image is a result of deblurring the reference image. In the fusion process of multiple frames of first images, deblurring the reference image alone can reduce time consumption and power consumption.

[0049] The first fused image is an image obtained by fusing multiple frames of first images. Since the accuracy of information represented by the first image in local areas of different images is different, fusing multiple frames of first images can improve the clarity of the first fused image relative to multiple frames of first images. Moreover, fusing the deblurred reference image with the multiple frames of first images is to fuse the multiple frames of first images into the deblurred reference image, so that the deblurred reference image has higher clarity, and there is no need to perform deblurring on the multiple frames of first images separately.

[0050] In one embodiment, deblurring a reference image includes: inputting the reference image into a deblurring model; deblurring the reference image through the deblurring model, and outputting the deblurred reference image. Thus, by processing the reference image through the deblurring model, the clarity of the reference image can be improved. Optionally, the clarity can be calculated by data, or can be represented by the details displayed by the image; optionally, the clarity can be represented by a clarity value, the number of details displayed by the image, or the number of textures within the details. Exemplarily, the higher the clarity, the more details or textures within the details displayed by the image; if the clarity is lower, the fewer details or textures within the details displayed by the image.

[0051] In one embodiment, it can be performed by a dynamic vision sensor (DVS), that is, a dynamic vision sensor deblurring algorithm is used to deblur the reference image. At this time, a dynamic vision sensor deblurring model can be used for image processing, and the reference image is used as the image to be processed in the dynamic vision sensor deblurring model; correspondingly, the reference image is deblurred through the deblurring model, and the deblurred reference image is output, including: performing multiple feature extractions on the image to be processed through a multi-layer downsampling network in a segmentation model to obtain a feature vector of each layer of the downsampling network; obtaining a corresponding reference feature vector based on the feature vector of each layer of the network, and performing feature fusion of the reference feature vector and the feature vector through a multi-layer upsampling network to obtain the category of each pixel.

[0052] In one embodiment, when multiple feature extractions are performed on the image to be processed, the feature extractions are performed separately on the image to be processed in the color space and the dynamic vision sensor image; the feature vector of each downsampling network layer includes the first image feature extracted by each downsampling network layer from the image to be processed in the color space, and the second image feature extracted by each downsampling network layer from the dynamic vision sensor image; correspondingly, a corresponding reference feature vector is obtained according to the feature vector of each layer of the network, including: fusing the first image feature and the second image feature of each layer of the network to obtain the corresponding reference feature vector; wherein the first image feature includes the convolution result and residual value of the image to be processed in each downsampling network layer, and the second image feature includes the convolution result and residual value of the dynamic vision sensor image in each downsampling network layer.

[0053] Alternatively, if Figure 3As shown, it includes a 3×3 convolution result (3×3Convoluton), a residual value (ResBlock) and a transposed convolution result (Tconv), wherein the transposed convolution result is a reference feature vector obtained by fusion; in the UNET network structure diagram, the UNET network may include a first part composed of a multi-layer downsampling network and a second part composed of a multi-layer downsampling network, the features extracted by the first part include the first image features and the second image features, and the second part features are reference feature vectors obtained by fusion of the first image features and the second image features. Optionally, the first part may be a downsampling module composed of two 3×3 convolution layers and one 2×2 maximum pooling layer maxpooling repeated multiple times, which can be used to realize feature extraction. The second part is an upsampling module composed of one upsampling convolution layer, feature splicing copyand crop and two 3×3 convolution layers repeated multiple times, which can be used to realize feature fusion. The last layer of the second part converts the number of channels into the desired number of categories through a 1×1 convolution. Among them, the first part can perform multiple downsampling operations, and the second part can perform multiple upsampling processes accordingly. For example, four downsampling and four upsampling may be performed.

[0054] Optionally, the upsampling module of the first part may include a multi-layer network, each of which includes two 3×3 convolutional layers and a maximum pooling layer, so the multi-layer network can be considered as a multi-layer downsampling network. When performing feature extraction in the first part, since each downsampling network includes two 3×3 convolutional layers and a maximum pooling layer, the convolution operation can be performed on the processed image in turn according to each downsampling network to obtain the convolution result; the convolution result is downsampled to obtain the downsampling result, so as to determine the downsampling result as the feature vector of each downsampling network. That is, feature extraction is performed in turn through the convolutional layer and the maximum pooling layer in the first downsampling network, and feature extraction is performed according to the convolutional layer and the maximum pooling layer in the second downsampling network, until the convolutional layer and the maximum pooling layer in all downsampling layer networks have completed feature extraction. The number of channels doubles after each downsampling.

[0055] The upsampling module of the second part can include a multi-layer network, each of which includes an upsampling convolution layer, a feature splicing copy and crop, and two 3×3 convolution layers, so that the multi-layer network of the second part is a multi-layer upsampling network. When the second part is processed, the feature vector of each downsampling network of the first part is firstly upsampled by each upsampling network of the second part to obtain the upsampling result. When the feature is spliced, the feature vector of the same size is first intercepted from the feature vector of the corresponding output of the same layer of the first part as the reference feature vector of each upsampling network of the second part, and the reference feature vector is fused with the upsampling result of the second part to obtain the fusion feature of each upsampling network. The fusion here can be channel splicing. The fusion feature of each upsampling network is further convolved to obtain the category of each pixel. After each upsampling convolution, the height and width are doubled, and the number of channels is halved, which is used to merge and splice with the shallow features of the first part. After each downsampling, the number of channels becomes twice that of the previous layer of the network; after each upsampling, the number of channels becomes half of the previous layer of the network. Figure 4 As shown in , downsampling is performed 4 times, and the number of channels changes from 64 to 1024; upsampling is performed 4 times, and the number of channels changes from 1024 to 64.

[0056] Therefore, through the segmentation model of the UNET structure, the category of each pixel is obtained for the reference image, and the upsampling part will fuse the output of the feature extraction part to fuse the multi-scale features together. The fusion of the features extracted by the shallow network and the deep network can improve the comprehensiveness and accuracy of the features, thereby ensuring a better deblurring effect of the image. Moreover, this process only acts on the reference image, with low power consumption and low data processing resources.

[0057] In an optional embodiment, the angle of spatial alignment and superposition of the first fused image in the image fusion process is described. The deblurred reference image and multiple frames of the first image are fused to obtain the first fused image, including: spatially aligning and superimposing multiple frames of the first image according to the deblurred reference image to obtain a superimposed image; generating the first fused image based on the superimposed image. The superimposed image can be used as the first fused image; the superimposed image can also be subjected to noise reduction processing to obtain the first fused image; the superimposed image can also be input into a neural network model for processing to output the first fused image.

[0058] Optionally, based on the deblurred reference image, multiple frames of first images are spatially aligned and superimposed to obtain a superimposed image, including: spatially aligning the multiple frames of first images with the deblurred reference image to obtain spatially aligned multiple frames of first images; superimposing the spatially aligned multiple frames of first images onto the reference image to obtain a superimposed image.

[0059] In an optional embodiment, a superposition processing method of the first fused image in the image fusion process is described. The deblurred reference image and the multiple frames of the first image are fused to obtain the first fused image, including: respectively calculating the local area similarity of the multiple frames of the first image relative to the reference image; and superimposing the local areas of the multiple frames of the first image on the reference image according to the local area similarity to obtain the first fused image.

[0060] Optionally, superimposing the local regions of the multiple frames of the first image onto the reference image based on the local region similarity includes: for the multiple frames of the first image, superimposing the local regions whose local region similarity is less than or equal to the similarity threshold onto the corresponding local regions in the reference image according to the spatial position correspondence. Thus, even if there is an error in the spatial alignment of the local regions due to the presence of a moving region, the local regions with too large an error can be avoided from being superimposed onto the same position on the reference image based on the local region similarity, so as to avoid too large an error.

[0061] Optionally, the methods for calculating the local area similarity include but are not limited to: error square sum algorithm (SSD), mean error square sum algorithm (MSD), normalized product correlation algorithm (NCC), sequential similarity detection algorithm (SSDA), Hadamard transform algorithm (SATD), absolute difference sum algorithm (SAD); taking the absolute difference sum algorithm as an example, its expression is as follows:

[0062]

[0063] Among them, SAD represents the sum of absolute errors of pixel values ​​in a local area; 7*7 represents the specifications of the local area, that is, 7 coordinate unit ranges under different dimensions; i represents a spatial position dimension in the image; j represents another spatial position dimension in the image; image1 represents the first image of a frame for absolute difference and calculation, image2 represents the reference image, image1i,j represents the pixel value at the (i,j)th position in the first image of a frame for absolute difference and calculation, image2i,j represents the pixel value at the (i,j)th position in the reference image; optionally, (i,j) is the pixel point pointed to by the i-th unit distance on the x-axis and the j-th unit distance on the y-axis.

[0064] In one embodiment, multiple frames of spatially aligned first images are superimposed on a reference image to obtain a superimposed image, including: in the multiple frames of spatially aligned first images, pixel values ​​of pixels at matching spatial positions are averaged to obtain averaged pixel values ​​of each pixel; according to the spatial position of each pixel, the pixel value of each pixel of the reference image is adjusted by the averaged pixel value of each pixel. The spatial position can be determined by the pixel coordinates of the pixel in the image, the grid area of ​​the image where the pixel is located, etc.; matching spatial positions refer to the pixel positions in different frames of first images being the same or corresponding. For example, pixel A is at position X1 in the A-th frame first image, and pixel B is at position X2 in the B-th frame first image, and the X2 position can be converted to the X1 position based on the K function used to calculate the matching spatial position, then pixel A and pixel B are pixels at matching spatial positions.

[0065] Step 208: Fuse the first fused image and the second image to obtain a second fused image.

[0066] The second fused image is an image obtained by improving the quality of the first fused image through the second image. Optionally, since the brightness of the first image is greater than the brightness of the second image, and the first fused image is obtained by fusion of the first images, when the first fused image and the second image are fused, information of different brightness can be complementary, so that the dynamic range of the second fused image is improved.

[0067] In one embodiment, fusing the first fused image and the second image to obtain the second fused image includes: performing weighted calculation of pixel values ​​on pixels or local areas of the first fused image and the second image to obtain the second fused image.

[0068] In one embodiment, the process of fusing the first fused image and the second image belongs to the processing process of high dynamic range imaging, which includes two steps: image alignment and image fusion. The image alignment method is consistent with the above. The image fusion adopts a pyramid fusion strategy, that is, the input image is first decomposed into a Laplacian pyramid, and then at each layer, several frames of input images are weighted averaged, and then the pyramid is reconstructed to obtain the second fused image.

[0069] The above-mentioned image processing method obtains multiple frames of first images and at least one frame of second images; the brightness of the first image is greater than the brightness of the second image, so that the information represented by the first image and the second image in different image regions is different, providing different information dimensions; determining a reference image from multiple frames of first images; deblurring the reference image, deblurring the reference image, and fusing the deblurred reference image and multiple frames of first images to obtain a first fused image; thus, a reference image is selected from multiple frames of first images, and multiple frames of first images are fused based on the deblurred reference image as a reference, without deblurring each frame of the first image, and in this way, a higher image quality is guaranteed, the time consumption is short, and the power consumption loss is low; the first fused image and the second image are fused to obtain a second fused image, and the information represented by the second image in certain regions can not only improve the quality of the first fused image again, but also has the advantages of short time consumption and low power consumption loss. Thus, when the second fused image is of good quality, the time consumption is short and the power consumption loss is low.

[0070] In one embodiment, determining a reference image from a plurality of first image frames includes: determining a first image that satisfies an acquisition order condition from a plurality of first image frames as a reference image; or determining a first image that satisfies a clarity condition from a plurality of first image frames as a reference image.

[0071] The acquisition order condition is the consistency requirement between the shooting result and the image displayed at the acquisition time point; according to the acquisition order of a certain frame image in multiple frames of first images when each frame of the first image is acquired, the conditions satisfied by the first image can be determined. Optionally, the acquisition order is related to the image displayed at the acquisition time point, and the later the first image in the acquisition order is, the higher the consistency is with the image displayed at the acquisition time point. Among them, the shooting result can be the second fused image, or it can be an image generated based on the second fused image; the acquisition time point is the time point when the shutter is opened, and the image displayed at the acquisition time point is the image displayed to the user when the shutter is opened.

[0072] The clarity condition is used to characterize the clarity requirements of the first image of each frame. The clarity comparison result obtained by comparing the clarity of multiple frames of first images can be used to screen out the first image with the best clarity as the reference image. A first image of a certain frame can also be screened out as the reference image through the first image whose clarity exceeds the clarity threshold.

[0073] Optionally, the clarity of the first image may be calculated using an operator for calculating clarity, where the operator includes but is not limited to a Sobel Operator and a Laplace Operator.

[0074] Taking the Sobel operator as an example, the first image of each frame can be convolved by Sobel operators in different directions to obtain the convolution results in each direction; then the convolution results in each direction are superimposed by splicing and other directions to obtain the clarity of the first image. Optionally, the Sobel operator is a high-pass filter that extracts the high-frequency details of the image. Generally, the clearer the image, the more high-frequency details; the Sobel operators in different directions can be horizontal and vertical, because some textures are horizontal and some are vertical. Of course, there are textures in other directions, even irregular textures; illustratively, the horizontal and vertical Sobel convolution operators are as follows:

[0075]

[0076] In one embodiment, determining a first image satisfying an acquisition sequence condition from among multiple frames of first images as a reference image includes: determining a first image acquired last from among multiple frames of first images as a reference image according to an acquisition sequence.

[0077] In one embodiment, determining a first image satisfying a definition condition as a reference image from among multiple frames of first images includes: determining a first image having the highest definition as the reference image from among multiple frames of first images according to the definition.

[0078] In a feasible embodiment, determining the reference image from the multiple frames of first images includes: determining the first image that meets the acquisition sequence condition from the multiple frames of first images; if the clarity of the first image that meets the acquisition sequence condition reaches a certain image clarity critical value, then using the first image that meets the acquisition sequence condition as the reference image; if the clarity of the first image that meets the acquisition sequence condition does not reach the image clarity critical value, then using the first image with the highest clarity as the reference image. Thus, the reference image can be determined in combination with different methods, and the image displayed at the image acquisition time point is preferentially used as the reference image. In the case where the clarity of the first image that meets the acquisition sequence condition is not good, the user cannot determine the image displayed at the acquisition time point, and instead uses the first image with the highest clarity, so as to acquire the image with the best clarity under the user's perception.

[0079] Determining the first image that meets the acquisition order condition as the reference image can make a certain correspondence between the shooting result and the image displayed by the camera, so that the shooting result is consistent with the image seen by the human eye when the image is taken. Determining the first image that meets the clarity condition as the reference image can make the shooting result the clearest.

[0080] In one embodiment, a reference image is deblurred, and the deblurred reference image and multiple frames of first images are fused to obtain a first fused image, including: based on the deblurred reference image, spatially aligning the multiple frames of first images to obtain multiple frames of spatially aligned first images; and superimposing the multiple frames of spatially aligned first images onto the deblurred reference image to obtain the first fused image.

[0081] The multiple first images after spatial domain alignment are the first images that have a pixel position correspondence relationship with the deblurred reference image; compared with the state where the multiple first images are not spatially aligned, the multiple first images after spatial domain alignment are spatially aligned for the deblurred reference image, which can improve the clarity of the multiple first images. Optionally, the multiple first images are acquired at different time points during an image acquisition process, and there may be position deviations at the pixel level. The multiple first images after spatial domain alignment establish a spatial position correspondence relationship, and can calculate pixel values ​​at the pixel level with the deblurred reference image.

[0082] In one embodiment, based on the deblurred reference image, multiple frames of first images are spatially aligned to obtain multiple frames of first images after spatial alignment, including: determining reference image blocks divided by the deblurred reference image, and image blocks to be aligned divided by the multiple first images respectively; based on feature point pairs between the reference image blocks and the image blocks to be aligned, determining the block space alignment mapping relationship between the reference image blocks and the image blocks to be aligned; based on the block space alignment mapping relationship, aligning the image blocks to be aligned with the reference image blocks to obtain multiple first images after spatial alignment.

[0083] Optionally, both the reference image block and the image block to be aligned are divided according to a certain specification, which may be an image block determined by pixel width and length, and the image block may be a pixel block (patch); for example, both the reference image block and the image block to be aligned are 4*4 pixel blocks. The feature point pairs include feature points in the reference image block and feature points in the image block to be aligned, and these feature points may be FPM feature points, Harris corner points, or other types. Optionally, the block space alignment mapping relationship may be a pixel change matrix between the reference image block and the image block to be aligned.

[0084] In an exemplary embodiment, multiple frames of first images after spatial alignment are superimposed on a deblurred reference image to obtain a first fused image, including: respectively calculating the local area similarity of the multiple frames of first images relative to the reference image; and superimposing local areas of the multiple frames of first images on the reference image according to the local area similarity to obtain the first fused image.

[0085] The method of superimposing the plurality of spatially aligned first images onto the deblurred reference image to obtain a fused first image includes: determining the similarity of the plurality of spatially aligned expected exposure images in local areas; and superimposing the local areas where the similarity is less than a superposition threshold onto the deblurred reference image to obtain a fused first image.

[0086] Therefore, spatial alignment of multiple frames of first images is first performed based on the deblurred reference image, so that the pixels of the multiple frames of first images can be adjusted in spatial position, thereby improving the clarity of the multiple frames of first images through the spatial alignment process, and reducing the time and power consumption of data processing; on this basis, the multiple frames of first images after spatial alignment are superimposed on the deblurred reference image to obtain a first fused image, and noise reduction in the time domain is achieved in this way, so that the clarity of the first fused image is further improved.

[0087] In one embodiment, the first fused image and the second image are fused to obtain the second fused image, including: deblurring at least one frame of the second image to obtain the deblurred second image; and performing high dynamic range imaging fusion on the first fused image and the deblurred second image to obtain the second fused image.

[0088] The deblurred second image is repaired based on the pixels of the second image to improve the clarity and details. The denoised second image is obtained by spatial denoising; optionally, based on the trade-off between the clarity and noise of the second image, the second image is subjected to spatial denoising to obtain the denoised second image.

[0089] High dynamic range imaging fusion is an image fusion method used to generate high dynamic range (HDR) images; the second fused image generated by the high dynamic range imaging fusion method can complement the brightness information of the first image and the second image respectively, so that the color expression is further improved, so that it can express corresponding information in both the highlight area and the dark area.

[0090] Optionally, the second image and the reference image may be deblurred in the same manner; correspondingly, deblurring at least one frame of the second image to obtain a deblurred second image includes: inputting the second image into a deblurring model; deblurring the second image through the deblurring model, and outputting the deblurred second image. Thus, by processing the second image through the deblurring model, the clarity of the second image can be improved.

[0091] Optionally, the second image can be deblurred by a dynamic vision sensor deblurring algorithm. In this case, the dynamic vision sensor deblurring algorithm uses a dynamic vision sensor deblurring model for image processing, and uses the second image as the image to be processed in the dynamic vision sensor deblurring model; correspondingly, the second image is deblurred by the deblurring model, and the deblurred second image is output, including: performing multiple feature extractions on the image to be processed by a multi-layer downsampling network in the segmentation model to obtain a feature vector of each layer of the downsampling network; obtaining a corresponding reference feature vector based on the feature vector of each layer of the network, and performing feature fusion of the reference feature vector and the feature vector through a multi-layer upsampling network to obtain the category of each pixel. Among them, since the second image can be one or two frames of images, the deblurring process has low power consumption and short time consumption.

[0092] Deblurring the second image can improve image clarity, and since the second image can be one or two frames, this deblurring process has the advantages of low power consumption and short time. Finally, high dynamic range imaging fusion is performed on the first fused image and the deblurred second image, which can improve the dynamic range of the second fused image with short time and low power loss.

[0093] In one embodiment, deblurring is performed on at least one frame of the second image to obtain a deblurred second image, including: determining an image block in the second image; filtering out a target image block from the image blocks in the second image according to the brightness of the image block and a motion event; deblurring the target image block in the second image to obtain a deblurred second image; and fusing the first fused image and the deblurred second image to obtain a second fused image.

[0094] The image blocks in the second image are image blocks that make up the second image; each image block in the second image has its own image block brightness and motion event status. The image block brightness is the brightness of the image block in the second image, and the motion event status is obtained by analyzing the motion events detected in the second image during the exposure process. The target image block is the image block in the second image that is used for deblurring processing.

[0095] In one embodiment, determining the image blocks in the second image includes: dividing the second image into image blocks of a certain size; wherein the size can be represented by a certain pixel resolution, for example, each image block has a pixel resolution of 256×256.

[0096] In one embodiment, according to the brightness of the image blocks and the motion event situation, the target image blocks are filtered out from the image blocks in the second image, including: according to the brightness of the image blocks and the motion event situation, the image blocks in the second image are filtered to obtain filtered image blocks; the filtered image blocks are the target image blocks filtered out from the image blocks in the second image.

[0097] In one embodiment, according to the brightness and motion blur of the image blocks, the target image blocks are screened out from the image blocks in the second image, including: for the image blocks in the second image, according to the brightness and motion blur, determining the image blocks that meet the deblurring conditions as the target image blocks.

[0098] In one embodiment, a target image block in a second image is deblurred to obtain a deblurred second image, including: inputting the second image into a deblurring model of the second image for deblurring to deblur the target image block to obtain a deblurred second image; wherein the deblurring model of the second image can still be deblurred based on a dynamic vision sensor deblurring algorithm and used to perform image processing on the target image block.

[0099] According to the brightness and motion events of the image blocks, the target image blocks are screened to perform deblurring on the second image, which can reduce the number of image blocks to be deblurred, thereby reducing the time consumption and power consumption of the deblurring process. Therefore, the first fused image and the deblurred second image are fused to make the second fused image clearer, with shorter time consumption and lower power consumption.

[0100] In one embodiment, the deblurring condition is described. Correspondingly, according to the brightness of the image block and the motion event situation, the target image block of the image block in the second image is determined, including: in the second image, determining that the image block with brightness greater than or equal to the brightness threshold is the target image block; or, in the second image, determining that the image block with a motion event amount greater than or equal to the motion event amount threshold is the target image block.

[0101] Optionally, the target image block includes at least one of a highlight area image block and a blurred image block in the second image; the highlight area image block represents more information, and deblurring the highlight area image block can improve the clarity of the highlight area image block to represent more information; the information represented by the motion event situation is more effective and may contain information that the user expects to obtain, so the blurred image block is deblurred.

[0102] The brightness threshold is the critical value of the brightness of the image block, and is used to divide the image blocks in the second image into target image blocks and non-deblurred image blocks outside the target image blocks according to the brightness. In particular, since the information represented by the second image is mainly for the highlight area of ​​the first image or the highlight area of ​​the first fused image, and the non-deblurred image blocks belong to the dark area image blocks of the first image, the information represented by the second image is not set for these dark area image blocks; the non-deblurred image blocks are image blocks that are not deblurred. Optionally, the brightness of the image and the brightness threshold can both be represented by the pixel value of the image block, which can be the average pixel value of the image block to evaluate the brightness of the image block.

[0103] The motion event amount threshold is a critical value of the motion event amount of the image block, and is used to divide the image blocks in the second image into target image blocks and non-deblurred image blocks other than the target image blocks according to the motion event amount. Optionally, the image blocks with a motion event amount greater than or equal to the motion event amount threshold have a higher effective event amount, and the image blocks may be blurred due to motion, so deblurring processing is required; correspondingly, the image blocks with a motion event amount less than the motion event amount threshold have a lower effective event amount, and the image blocks are not blurred due to motion, so deblurring processing is not required.

[0104] Optionally, the motion event quantity threshold is determined based on the background noise of the pixels covered by the dynamic vision sensor. The background noise is related to the illumination. The background noise under different illuminations is different, and the illumination can be determined by an illuminance meter. Optionally, the illumination can be obtained by an illuminance meter, and then the background noise under the illumination is determined, so as to dynamically adjust the motion event quantity threshold based on the illumination, so that the evaluation standard of the effective event quantity is more accurate. Optionally, the background noise N_noise of the pixels of the dynamic vision sensor under different illuminations already exists, and the background noise is adjusted according to the adjustment parameter (such as 1.5 times), and the motion event quantity threshold can be obtained.

[0105] Exemplarily, the second image is first divided into multiple image blocks, each image block in the second image has a pixel resolution of approximately 256 times 256, and then its average pixel value P_value is calculated, and at the same time, the number of DVS events N_event of the image block in the entire RGB exposure process is counted. As long as the image block in the second image meets any of the following two conditions, the image block will not be processed by the DVS deblurring algorithm, saving power consumption and processing time: 1. P_value is less than the brightness threshold, 2. N_event<1.5 times N_noise.

[0106] In this embodiment, the image blocks with brightness greater than or equal to the brightness threshold can better represent the information missing in the first image, so these image blocks are used as target image blocks for deblurring, and the image blocks with motion event volume greater than or equal to the motion event volume threshold belong to the image blocks with a high effective event volume detected by the dynamic vision sensor. Therefore, the information represented by these image blocks can better compensate for the missing information of the first fused image, thereby making the clarity of the second fused image better.

[0107] In one embodiment, high dynamic range imaging fusion is performed on the first fused image and the deblurred second image to obtain the second fused image, including: denoising the first fused image to obtain the first denoised fused image; denoising the deblurred second image to obtain the second denoised image; and high dynamic range imaging fusion is performed on the denoised first image and the denoised second image to obtain the second fused image.

[0108] The first fused image after denoising is obtained by spatial denoising; optionally, based on the trade-off between the clarity and noise of the first fused image, the first fused image is subjected to spatial denoising to obtain the first fused image after denoising.

[0109] Optionally, the first fused image and the deblurred second image are subjected to denoising by performing spatial denoising of a single frame respectively, and each central pixel is subjected to denoising by using its neighboring pixels to ensure image noise reduction; wherein the central pixel is a pixel point having adjacent pixels in space, and the adjacent pixels may be four-neighborhood pixels or eight-neighborhood pixels at a distance of one pixel, or other neighboring pixels. Optionally, spatial denoising may be performed on each frame of the image by using a neural network model for single-frame denoising.

[0110] The first fused image is subjected to noise reduction processing, which mainly involves noise reduction processing in the spatial domain, and the noise reduction processing has low power consumption and short time consumption; the second image after deblurring is subjected to noise reduction processing, which can further improve the clarity of the second image, and because the second image can be one frame or two frames of images, the noise reduction processing has the advantages of low power consumption and short time consumption; finally, the first image after noise reduction and the second image after noise reduction are subjected to high dynamic range imaging fusion, which can improve the dynamic range of the second fused image with short time consumption and low power consumption loss.

[0111] In one embodiment, at least one frame of the second image includes a first exposure image and a second exposure image, and the brightness of the first exposure image is greater than the brightness of the second exposure image. The first exposure image and the second exposure image are two different frames of second images, and the two are used to characterize images of different brightness. Optionally, the brightness of the first exposure image and the second exposure image is determined based on the exposure amount; optionally, the exposure amount relationship between the first exposure image and the second exposure image can be adjusted according to the shooting scene. Optionally, the backlight degree can be determined by statistically analyzing the distribution of the image histogram, so as to determine the shooting scene according to the backlight degree, and then determine that the exposure amount of the first exposure image is the exposure amount relationship between the second exposure images. Optionally, the exposure amount relationship is that the exposure amount of the first exposure image is 2N times the exposure amount of the second exposure image, N is determined based on the shooting scene, and the shooting scene is determined based on the backlight degree. Exemplarily, for a scene with strong backlight, n needs to be relatively large, such as 4, while for a scene with weak backlight, n may be 1.

[0112] The first fused image and the second image are fused to obtain a second fused image, including: fusing the first fused image, the first exposed image, and the second exposed image to obtain the second fused image.

[0113] In one embodiment, acquiring multiple frames of first images and at least one frame of second image includes: acquiring a first exposure image, a second exposure image and multiple frames of first images.

[0114] Optionally, the first fused image, the first exposure image and the second exposure image are fused to obtain the second fused image, including: performing deblurring processing on the first exposure image and the second exposure image respectively to obtain a deblurred first exposure image and a deblurred second exposure image; performing high dynamic range imaging fusion on the first fused image, the deblurred first exposure image and the deblurred second exposure image to obtain the second fused image.

[0115] Optionally, high dynamic range imaging fusion is performed on the first fused image, the deblurred first exposure image and the deblurred second exposure image to obtain a second fused image, including: performing denoising processing on the first fused image to obtain a denoised first fused image; performing denoising processing on the deblurred first exposure image and the deblurred second exposure image respectively to obtain a denoised first exposure image and a denoised second exposure image; and performing high dynamic range imaging fusion on the denoised first image, the denoised first exposure image and the denoised second exposure image to obtain a second fused image.

[0116] Optionally, the first exposure image and the second exposure image are respectively deblurred to obtain the deblurred first exposure image and the deblurred second exposure image, including: respectively determining an image block in the first exposure image and an image block in the second exposure image; filtering out a target image block in the first exposure image and a target image block in the second exposure image from the image blocks in the first exposure image and the image blocks in the second exposure image according to the brightness of the image blocks and the motion event conditions; and respectively deblurring the target image block in the first exposure image and the target image block in the second exposure image to obtain the deblurred first exposure image and the deblurred second exposure image.

[0117] Optionally, according to the brightness of the image blocks and the motion events, the target image blocks in the first exposure image and the target image blocks in the second exposure image are respectively screened out from the image blocks in the first exposure image and the image blocks in the second exposure image, including: in the first exposure image, determining that the image blocks with brightness greater than or equal to the brightness threshold are the target image blocks in the first exposure image; or, in the first exposure image, determining that the image blocks with motion event amounts greater than or equal to the motion event amount threshold are the target image blocks in the first exposure image.

[0118] Optionally, according to the brightness of the image blocks and the motion events, the target image blocks in the first exposure image and the target image blocks in the second exposure image are respectively screened out from the image blocks in the first exposure image and the image blocks in the second exposure image, including: in the second exposure image, determining that the image blocks with brightness greater than or equal to a brightness threshold are the target image blocks in the second exposure image; or, in the second exposure image, determining that the image blocks with a motion event amount greater than or equal to a motion event amount threshold are the target image blocks in the second exposure image.

[0119] When the second image includes the first exposure image and the second exposure image, the second fused image can be made to have higher quality, ensuring better clarity of the first exposure image and the second exposure image, and a higher dynamic range of the second fused image; at the same time, the first exposure image and the second exposure image are subjected to spatial noise reduction processing, which takes a short time and has low power consumption loss.

[0120] In an exemplary embodiment, Figure 4As shown in the figure, the algorithm architecture includes three branches from top to bottom: the first branch first collects multiple frames of equal exposure 0EV first images, selects only the reference frame image for deblurring, and then performs spatial alignment of multiple frames of first images based on the deblurred reference frame image, and then superimposes multiple frames of first images. The superposition process has the effect of temporal denoising, and finally performs AI denoising on the synthesized first fused image to achieve spatial denoising and obtain the first fused image after denoising; the processing of the second image includes the first exposure image processing of the second branch and the second exposure image processing of the third branch. The second and third branches are both based on the dynamic visual sensor deblurring algorithm. The difference from the first branch is that the processing of the second image only processes image blocks with brightness and event volume reaching a certain level, and then performs single-frame AI denoising to obtain the first exposure image after denoising and the second exposure image after denoising. After the three branches are processed, three frames of high dynamic range imaging fusion are performed to obtain the second fused image.

[0121] Therefore, the dynamic vision sensor deblurring algorithm is embedded in the image algorithm for multi-frame processing, avoiding deblurring for each frame, which saves time and reduces power consumption. Secondly, frame selection is performed on multiple frames of equally exposed images instead of deblurring all frames, which saves time and reduces power consumption. Furthermore, the brightness and event volume of the reduced exposure frames are counted to avoid image blocks that do not need to be processed, which saves time and reduces power consumption.

[0122] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0123] Based on the same inventive concept, the embodiment of the present application also provides an image processing device for implementing the above-mentioned image processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image processing device embodiments provided below can refer to the limitations on the image processing method above, and will not be repeated here.

[0124] In one embodiment, Figure 5 As shown, an image processing device is provided, comprising:

[0125] An acquisition module, used to acquire multiple frames of first images and at least one frame of second image; the brightness of the first image is greater than the brightness of the second image;

[0126] A screening module 502, configured to determine a reference image from the plurality of first image frames;

[0127] A first fusion module 504 is used to perform a deblurring process on the reference image, and fuse the deblurred reference image with the multiple frames of first images to obtain a first fused image;

[0128] The second fusion module 506 is configured to fuse the first fused image and the second image to obtain a second fused image.

[0129] In one embodiment, the screening module 502 is used to:

[0130] Determine, from the multiple frames of first images, a first image that meets the acquisition sequence condition as a reference image; or,

[0131] From the multiple frames of first images, a first image that meets a definition condition is determined as a reference image.

[0132] In one embodiment, the first fusion module 504 is used to:

[0133] Based on the deblurred reference image, spatially aligning the multiple frames of first images to obtain the multiple frames of first images after spatial alignment;

[0134] The multiple frames of first images after the spatial domain alignment are superimposed on the deblurred reference image to obtain a first fused image.

[0135] In one embodiment, the second fusion module 506 is used to:

[0136] Performing a deblurring process on the at least one frame of the second image to obtain a deblurred second image;

[0137] The first fused image and the deblurred second image are fused by high dynamic range imaging to obtain a second fused image.

[0138] In one embodiment, the second fusion module 506 is used to:

[0139] determining an image block in the second image;

[0140] Filtering out target image blocks from image blocks in the second image according to image block brightness and motion event conditions;

[0141] Deblurring the target image block in the second image to obtain a deblurred second image;

[0142] The first fused image and the deblurred second image are fused to obtain a second fused image.

[0143] In one embodiment, the second fusion module 506 is used to:

[0144] In the second image, an image block whose brightness is greater than or equal to a brightness threshold is determined as a target image block; or,

[0145] In the second image, the image block whose motion event amount is greater than or equal to a motion event amount threshold is determined as a target image block.

[0146] In one embodiment, the second fusion module 506 is used to:

[0147] Performing noise reduction processing on the first fused image to obtain a first fused image after noise reduction;

[0148] Performing noise reduction processing on the deblurred second image to obtain a noise-reduced second image;

[0149] High dynamic range imaging fusion is performed on the first image after noise reduction and the second image after noise reduction to obtain a second fused image.

[0150] In one embodiment, the at least one frame of the second image includes a first exposure image and a second exposure image, and the brightness of the first exposure image is greater than the brightness of the second exposure image;

[0151] The second fusion module 506 is used to:

[0152] The first fused image, the first exposed image, and the second exposed image are fused to obtain a second fused image.

[0153] Each module in the above image processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0154] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an image processing method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0155] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0156] The embodiment of the present application further provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, enable the processors to perform the steps of the image processing method.

[0157] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the image processing method.

[0158] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards.

[0159] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0160] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0161] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: include: Acquire multiple frames of first image and at least one frame of second image; The brightness of the first image is greater than the brightness of the second image; Determine a reference image from the plurality of frames of first images; Deblurring the reference image, and fusing the deblurred reference image with the multiple frames of first images to obtain a first fused image; The first fused image and the second image are fused to obtain a second fused image.

2. The method according to claim 1, characterized in that The determining of the reference image from the multiple frames of first images comprises: Determine, from the multiple frames of first images, a first image that meets the acquisition sequence condition as a reference image; or, From the multiple frames of first images, a first image that meets a definition condition is determined as a reference image.

3. The method according to claim 1, characterized in that The deblurring the reference image, and fusing the deblurred reference image with the multiple frames of first images to obtain a first fused image, comprises: Based on the deblurred reference image, spatially aligning the multiple frames of first images to obtain the multiple frames of first images after spatial alignment; The multiple frames of first images after the spatial domain alignment are superimposed on the deblurred reference image to obtain a first fused image.

4. The method according to claim 1, characterized in that: The fusing the first fused image and the second image to obtain a second fused image includes: Performing a deblurring process on the at least one frame of the second image to obtain a deblurred second image; The first fused image and the deblurred second image are fused by high dynamic range imaging to obtain a second fused image.

5. The method according to claim 4, characterized in that The deblurring process is performed on the at least one frame of the second image to obtain a deblurred second image, comprising: determining an image block in the second image; Filtering out target image blocks from image blocks in the second image according to image block brightness and motion event conditions; Deblurring the target image block in the second image to obtain a deblurred second image.

6. The method according to claim 5, characterized in that The step of determining the target image block of the image block in the second image according to the image block brightness and the motion event situation includes: In the second image, an image block whose brightness is greater than or equal to a brightness threshold is determined as a target image block; or, In the second image, the image block whose motion event amount is greater than or equal to a motion event amount threshold is determined as a target image block.

7. The method according to claim 4, characterized in that The step of performing high dynamic range imaging fusion on the first fused image and the deblurred second image to obtain a second fused image includes: Performing noise reduction processing on the first fused image to obtain a first fused image after noise reduction; Performing noise reduction processing on the deblurred second image to obtain a noise-reduced second image; High dynamic range imaging fusion is performed on the first image after noise reduction and the second image after noise reduction to obtain a second fused image.

8. The method according to claim 1, characterized in that The at least one frame of the second image includes a first exposure image and a second exposure image, and the brightness of the first exposure image is greater than the brightness of the second exposure image; The fusing the first fused image and the second image to obtain a second fused image includes: The first fused image, the first exposed image, and the second exposed image are fused to obtain a second fused image.

9. An image processing device, characterized in that: include: An acquisition module, used for acquiring a plurality of frames of first images and at least one frame of second image; The brightness of the first image is greater than the brightness of the second image; A screening module, used for determining a reference image from the plurality of frames of first images; A first fusion module is used to perform a deblurring process on the reference image, and fuse the deblurred reference image with the multiple frames of first images to obtain a first fused image; The second fusion module is used to fuse the first fused image and the second image to obtain a second fused image.

10. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the computer program is executed by the processor, the processor is caused to perform the steps of the image processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images