Image processing method and electronic device

Abstract

This application discloses an image processing method and electronic device, belonging to the field of image processing technology. The specific solution includes: acquiring long-exposure image frames and short-exposure image frames; performing exposure normalization processing on the long-exposure image frames and the short-exposure image frames; performing motion alignment processing on the exposure-normalized long-exposure image frames to obtain a first image, and performing motion alignment processing on the exposure-normalized short-exposure image frames to obtain a second image; and inputting the first and second images into a feature fusion model for feature fusion processing to obtain a target image; wherein the feature fusion model is used to implement image feature extraction and image feature fusion.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to an image processing method and electronic equipment. Background Art

[0002] In a motion shooting scene, since the subject is in motion, it is easy for the captured image to produce motion blur.

[0003] In related technologies, two frames of images with different exposure times can be obtained first, and then optical flow estimation is performed based on the brightness and color of the two frames of images to obtain an optical flow map. Then, motion alignment of the images is achieved based on the optical flow map, and finally the final image is obtained through algorithm fusion.

[0004] However, since there are certain differences in brightness and color between two frames of images with different exposure times, the optical flow map may not be accurate and the motion alignment effect is poor, resulting in ghosting distortion in the final motion image. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide an image processing method and electronic device that can solve the problem of ghosting distortion in moving images.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, which includes: acquiring a long-exposure image frame and a short-exposure image frame; performing exposure normalization processing on the long-exposure image frame and the short-exposure image frame; performing motion alignment processing on the long-exposure image frame after the exposure normalization processing to obtain a first image, and performing motion alignment processing on the short-exposure image frame after the exposure normalization processing to obtain a second image; inputting the first image and the second image into a feature fusion model for feature fusion processing to obtain a target image; wherein the feature fusion model is used to realize image feature extraction and image feature fusion.

[0007] In a second aspect, an embodiment of the present application provides an image processing device, comprising: an acquisition module and a processing module; the acquisition module is used to acquire long-exposure image frames and short-exposure image frames; the processing module is used to perform exposure normalization processing on the long-exposure image frames and the short-exposure image frames; motion alignment processing is performed on the long-exposure image frames after exposure normalization processing to obtain a first image, and motion alignment processing is performed on the short-exposure image frames after exposure normalization processing to obtain a second image; the first image and the second image are input into a feature fusion model for feature fusion processing to obtain a target image; wherein the feature fusion model is used to realize image feature extraction and image feature fusion.

[0008] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method described in the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the method described in the first aspect.

[0012] In an embodiment of the present application, a long-exposure image frame and a short-exposure image frame can be obtained; exposure normalization processing is performed on the long-exposure image frame and the short-exposure image frame; motion alignment processing is performed on the long-exposure image frame after exposure normalization processing to obtain a first image, and motion alignment processing is performed on the short-exposure image frame after exposure normalization processing to obtain a second image; the first image and the second image are input into a feature fusion model for feature fusion processing to obtain a target image; wherein the feature fusion model is used to implement image feature extraction and image feature fusion. Through this solution, since the long-exposure image frame and the short-exposure image frame are first subjected to exposure normalization processing before the motion alignment processing is performed, the visual differences caused by the different exposure times of the long-exposure image frame and the short-exposure image frame can be eliminated, thereby obtaining a more accurate optical flow estimation result, thereby improving the motion alignment effect and alleviating the ghosting distortion phenomenon in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is one of the flowcharts of the image processing method provided in the embodiment of the present application;

[0014] Figure 2 This is one of the structural diagrams of the feature fusion model provided in the embodiment of the present application;

[0015] Figure 3 This is the second flowchart of the image processing method provided in the embodiment of the present application;

[0016] Figure 4 This is the second structural diagram of the feature fusion model provided in the embodiment of the present application;

[0017] Figure 5 is a structural diagram of an image processing device provided in an embodiment of the present application;

[0018] Figure 6 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0019] Figure 7 This is a hardware diagram of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field fall within the scope of protection of this application.

[0021] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0022] The image processing method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0023] The image processing method provided in the embodiments of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the image processing method. The electronic devices mentioned in the embodiments of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, wearable devices, etc. The image processing method provided in the embodiments of the present application is described below using an electronic device as an example of the execution subject.

[0024] like Figure 1 As shown, the embodiment of the present application provides an image processing method, which may include steps 101 to 104:

[0025] Step 101: Acquire a long-exposure image frame and a short-exposure image frame.

[0026] When an electronic device photographs a moving object, the electronic device may photograph a long-exposure image frame and a short-exposure image frame through a camera in a motion photographing mode.

[0027] Step 102: Perform exposure normalization processing on the long-exposure image frame and the short-exposure image frame.

[0028] Optionally, the exposure normalization processing includes global normalization processing. The electronic device may perform intrinsic decomposition processing on the long-exposure image frame to obtain a first luminance image, and perform intrinsic decomposition processing on the short-exposure image frame to obtain a second luminance image; then perform global normalization processing on the first luminance image based on the first exposure time of the first luminance image to obtain a third luminance image, and perform global normalization processing on the second luminance image based on the second exposure time of the second luminance image to obtain a fourth luminance image.

[0029] Specifically, the electronic device processes the long exposure image frame I L0 Performing intrinsic decomposition processing can obtain the first brightness image I L1 , by short exposure image frame I S0 Performing intrinsic decomposition processing can obtain the second brightness image I S1 , where the long exposure image frame I L0 The corresponding exposure time is the first exposure time t1, and the short exposure image frame I S0 The corresponding exposure time is the second exposure time t2. L1 The third brightness image I can be obtained by dividing each pixel value of the first exposure time t1 L2 , the second brightness image I S1 The fourth brightness image I can be obtained by dividing each pixel value of S2 .

[0030] Based on the above scheme, long-exposure image frames and short-exposure image frames can be normalized through global normalization. Since different exposure times can change the imaging brightness and color of the image, dividing by the exposure time can eliminate the optical flow alignment error caused by different brightness.

[0031] Optionally, the exposure normalization processing further includes local normalization processing. The electronic device may perform local normalization processing based on the eight-neighborhood mean and eight-neighborhood standard deviation of each pixel in the third luminance image to obtain the long-exposure image frame after exposure normalization processing; and perform local normalization processing based on the eight-neighborhood mean and eight-neighborhood standard deviation of each pixel in the fourth luminance image to obtain the short-exposure image frame after exposure normalization processing.

[0032] Specifically, the electronic device may perform local normalization processing on the third luminance image and the fourth luminance image according to formula (1):

[0033]

[0034] Where f′(i) is the pixel i in the third luminance image or the fourth luminance image, u(i) is the mean of the eight neighborhoods of pixel i, σ(i) is the standard deviation of the eight neighborhoods of pixel i, and ε is an integer used to maintain the stability of formula (1).

[0035] Optionally, in the embodiment of the present application, the value of ε may be 0.5.

[0036] Based on the above scheme, exposure normalization can be performed on long-exposure image frames and short-exposure image frames through local normalization. Since optical flow estimation is calculated by matching key pixel points of long-exposure image frames and short-exposure image frames, local normalization can enhance the contrast of image texture information, thereby reducing the number of similar key points and improving pixel matching accuracy.

[0037] Step 103 : performing motion alignment processing on the long-exposure image frames after the exposure normalization processing to obtain a first image, and performing motion alignment processing on the short-exposure image frames after the exposure normalization processing to obtain a second image.

[0038] Specifically, the electronic device may first perform optical flow estimation on the long exposure image frame and the short exposure image frame after exposure normalization processing to obtain an optical flow map I f , then based on the optical flow graph I f , for the long exposure image frame I L0 Perform motion alignment processing to obtain the first image I L3 , for the short exposure image frame I S0 Perform motion alignment processing to obtain the second image I S3 .

[0039] It should be noted that, when the exposure normalization processing only includes global normalization processing, the third brightness image is the long-exposure image frame after the exposure normalization processing, and the fourth brightness image is the short-exposure image frame after the exposure normalization processing.

[0040] Step 104: Input the first image and the second image into a feature fusion model for feature fusion processing to obtain a target image.

[0041] Among them, the above feature fusion model can be used to realize image feature extraction and image feature fusion.

[0042] Optionally, the feature fusion model may include a first fusion network and a second fusion network, wherein the first fusion network includes 2 perceptual convolution blocks and a PixelShuffle convolution kernel with a magnification of 2, and the second fusion network includes 3 perceptual convolution blocks and a PixelShuffle convolution kernel.

[0043] Specifically, if Figure 2 As shown, the feature fusion model may include a first fusion network 210 and a second fusion network 220, the first fusion network 210 includes a perceptual convolution block 211, a perceptual convolution block 212 and a PixelShuffle convolution kernel 213, and the second fusion network 220 includes a perceptual convolution block 221, a perceptual convolution block 222, a PixelShuffle convolution kernel 223 and a perceptual convolution block 224, wherein each perceptual convolution block in the first fusion network 210 and the second fusion network 220 includes two convolution kernels of size 3×3, padding of 1 and a Relu activation function.

[0044] In the embodiment of the present application, since the long-exposure image frames and the short-exposure image frames are first subjected to exposure normalization processing before the motion alignment processing is performed, the visual differences between the long-exposure image frames and the short-exposure image frames caused by different exposure times can be eliminated, thereby obtaining a more accurate optical flow estimation result, thereby improving the motion alignment effect and alleviating the ghost distortion phenomenon in the image.

[0045] Alternatively, as Figure 3 As shown, before the above step 104, an image processing method provided by an embodiment of the present application may further include step 105. The above step 104 may be implemented by the following step 104a.

[0046] Step 105: Perform ghost distortion estimation processing on the first image and the second image to obtain a ghost distortion map.

[0047] Optionally, the electronic device may first calculate the gradient amplitude image of the first image to obtain the third image, calculate the gradient amplitude image of the second image to obtain the fourth image; and then determine the ghost distortion image I according to the following formula (2): g :

[0048]

[0049] Among them, g l1 (x) is the pixel value of the pixel point x in the third image, g s1 (x) is the pixel value of the pixel point x in the fourth image, f l1 (x) is the grayscale value of pixel x in the first image, f s1(x) is the grayscale value of the pixel point x in the second image, and x is the pixel coordinate of a pixel point.

[0050] Optionally, the electronic device may calculate the gradient magnitude image of the first image and the second image by using a Sobel operator.

[0051] Based on the above solution, since the ghost distortion map can be determined, the position of ghosts that may appear in the moving image can be predicted, thereby providing a basis for eliminating ghosts.

[0052] Step 104a: Input the first image, the second image, and the ghost distortion image into the feature fusion model for feature fusion processing to obtain the target image.

[0053] Specifically, if Figure 4 As shown, the first image, the second image and the ghost distortion image can first be feature sampled by a downsampling function of 4 times to obtain the first image feature, and then the first image feature is feature stacked and input into the first fusion network for feature fusion to obtain the second image feature. At the same time, the first image, the second image and the ghost distortion image can also be feature sampled by a downsampling function of 2 times to obtain the third image feature, and then the third image feature and the second image feature are feature fused and input into the second fusion network for feature fusion to finally obtain the target image.

[0054] It should be noted that Figure 4 The "D_4" in the figure represents the downsampling function by a factor of 4, which reduces the image by a factor of 4; the "D_2" represents the downsampling function by a factor of 2, which reduces the image by a factor of 2.

[0055] Optionally, the electronic device may obtain a motion image F captured by a motion camera. m As the learning objective of the above feature fusion model, the loss function of the feature fusion model is the following formula (3):

[0056]

[0057] Among them, d2 represents the downsampling 2-fold function, d4 represents the downsampling 4-fold function, G1 represents the first fusion network, G2 represents the second fusion network, θ1 and θ2 are the model parameters that need to be learned by the feature fusion model, and N represents the number of training samples.

[0058] Optionally, the feature fusion model can be built and trained based on the Pytorch deep learning framework. The loss function can be minimized as much as possible by updating the model parameters. Usually, the feature fusion model can be stabilized after 150 epochs of training.

[0059] The feature fusion model in the embodiment of the present application can achieve the fusion of effective information of the first image, the second image and the ghost distortion image by constructing a first fusion network and a second fusion network with different spatial scales, thereby effectively suppressing motion blur in the fused target image.

[0060] In an embodiment of the present application, a ghost distortion map can be obtained by performing ghost distortion estimation processing on the first image and the second image, and input into the feature fusion model together with the first image and the second image. Since the areas in the first image and the second image where ghost distortion is likely to occur can be predicted in advance, the learning pressure of the feature fusion model can be alleviated and the prediction accuracy of the model can be improved.

[0061] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, the image processing device provided in the embodiment of the present application is described by taking the image processing device executing the image processing method as an example.

[0062] like Figure 5 As shown, the embodiment of the present application also provides an image processing device 500, including: an acquisition module 501 and a processing module 502; the acquisition module 501 is used to acquire long-exposure image frames and short-exposure image frames; the processing module 502 is used to perform exposure normalization processing on the long-exposure image frames and the short-exposure image frames; motion alignment processing is performed on the long-exposure image frames after the exposure normalization processing to obtain a first image, and motion alignment processing is performed on the short-exposure image frames after the exposure normalization processing to obtain a second image; the first image and the second image are input into a feature fusion model for feature fusion processing to obtain a target image; wherein, the feature fusion model is used to realize image feature extraction and image feature fusion.

[0063] Optionally, the exposure normalization processing includes global normalization processing; the processing module 502 is specifically used to perform intrinsic decomposition processing on the long exposure image frame to obtain a first brightness image, and perform intrinsic decomposition processing on the short exposure image frame to obtain a second brightness image; perform global normalization processing on the first brightness image according to the first exposure time of the first brightness image to obtain a third brightness image, and perform global normalization processing on the second brightness image according to the second exposure time of the second brightness image to obtain a fourth brightness image.

[0064] Optionally, the exposure normalization processing also includes local normalization processing; the processing module 502 is specifically used to perform local normalization processing based on the eight-neighborhood mean and the eight-neighborhood standard deviation of each pixel point in the third brightness image to obtain the long exposure image frame after exposure normalization processing; and perform local normalization processing based on the eight-neighborhood mean and the eight-neighborhood standard deviation of each pixel point in the fourth brightness image to obtain the short exposure image frame after exposure normalization processing.

[0065] Optionally, the processing module 502 is further used to perform ghost distortion estimation processing on the first image and the second image to obtain a ghost distortion map; and input the first image, the second image and the ghost distortion map into the feature fusion model for feature fusion processing to obtain the target image.

[0066] Optionally, the processing module 502 is specifically configured to calculate the gradient magnitude image of the first image to obtain a third image, and calculate the gradient magnitude image of the second image to obtain a fourth image; according to the formula Determine ghost distortion image I g ; Among them, g l1 (x) is the pixel value of the pixel point x in the third image, g s1 (x) is the pixel value of the pixel point x in the fourth image, f l1 (x) is the grayscale value of pixel x in the first image, f s1 (x) is the grayscale value of the pixel point x in the second image, and x is the pixel coordinate of a pixel point.

[0067] Optionally, the feature fusion model includes a first fusion network and a second fusion network, the first fusion network includes 2 perceptual convolution blocks and a PixelShuffle convolution kernel with a magnification of 2, and the second fusion network includes 3 perceptual convolution blocks and a PixelShuffle convolution kernel.

[0068] In the embodiment of the present application, since the long-exposure image frames and the short-exposure image frames are first subjected to exposure normalization processing before the motion alignment processing is performed, the visual differences between the long-exposure image frames and the short-exposure image frames caused by different exposure times can be eliminated, thereby obtaining a more accurate optical flow estimation result, thereby improving the motion alignment effect and alleviating the ghost distortion phenomenon in the image.

[0069] The image processing device in the embodiment of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or a device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not specifically limit it.

[0070] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0071] The image processing device provided in the embodiment of the present application can achieve Figures 1 to 4 To avoid repetition, the various processes implemented in the method embodiment are not described here.

[0072] Alternatively, as Figure 6 As shown, an embodiment of the present application also provides an electronic device 600, including a processor 601 and a memory 602, wherein the memory 602 stores a program or instruction that can be run on the processor 601, and when the program or instruction is executed by the processor 601, the various steps of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0073] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0074] Figure 7 A schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0075] The electronic device 1000 includes but is not limited to components such as a radio frequency unit 1001 , a network module 1002 , an audio output unit 1003 , an input unit 1004 , a sensor 1005 , a display unit 1006 , a user input unit 1007 , an interface unit 1008 , a memory 1009 , and a processor 1010 .

[0076] Those skilled in the art will understand that the electronic device 1000 may also include a power source (such as a battery) to power each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0077] Among them, processor 1010 is used to obtain long-exposure image frames and short-exposure image frames; perform exposure normalization processing on the long-exposure image frames and the short-exposure image frames; perform motion alignment processing on the long-exposure image frames after exposure normalization processing to obtain a first image, and perform motion alignment processing on the short-exposure image frames after exposure normalization processing to obtain a second image; input the first image and the second image into a feature fusion model for feature fusion processing to obtain a target image; wherein the feature fusion model is used to realize image feature extraction and image feature fusion.

[0078] In the embodiment of the present application, since the long-exposure image frames and the short-exposure image frames are first subjected to exposure normalization processing before the motion alignment processing is performed, the visual differences between the long-exposure image frames and the short-exposure image frames caused by different exposure times can be eliminated, thereby obtaining a more accurate optical flow estimation result, thereby improving the motion alignment effect and alleviating the ghost distortion phenomenon in the image.

[0079] Optionally, the exposure normalization processing includes global normalization processing; the processor 1010 is specifically used to perform intrinsic decomposition processing on the long exposure image frame to obtain a first brightness image, and perform intrinsic decomposition processing on the short exposure image frame to obtain a second brightness image; perform global normalization processing on the first brightness image according to the first exposure time of the first brightness image to obtain a third brightness image, and perform global normalization processing on the second brightness image according to the second exposure time of the second brightness image to obtain a fourth brightness image.

[0080] In an embodiment of the present application, exposure normalization processing can be performed on long-exposure image frames and short-exposure image frames through global normalization processing. Since different exposure times can change the imaging brightness and color of the image, dividing by the exposure time can eliminate the optical flow alignment error caused by different brightness.

[0081] Optionally, the exposure normalization processing also includes local normalization processing; the processor 1010 is specifically used to perform local normalization processing based on the eight-neighborhood mean and the eight-neighborhood standard deviation of each pixel point in the third brightness image to obtain the long exposure image frame after exposure normalization processing; and perform local normalization processing based on the eight-neighborhood mean and the eight-neighborhood standard deviation of each pixel point in the fourth brightness image to obtain the short exposure image frame after exposure normalization processing.

[0082] In an embodiment of the present application, exposure normalization processing can be performed on long-exposure image frames and short-exposure image frames through local normalization processing. Since optical flow estimation is calculated by matching key pixel points of long-exposure image frames and short-exposure image frames, local normalization processing can enhance the contrast of image texture information, thereby reducing the number of similar key points and improving the pixel matching accuracy.

[0083] Optionally, the processor 1010 is further configured to perform ghost distortion estimation processing on the first image and the second image to obtain a ghost distortion map; and input the first image, the second image and the ghost distortion map into the feature fusion model for feature fusion processing to obtain the target image.

[0084] In an embodiment of the present application, a ghost distortion map can be obtained by performing ghost distortion estimation processing on the first image and the second image, and input into the feature fusion model together with the first image and the second image. Since the areas in the first image and the second image where ghost distortion is likely to occur can be predicted in advance, the learning pressure of the feature fusion model can be alleviated and the prediction accuracy of the model can be improved.

[0085] Optionally, the processor 1010 is specifically configured to calculate the gradient magnitude image of the first image to obtain a third image, and calculate the gradient magnitude image of the second image to obtain a fourth image; according to the formula Determine ghost distortion image I g ; Among them, g l1 (x) is the pixel value of the pixel point x in the third image, g s1 (x) is the pixel value of the pixel point x in the fourth image, f l1 (x) is the grayscale value of pixel x in the first image, f s1 (x) is the grayscale value of the pixel point x in the second image, and x is the pixel coordinate of a pixel point.

[0086] In the embodiment of the present application, since the ghost distortion map can be determined, the position of ghosts that may appear in the moving image can be predicted, thereby providing a basis for eliminating ghosts.

[0087] It should be understood that in an embodiment of the present application, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0088] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0089] Processor 1010 may include one or more processing units. Optionally, processor 1010 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 1010.

[0090] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0091] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0092] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0093] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0094] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0095] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0096] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0097] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. An image processing method, characterized in that: include: Acquire a long-exposure image frame and a short-exposure image frame; performing exposure normalization processing on the long-exposure image frame and the short-exposure image frame; Performing motion alignment processing on the long-exposure image frames after exposure normalization processing to obtain a first image, and performing motion alignment processing on the short-exposure image frames after exposure normalization processing to obtain a second image; Inputting the first image and the second image into a feature fusion model for feature fusion processing to obtain a target image; Wherein, the feature fusion model is used to realize image feature extraction and image feature fusion; Before inputting the first image and the second image into a feature fusion model for feature fusion processing to obtain a target image, the method further includes: performing ghost distortion estimation processing on the first image and the second image to obtain a ghost distortion map; The step of inputting the first image and the second image into a feature fusion model for feature fusion processing to obtain a target image includes: The first image, the second image and the ghost distortion image are input into the feature fusion model for feature fusion processing to obtain the target image.

2. The image processing method according to claim 1, wherein: The exposure normalization process includes a global normalization process; The performing exposure normalization processing on the long-exposure image frame and the short-exposure image frame includes: Performing intrinsic decomposition processing on the long-exposure image frame to obtain a first brightness image, and performing intrinsic decomposition processing on the short-exposure image frame to obtain a second brightness image; A third luminance image is obtained by performing global normalization processing on the first luminance image according to the first exposure time of the first luminance image, and a fourth luminance image is obtained by performing global normalization processing on the second luminance image according to the second exposure time of the second luminance image.

3. The image processing method according to claim 2, wherein: The exposure normalization processing further includes local normalization processing; and the exposure normalization processing performed on the long-exposure image frame and the short-exposure image frame further includes: performing local normalization processing according to the eight-neighborhood mean and the eight-neighborhood standard deviation of each pixel in the third brightness image to obtain the long-exposure image frame after exposure normalization processing; Local normalization processing is performed according to the eight-neighborhood mean and the eight-neighborhood standard deviation of each pixel point in the fourth brightness image to obtain the short exposure image frame after exposure normalization processing.

4. The image processing method according to claim 1, wherein: The performing ghost distortion estimation processing on the first image and the second image to obtain a ghost distortion map includes: Calculating a gradient magnitude image of the first image to obtain a third image, and calculating a gradient magnitude image of the second image to obtain a fourth image; According to the formula Determine ghost distortion image I g ; Among them, g l1 (x) is the pixel value of the pixel point x in the third image, g s1 (x) is the pixel value of the pixel point x in the fourth image, f l1 (x) is the grayscale value of pixel x in the first image, f s1 (x) is the grayscale value of the pixel point x in the second image, and x is the pixel coordinate of a pixel point.

5. The image processing method according to any one of claims 1 to 4, characterized in that: The feature fusion model includes a first fusion network and a second fusion network, the first fusion network includes 2 perceptual convolution blocks and a PixelShuffle convolution kernel with a magnification of 2, and the second fusion network includes 3 perceptual convolution blocks and a PixelShuffle convolution kernel.

6. An image processing device, characterized in that include: Acquisition module and processing module; The acquisition module is used to acquire long-exposure image frames and short-exposure image frames; The processing module is configured to perform exposure normalization processing on the long-exposure image frame and the short-exposure image frame; Performing motion alignment processing on the long-exposure image frames after exposure normalization processing to obtain a first image, and performing motion alignment processing on the short-exposure image frames after exposure normalization processing to obtain a second image; inputting the first image and the second image into a feature fusion model to perform feature fusion processing to obtain a target image; Wherein, the feature fusion model is used to realize image feature extraction and image feature fusion; The processing module is used to perform ghost distortion estimation processing on the first image and the second image to obtain a ghost distortion map; and input the first image, the second image and the ghost distortion map into the feature fusion model for feature fusion processing to obtain the target image.

7. The image processing device according to claim 6, wherein: The exposure normalization process includes a global normalization process; The processing module is specifically configured to perform intrinsic decomposition processing on the long-exposure image frame to obtain a first luminance image, and perform intrinsic decomposition processing on the short-exposure image frame to obtain a second luminance image; perform global normalization processing on the first luminance image according to a first exposure time of the first luminance image to obtain a third luminance image, and perform global normalization processing on the second luminance image according to a second exposure time of the second luminance image to obtain a fourth luminance image.

8. An electronic device, characterized in that: The image processing method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the image processing method according to any one of claims 1 to 5 is implemented.

9. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the image processing method according to any one of claims 1 to 5 is implemented.

Images