Image processing method, device and equipment

Abstract

The invention provides an image processing method, device and equipment. The method comprises the following steps: acquiring a first image set and a second image set obtained by shooting a target scene by a camera; wherein the exposure duration of the images in the first image set is different from that of the second image set; determining first motion information of the target scene in the first image set and / or jitter information of the camera during shooting of the first image set; determining image quality parameters corresponding to the second image set; determining image data to be fused from the first image set and the second image set according to the image quality parameters and the jitter information and / or the first motion information; and performing fusion processing on the to-be-fused image data to obtain a target image of the target scene.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and relates to but is not limited to an image processing method, apparatus, and device. Background Art

[0002] In order to obtain high-quality images in complex lighting scenes, the HDR algorithm can be used to merge multiple images with different exposure levels so that the generated image can maintain rich details in both highlights and shadows, thereby obtaining a wider color range and better light and shadow effects.

[0003] However, when there are moving objects in the picture, a "ghosting" phenomenon will occur during the fusion process, resulting in an inaccurate generated image. Summary of the invention

[0004] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising: obtaining a first image set and a second image set captured by a camera for a target scene; wherein the exposure duration of images in the first image set is different from that of the second image set; determining first motion information of the target scene in the first image set and / or camera jitter information during capturing the first image set; determining image quality parameters corresponding to the second image set; determining image data to be fused from the first image set and the second image set based on the image quality parameters, as well as the jitter information and / or the first motion information; and performing fusion processing on the image data to be fused to obtain a target image of the target scene.

[0005] In a second aspect, an embodiment of the present application provides an image processing device, comprising: an acquisition module, configured to acquire a first image set and a second image set captured by a camera for a target scene; wherein the exposure time of the images in the first image set is different from that of the second image set; a first determination module, configured to determine first motion information of the target scene in the first image set and / or jitter information of the camera during the shooting of the first image set; a second determination module, configured to determine image quality parameters corresponding to the second image set; a third determination module, configured to determine image data to be fused from the first image set and the second image set based on the image quality parameters, the jitter information and / or the first motion information; and a fusion module, configured to perform fusion processing on the image data to be fused to obtain a target image of the target scene.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and when the processor executes the program, the method described in the first aspect is implemented.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program enables a computer to execute the method described in the first aspect.

[0008] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer program instructions, which enable a computer to execute the method described in the first aspect.

[0009] In a sixth aspect, an embodiment of the present application provides a computer program, which enables a computer to execute the method described in the first aspect.

[0010] In an embodiment of the present application, the image data to be fused is determined from the first image set and the second image set according to the image quality parameter corresponding to the second image set, and the first motion information of the target scene in the first image set and / or the camera shaking information during the shooting of the first image set; based on this, the image data to be fused is fused to obtain the target image of the target scene. That is to say, for the determination of the image data to be fused, not only the image quality parameter corresponding to the second image set is taken into account, but also the first motion information of the target scene in the first image set and / or the camera shaking information during the shooting of the first image set is taken into account, rather than determining the target image only according to the sharpness of the image. In this way, for scenes with motion blur, the clarity of the determined image data to be fused can also be improved according to the image quality parameter corresponding to the second image set, and the first motion information of the target scene in the first image set and / or the camera shaking information during the shooting of the first image set; thereby, it is beneficial to improve the clarity of the target image.

[0011] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0014] Figure 1 A schematic diagram of an implementation process of an image processing method provided in an embodiment of the present application Figure 1 ;

[0015] Figure 2 A schematic diagram of an implementation flow of determining a first arc provided in an embodiment of the present application;

[0016] Figure 3 A schematic diagram of a further implementation flow of step 203 provided in an embodiment of the present application;

[0017] Figure 4 A schematic diagram of an implementation flow of determining a first motion amplitude provided in an embodiment of the present application;

[0018] Figure 5 A schematic diagram of an implementation flow of determining a signal-to-noise ratio provided in an embodiment of the present application;

[0019] Figure 6 A schematic diagram of an implementation flow of determining image data to be fused provided in an embodiment of the present application;

[0020] Figure 7 A schematic diagram of an implementation flow of determining a second motion amplitude provided in an embodiment of the present application;

[0021] Figure 8 An example diagram of blur degree provided in an embodiment of the present application;

[0022] Fig. 9 A schematic diagram of an implementation process of an image processing method provided in an embodiment of the present application Figure 2 ;

[0023] Fig.10 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0024] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0027] In the following description, reference is made to “some embodiments”, “this embodiment”, “embodiments of the present application” and examples, etc., which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0028] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies or terms of the embodiments of the present application. The following related technologies or related terms as optional solutions can be combined with the technical solutions of the embodiments of the present application at will, and they all belong to the protection scope of the embodiments of the present application.

[0029] In a related technology, when selecting a fusion area based on the High Dynamic Range (HDR) technology, the blurriness of the image is generally determined by the sharpness of the image.

[0030] However, the inventors of the present application have found through research and analysis of the above-mentioned related technologies that judging the blur degree of an image based on the image itself (i.e., the sharpness or edge statistics of the image) is prone to the following problems:

[0031] (1) It is easily affected by noise when the signal-to-noise ratio is low;

[0032] (2) For motion blur, it is difficult to effectively determine the degree of image blur through image sharpness or edge statistics.

[0033] In view of this, an embodiment of the present application provides an image processing method, which is applied to an electronic device. Figure 1 A schematic diagram of an implementation process of an image processing method provided in an embodiment of the present application Figure 1 ,like Figure 1 As shown, the method includes the following steps 101 to 105:

[0034] Step 101, obtaining a first image set and a second image set captured by a camera for a target scene; wherein the exposure time of the images in the first image set is different from that of the images in the second image set;

[0035] Step 102, determining first motion information of the target scene in the first image set and / or jitter information of the camera during shooting of the first image set;

[0036] Step 103, determining an image quality parameter corresponding to the second image set;

[0037] Step 104, determining image data to be fused from the first image set and the second image set according to the image quality parameter, the jitter information and / or the first motion information;

[0038] Step 105: performing fusion processing on the image data to be fused to obtain a target image of the target scene.

[0039] It can be understood that in the embodiment of the present application, the image data to be fused is determined from the first image set and the second image set according to the image quality parameter corresponding to the second image set, and the first motion information of the target scene in the first image set and / or the camera shaking information during the shooting of the first image set; based on this, the image data to be fused is fused to obtain the target image of the target scene. That is to say, for the determination of the image data to be fused, not only the image quality parameter corresponding to the second image set is taken into account, but also the first motion information of the target scene in the first image set and / or the camera shaking information during the shooting of the first image set is taken into account, rather than determining the target image only according to the sharpness of the image. In this way, for scenes with motion blur, the clarity of the determined image data to be fused can also be improved according to the image quality parameter corresponding to the second image set, and the first motion information of the target scene in the first image set and / or the camera shaking information during the shooting of the first image set; thereby, it is beneficial to improve the clarity of the target image.

[0040] The following describes further optional implementations and related terms of each of the above steps.

[0041] In step 101, a first image set and a second image set captured by a camera for a target scene are acquired; wherein the exposure duration of images in the first image set is different from that in the second image set.

[0042] It should be understood that in the embodiments of the present application, the exposure duration of the images in the first image set and the exposure duration of the images in the second image set are not limited. In some embodiments, the exposure duration of the images in the first image set is greater than the exposure duration of the images in the second image set; the images in the first image set are long-frame images, and the images in the second image set are short-frame images.

[0043] In the embodiments of the present application, the target scene is not limited. In some embodiments, the target scene includes: a moving first object.

[0044] It should be understood that in the embodiments of the present application, the first object is not limited. In some embodiments, the moving first object may be a first object capable of autonomous movement, such as a person, an animal, a robot, etc. In other embodiments, the moving first object may be a first object capable of passive movement, such as leaves blown by the wind, rippling water, etc.

[0045] In step 102, first motion information of the target scene in the first image set and / or jitter information of the camera during shooting of the first image set are determined.

[0046] In some embodiments, the jitter information includes a first radian; Figure 2 A schematic diagram of an implementation flow of determining a first arc provided in an embodiment of the present application is as follows: Figure 2 As shown, the first arc can be determined by the following steps 201 to 203:

[0047] Step 201, determining angular velocity data collected by an angular velocity sensor during shooting of a first image set;

[0048] Step 202, determining the second arc of the camera in multiple direction dimensions according to the angular velocity data;

[0049] Step 203: determining the first arc according to the second arc of the camera in multiple direction dimensions, the exposure time of the first image set, and the acquisition time of the angular velocity data.

[0050] It should be understood that in the embodiments of the present application, the angular velocity sensor is not limited. In some embodiments, the angular velocity sensor is an angular velocity sensor on the camera, for example, a gyroscope on the camera; accordingly, the angular velocity information is gyroscope data. In other embodiments, the angular velocity sensor is a sensor on the device where the camera is located.

[0051] In the embodiment of the present application, the second curvature of the camera in multiple directions is not limited. In some embodiments, the second curvature of the camera in multiple directions includes: the second curvature of the camera in the X-axis, Y-axis and Z-axis directions.

[0052] In some embodiments, Figure 3 A schematic diagram of a further implementation flow of step 203 provided in the embodiment of the present application is as follows: Figure 3 As shown, step 203 may include the following steps 301 to 303:

[0053] Step 301, determining a third arc according to the second arc of the camera in multiple direction dimensions;

[0054] Step 302, determining a first coefficient according to the exposure duration of the first image set and the collection duration of the angular velocity data;

[0055] Step 303: determine the first radian according to the product of the first coefficient and the third radian.

[0056] It should be understood that in the embodiment of the present application, the third curvature is not limited, and the third curvature is determined according to the second curvatures of the camera in multiple directional dimensions. In some embodiments, the third curvature is the maximum second curvature of the camera in multiple directional dimensions. In other embodiments, the third curvature is the average of the second curvatures of the camera in multiple directional dimensions.

[0057] In the embodiment of the present application, the first radian is not limited, and the first radian is determined according to the product of the first coefficient and the third radian. In some embodiments, the first radian is the product of the first coefficient and the third radian. In other embodiments, the first radian is the result of processing the product of the first coefficient and the third radian.

[0058] For example, in a possible implementation, firstly, the gyroscope data of multiple frames of images in the shooting stage is collected by the gyroscope of a mobile phone (an example of an electronic device), and the arc of movement of the gyroscope data in the X-axis, Y-axis and Z-axis directions is calculated. The three reverse gyroscope data are recorded as g x , g y , g z , calculate the gyroscope radians as follows:

[0059] rad x =g x0 *t0+g x1 ×t1+…+g xn *t n

[0060] Where t is the time interval of the sequence. Similarly, the radians in the Y and Z directions are calculated (i.e., an example of the first radian). Then the maximum radian in the X, Y, and Z directions is obtained as rad max (i.e., an example of the third arc), and then the arc during the long frame image shooting is calculated by the following formula:

[0061]

[0062] Among them, shutter L is the exposure time for long frames.

[0063] In some embodiments, the first motion information includes a first motion amplitude of a plurality of pixels; Figure 4 A schematic diagram of an implementation flow of determining a first motion amplitude provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the first motion amplitude can be determined through the following steps 401 to 402:

[0064] Step 401, determining corresponding first optical flow information according to the i-th first image in the first image set and the j-th second image in the second image set;

[0065] Wherein, i is greater than 0 and less than or equal to the number of images in the first image set; j is greater than 0 and less than or equal to the number of images in the second image set; the i-th first image frame corresponds to the j-th second image frame;

[0066] Step 402, determining a first motion amplitude of the nth pixel point according to the first optical flow information corresponding to the nth pixel point in the i-th frame of the first image, the exposure duration of the i-th frame of the first image, and the exposure duration of the j-th frame of the second image;

[0067] Wherein, n is greater than 0 and less than or equal to the number of pixels of the i-th frame of the first image.

[0068] It should be understood that in the embodiment of the present application, the first optical flow information is not limited, and the first optical flow information is used to characterize the motion vector between the image of the pixel point in the first image set and the corresponding image in the second image set.

[0069] In an embodiment of the present application, the correspondence between the first image and the second image includes but is not limited to at least one of the following: the first image and the second image capture images of the same scene, the viewing angle error between the first image and the second image is less than a certain threshold, the shooting time interval between the first image and the second image is less than a certain threshold, and the motion state captured by the first image and the second image is a continuous state.

[0070] In some embodiments, step 402 can be implemented as follows: determining a first norm of the first optical flow information corresponding to the n-th pixel; when the exposure time of the i-th frame of the first image is less than a preset exposure interval, determining a second coefficient according to the exposure time of the i-th frame of the first image, the exposure time of the j-th frame of the second image, the preset exposure interval, and the exposure time of other frames between the i-th frame of the first image and the j-th frame of the second image; and determining a first motion amplitude of the n-th pixel according to the product of the second coefficient and the first norm; or

[0071] When the exposure time of the i-th frame of the first image is greater than or equal to the preset exposure interval, a third coefficient is determined according to the exposure time of the i-th frame of the first image, the exposure time of the j-th frame of the second image, and the exposure time of other intermediate frames when shooting the i-th frame of the first image and the j-th frame of the second image; and a first motion amplitude of the n-th pixel is determined according to the product of the third coefficient and the first norm.

[0072] Exemplarily, in a possible implementation, a dense inverse search (DIS) optical flow algorithm is used to calculate the optical flow corresponding to the long frame image according to the long frame image and the corresponding short frame image, and then the modulus of the optical flow at a certain point (i.e., an example of the first norm) is calculated:

[0073]

[0074] Among them, f x Indicates the movement amplitude of a point in the X-axis direction, f y Indicates the movement amplitude of a point in the Y-axis direction.

[0075] Then, the motion amplitude of each point (i, j) is determined according to the exposure duration (i.e., an example of the second motion amplitude); when the exposure duration of the long frame image is less than the exposure interval:

[0076]

[0077] When the exposure time of a long frame image is greater than or equal to the exposure interval:

[0078]

[0079] Among them, shutter s is the exposure time of the short frame image, shutter L is the exposure time of the long frame image, shutter ex The exposure time and other frames in between when taking short-frame images and long-frame images.

[0080] In step 103, image quality parameters corresponding to the second image set are determined.

[0081] It should be understood that in the embodiments of the present application, the image quality parameters are not limited. In some embodiments, the image quality parameters include but are not limited to at least one of the following: image sharpness, image brightness, signal-to-noise ratio, image intensity, number of pixels, dynamic range, contrast, etc.

[0082] In some embodiments, the image quality parameter includes a signal-to-noise ratio of a plurality of pixels; Figure 5 A schematic diagram of a process flow for determining a signal-to-noise ratio is provided in an embodiment of the present application, such as Figure 5 As shown, the signal-to-noise ratio can be determined through the following steps 501 to 502:

[0083] Step 501, obtaining the noise parameter and sensitivity of the camera;

[0084] Step 502, determining the signal-to-noise ratio of the m-th pixel point according to the pixel value of the m-th pixel point of the j-th frame second image in the second image set, the sensitivity and the noise parameter;

[0085] Among them, j is greater than 0 and less than or equal to the number of images in the second image set, and m is greater than 0 and less than or equal to the number of pixels in the j-th frame of the second image.

[0086] In some embodiments, step 502 can be implemented as follows: determine a fourth coefficient based on the sensitivity and a Gaussian noise parameter in the noise parameter; determine a correction parameter based on the sensitivity and a Poisson noise parameter in the noise parameter; determine a first value based on the product of the fourth coefficient and the pixel value of the m-th pixel; determine a second value based on the first value and the correction parameter; and determine the signal-to-noise ratio of the m-th pixel based on the square root of the second value.

[0087] For example, in a possible implementation, firstly, the noise parameters of the camera of the shooting device are obtained, which are Gaussian noise parameters: p guss = {g0, g1} and Poisson noise parameter p po ={p0, p1, p2}. Take any short frame image I s If the three-channel value of point (i, j) is (x r , x g , x b ), then the SNR of channel r is:

[0088] Coff=iso*g0+g1

[0089] Bias=iso*iso*p0+iso*p1+p2

[0090]

[0091] Similarly, the SNR of the other two channels can be calculated. xg and SNR xb , then the SNR at (i, j) is:

[0092] SNR (i，j) =0.299*SNR xr +0.587*SNR xg +0.114*SNR xb

[0093] In step 104, image data to be fused is determined from the first image set and the second image set according to the image quality parameter, the jitter information and / or the first motion information.

[0094] In some embodiments, the image quality parameter includes a signal-to-noise ratio of a plurality of pixels; the jitter information includes a first arc of jitter of the camera during shooting of the first image set; and the first motion information includes a first motion amplitude of a plurality of pixels.

[0095] Correspondingly, the image data to be fused is determined from the first image set and the second image set according to the image quality parameter, the jitter information and / or the first motion information, including: when the signal-to-noise ratio, the first curvature and / or the first motion amplitude satisfy a first condition, the second image set is used as the image data to be fused; wherein the first condition includes: the proportion or number of the signal-to-noise ratio greater than or equal to the first threshold is greater than or equal to the second threshold, and the first curvature is greater than or equal to the third threshold and / or the average value of the first motion amplitude is greater than or equal to the fourth threshold.

[0096] It can be understood that in the embodiment of the present application, the proportion or number of the signal-to-noise ratio greater than or equal to the first threshold is greater than or equal to the second threshold, indicating that there are more pixels with a high signal-to-noise ratio, and therefore the quality of the image in the second image set is higher; the first arc is greater than or equal to the third threshold and / or the average value of the first motion amplitude is greater than or equal to the fourth threshold, indicating that the amplitude of the pixel point movement is large, and therefore the degree of motion blur of the image in the first image set is high. Therefore, when the signal-to-noise ratio, the first arc and / or the first motion amplitude meet the first condition, the second image set is used as the image data to be fused; that is, the second image set with higher image quality is used as the image data to be fused, and the images in the first image set with higher motion blur are discarded, so that the clarity of the determined image data to be fused is higher; which is beneficial to improve the clarity of the target image.

[0097] It should be understood that in the embodiment of the present application, there is no limitation on the first threshold, the second threshold, the third threshold and the fourth threshold, and there is no limitation on the proportion and the quantity in "the proportion of the signal-to-noise ratio greater than or equal to the first threshold or the quantity greater than or equal to the second threshold".

[0098] In some embodiments, the image quality parameter includes a signal-to-noise ratio of a plurality of pixels; the jitter information includes a first arc of jitter of the camera during shooting of the first image set; and the first motion information includes a first motion amplitude of a plurality of pixels.

[0099] Figure 6 A schematic diagram of an implementation flow of determining image data to be fused provided in an embodiment of the present application, such as Figure 6 As shown, the image data to be fused can be determined through the following steps 601 to 603:

[0100] Step 601, aligning the i-th first image in the first image set and the j-th second image in the second image set to obtain a misaligned area when the signal-to-noise ratio, the first arc and / or the first motion amplitude satisfy a second condition;

[0101] Wherein, i is greater than 0 and less than or equal to the number of images in the first image set; j is greater than 0 and less than or equal to the number of images in the second image set; the i-th first image corresponds to the j-th second image;

[0102] Step 602, determining second motion information of the misaligned area;

[0103] Step 603: determining image data to be fused from the first image set and the second image set according to the signal-to-noise ratio of the misaligned area and the second motion information;

[0104] The second condition includes at least one of the following: the proportion or number of the signal-to-noise ratio greater than or equal to the first threshold is less than the second threshold, the first arc is less than the third threshold, and the average value of the first motion amplitude is less than the fourth threshold.

[0105] It can be understood that in the embodiment of the present application, when the signal-to-noise ratio, and the first arc and / or the first motion amplitude meet the second condition, that is, when the signal-to-noise ratio, and the first arc and / or the first motion amplitude do not meet the first condition, the image data to be fused is determined from the first image set and the second image set according to the second motion information of the unaligned area corresponding to the first image set and the signal-to-noise ratio of the unaligned area. In this way, since the probability that the unaligned area is a blurred area is relatively high, when determining the image data to be fused, the second motion information of the unaligned area corresponding to the first image set and the signal-to-noise ratio of the unaligned area in the second image set are taken into consideration, which is beneficial to further improve the degree of blur of the image in the determined first image set in the unaligned area, so that according to the degree of blur of the image in the first image set in the unaligned area and the image quality of the unaligned area in the second image set, the image data to be fused is determined to be more accurate, that is, more in line with user needs, which is beneficial to improve the quality of the target image.

[0106] In some embodiments, the second motion information includes a second motion amplitude of a plurality of pixels; Figure 7 A schematic diagram of an implementation flow of determining a second motion amplitude provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the second motion amplitude can be determined through the following steps 701 to 704:

[0107] Step 701, clustering the misaligned regions to obtain at least one connected region;

[0108] Step 702, determining first optical flow information and grayscale change characteristics corresponding to the i-th frame of the first image;

[0109] Step 703, determining second optical flow information corresponding to the at least one connected domain according to the first optical flow information corresponding to the i-th frame of the first image and the grayscale change feature;

[0110] Step 704, determining a second motion amplitude of the p-th pixel point according to the second optical flow information corresponding to the p-th pixel point in the at least one connected domain, the exposure duration of the i-th frame of the first image, and the exposure duration of the j-th frame of the second image;

[0111] Wherein, p is greater than 0 and less than or equal to the number of pixels of the at least one connected domain.

[0112] In some embodiments, step 703 can be implemented as follows: determine an affine matrix based on the first optical flow information corresponding to the first image of the i-th frame and the grayscale change characteristics; determine the second optical flow information corresponding to the at least one connected domain based on the difference between the first optical flow information and the affine matrix.

[0113] In some embodiments, step 704 can be implemented as follows: determining the second norm of the optical flow residual corresponding to the p-th pixel in the at least one connected domain; when the exposure time of the i-th frame of the first image is less than the preset exposure interval, determining the fifth coefficient according to the exposure time of the i-th frame of the first image, the exposure time of the j-th frame of the second image, the preset exposure interval, and the exposure time of other frames between the i-th frame of the first image and the j-th frame of the second image; and determining the second motion amplitude of the p-th pixel according to the product of the fifth coefficient and the second norm; or

[0114] When the exposure time of the i-th frame first image is greater than or equal to the preset exposure interval, a sixth coefficient is determined according to the exposure time of the i-th frame first image, the exposure time of the j-th frame second image, and the exposure time of other intermediate frames when shooting the i-th frame first image and the j-th frame second image; and a second motion amplitude of the p-th pixel point in the at least one connected domain is determined according to the product of the sixth coefficient and the second norm.

[0115] In some embodiments, the grayscale change feature includes a Harris corner point.

[0116] In some embodiments, determining the first optical flow information corresponding to the i-th frame first image includes: determining the corresponding first optical flow information based on the i-th frame first image in the first image set and the j-th frame second image in the second image set; wherein i is greater than 0 and less than or equal to the number of images in the first image set; j is greater than 0 and less than or equal to the number of images in the second image set; and the i-th frame first image corresponds to the j-th frame second image.

[0117] For example, in a possible implementation, the misaligned areas of the long frame image and the corresponding short frame image are calculated; the obtained misaligned areas are called ghosts, and the ghost areas are clustered to obtain a series of connected domains. For the blur of each connected domain: first, the global affine matrix is ​​obtained through the original DIS optical flow (i.e., an example of the first optical flow information) and the statistical Harris corner points, and then the remapTable composed of the affine matrix is ​​subtracted from the original DIS optical flow to obtain the optical flow residual representing the local motion (i.e., an example of the second optical flow information), which is recorded as flow local , and then through flow local The calculated local blur map. If, flow local The optical flow residual of a point in the connected domain is (f x1 , f y1 ), then the second motion amplitude is calculated as follows:

[0118]

[0119] Among them, f x1 Indicates the movement amplitude of a point in the connected domain in the X-axis direction, f y1 Indicates the movement amplitude of a point in the connected domain in the Y-axis direction.

[0120] Then the motion amplitude of each point (i, j) in the connected domain is determined according to the exposure time. When the exposure time of the long-frame image is less than the exposure interval:

[0121]

[0122] When the exposure time of a long frame image is greater than or equal to the exposure interval:

[0123]

[0124] Among them, shutter s is the exposure time of the short frame image, shutter L is the exposure time of the long frame image, shutter exThe exposure time of the other frames between the short frame image and the long frame image. The local blur map includes B1 of each pixel in the connected domain. (i，j) .

[0125] In some embodiments, step 603 may be implemented as follows: when the signal-to-noise ratio of the misaligned area and the second motion amplitude satisfy a third condition, using the image data corresponding to the at least one connected domain in the second image set as the image data to be fused of the at least one connected domain; and determining the image data to be fused of the area other than the at least one connected domain from the first image set and the second image set according to the brightness and / or sharpness of the first image set and the second image set;

[0126] The third condition includes: the proportion or number of the misaligned areas having a signal-to-noise ratio less than the fifth threshold is less than the sixth threshold, and the proportion or number of the second motion amplitude greater than or equal to the seventh threshold is greater than or equal to the eighth threshold.

[0127] It can be understood that in the embodiment of the present application, the proportion or number of the signal-to-noise ratio of the unaligned area less than the fifth threshold is less than the sixth threshold, indicating that the unaligned area has fewer pixels with low signal-to-noise ratio, that is, the unaligned area has more pixels with high signal-to-noise ratio, so the image quality of the unaligned area in the second image set is higher; the proportion or number of the second motion amplitude greater than or equal to the seventh threshold is greater than or equal to the eighth threshold, indicating that the amplitude of the pixel movement of the unaligned area (that is, the at least one connected domain) is large, so the motion blur of the unaligned area in the first image set is high. Therefore, when the signal-to-noise ratio and the second motion amplitude of the unaligned area meet the third condition, the image data corresponding to the unaligned area in the second image set is used as the image data to be fused in the unaligned area; and according to the brightness and / or sharpness of the first image set and the second image set, the image data to be fused of the area other than the at least one connected domain is determined. That is to say, the image data corresponding to the unaligned area in the second image set with higher image quality is used as the image data to be fused in the unaligned area, and the image data corresponding to the unaligned area in the first image set with higher motion blur is discarded, so that the clarity of the determined image data to be fused is higher; and it is beneficial to improve the clarity of the target image.

[0128] It should be understood that in the embodiment of the present application, the fifth threshold, the sixth threshold, the seventh threshold and the eighth threshold are not limited, the proportion and the quantity in "the proportion or the number of the signal-to-noise ratio less than the fifth threshold is less than the sixth threshold" are not limited, and the proportion and the number in "the proportion or the number of the second motion amplitude greater than or equal to the seventh threshold is greater than or equal to the eighth threshold" are not limited.

[0129] In some other embodiments, step 603 may also be implemented as follows: when the signal-to-noise ratio and the second motion amplitude of the misaligned area satisfy a fourth condition, using the image data corresponding to the at least one connected domain in the first image set as the image data to be fused of the at least one connected domain; and determining the image data to be fused of the area other than the at least one connected domain from the first image set and the second image set according to the brightness and / or sharpness of the first image set and the second image set;

[0130] Among them, the fourth condition includes at least one of the following: the proportion or number of the misaligned areas whose signal-to-noise ratio is less than the fifth threshold is greater than or equal to the sixth threshold, and the proportion or number of the second motion amplitude greater than or equal to the seventh threshold is less than the eighth threshold.

[0131] It can be understood that in the embodiment of the present application, when the signal-to-noise ratio and the second motion amplitude of the unaligned area meet the fourth condition, that is, when the signal-to-noise ratio and the second motion amplitude of the unaligned area do not meet the third condition, the proportion of the signal-to-noise ratio of the unaligned area less than the fifth threshold or the number is greater than or equal to the sixth threshold, indicating that there are more pixels with low signal-to-noise ratio in the unaligned area, that is, there are fewer pixels with high signal-to-noise ratio in the unaligned area, so the image quality of the unaligned area in the second image set is poor; the proportion of the second motion amplitude greater than or equal to the seventh threshold or the number is less than the eighth threshold, indicating that the amplitude of the pixel movement of the unaligned area (that is, the at least one connected domain) is small, so the degree of motion blur of the unaligned area in the first image set is low. That is to say, when the quality of the unaligned area in the second image set is poor and / or the degree of motion blur of the unaligned area in the first image set is low, the image data corresponding to the unaligned area in the first image set is used as the image data to be fused of the unaligned area; and according to the brightness and / or sharpness of the first image set and the second image set, the image data to be fused of the area other than the at least one connected domain is determined. In this way, the image data corresponding to the unaligned area in the first image set with higher image quality is used as the image data to be fused in the unaligned area, and the image data corresponding to the unaligned area in the second image set is discarded, so that the clarity of the determined image data to be fused is higher, which is beneficial to improving the clarity of the target image.

[0132] In step 105, the image data to be fused is fused to obtain a target image of the target scene.

[0133] The following example describes a possible implementation of an image processing method described in one or more of the above embodiments.

[0134] The embodiment of the present application provides an image processing method, which determines the image data to be fused based on the fuzzy judgment method of optical flow and gyroscope (Gyroscope, gyro), and the processing process of the method includes:

[0135] S1, firstly collects gyroscope data of multiple frames of image data during the shooting phase through the gyroscope of the mobile phone.

[0136] S2, according to the gyroscope data, calculate the arc of movement in the X-axis, Y-axis and Z-axis directions; and according to the calculated arc of movement in the X-axis, Y-axis and Z-axis directions, calculate the movement arc of the long frame image (i.e., an example of the first arc).

[0137] S3, for the long frame image and the corresponding short frame image, calculating the optical flow between the two (ie, an example of the first optical flow information).

[0138] S4, calculating the motion amplitude of the long frame image according to the optical flow (ie, an example of the first motion amplitude).

[0139] S5, down-sampling the short frame image, and calculating the SNR of the image according to the calibrated noise parameters.

[0140] S6: If the arc of the gyroscope movement (i.e., an example of the first arc) is greater than a certain value, and the proportion of the SNR in multiple short-frame images that is lower than a certain threshold is less than a certain proportion, and the average value of the first motion amplitude is greater than a certain value; then the long-frame image is directly discarded during fusion and only the short-frame image is retained.

[0141] If the value of each SNR point is counted, the number of pixels in the image whose SNR is greater than the threshold value n is obtained. SNR Calculator accounts for the proportion of short frame images:

[0142]

[0143] Among them, r SNR_global It represents the ratio of the number of pixels in a short frame image whose SNR is greater than the threshold, w represents the width of the short frame image, and h represents the height of the short frame image.

[0144] S7, calculating the misaligned area for the long frame image and the corresponding short frame image.

[0145] S8, cluster the unaligned regions to obtain a series of connected domains.

[0146] S9, according to the optical flow and the statistical Harris corner points, a series of second motion amplitudes corresponding to the connected domains are obtained.

[0147] S9, traverse each misaligned area and calculate the proportion r of the second motion amplitude in this area that is greater than a certain thresholdf At the same time, count the number of r cells in this area whose SNR is less than a certain threshold s , if r f >th f And r s <th s , then this area chooses to fuse short frames, otherwise it chooses to fuse long frames.

[0148] Traverse each misaligned area and calculate the ratio r of the value of the blur degree map (i.e., the set of the second motion amplitude) calculated by the local optical flow (i.e., an example of the second optical flow information) in this area that is greater than a certain threshold f At the same time, count the number of r cells in this area whose SNR is less than a certain threshold s , if r f >th f And r s <th s If this area selects to fuse short frames, otherwise it selects to fuse long frames.

[0149] This application designs a blur judgment method based on optical flow and gyro, which estimates the degree of motion of a long frame image through inter-frame optical flow and gyro information to judge the blur, while assisting with noise to ensure the clarity of the target image, and ensure that the local blur is relatively small while the image quality is also relatively good.

[0150] A blur judgment method based on optical flow and gyro: First, the global blur degree is judged by global optical flow and gyro, and then the local optical flow is used to judge each area, and the signal-to-noise ratio is used to ensure the accuracy of the judgment.

[0151] Figure 8 An example diagram of blur degree provided in an embodiment of the present application, wherein Figure 8 In the figure, the left 1 represents a long frame image, the left 2 represents a short frame image, and the right 1 represents the blur degree.

[0152] Fig. 9 A schematic diagram of an implementation process of an image processing method provided in an embodiment of the present application Figure 2 ,like Fig. 9 As shown, the method includes the following steps 901 to 911:

[0153] Step 901, calculating SNR according to the short frame image;

[0154] Step 902, estimating the motion arc;

[0155] Step 903, determining the optical flow according to the long frame image and the short frame image;

[0156] Step 904, calculating the long frame motion amplitude;

[0157] Step 905: Lost long frame? If yes, go to step 906; otherwise, go to step 907;

[0158] Step 906, only short frames are melted;

[0159] Step 907, determining a ghost area according to the long frame image and the short frame image;

[0160] Step 908, clustering;

[0161] That is, clustering the ghost regions (i.e., an example of misaligned regions);

[0162] Step 909, determine whether the area is blurred; if so, execute step 910, otherwise execute step 911;

[0163] Step 910, short frames are melted in this area;

[0164] Step 911, this area melts the long frame.

[0165] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.; or, steps in different embodiments may be combined into a new technical solution.

[0166] Fig.10 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application is shown in FIG. Fig.10 As shown, the device 100 includes: an acquisition module 1001, a first determination module 1002, a second determination module 1003, a third determination module 1004 and a fusion module 1005; wherein,

[0167] An acquisition module 1001 is configured to acquire a first image set and a second image set captured by a camera for a target scene; wherein the exposure time of images in the first image set is different from that of the second image set;

[0168] A first determining module 1002 is configured to determine first motion information of the target scene in the first image set and / or jitter information of the camera during shooting of the first image set;

[0169] A second determination module 1003, configured to determine an image quality parameter corresponding to the second image set;

[0170] A third determination module 1004 is configured to determine image data to be fused from the first image set and the second image set according to the image quality parameter, the jitter information and / or the first motion information;

[0171] The fusion module is configured to perform fusion processing on the image data to be fused to obtain a target image of the target scene.

[0172] In some embodiments, the jitter information includes a first arc; a first determination module 1002 is configured to determine angular velocity data collected by an angular velocity sensor during shooting of the first image set; based on the angular velocity data, determine a second arc of the camera in multiple directional dimensions; determine the first arc based on the second arc of the camera in multiple directional dimensions, the exposure time of the first image set and the collection time of the angular velocity data.

[0173] In some embodiments, the first determination module 1002 is configured to determine the third curvature according to the second curvature of the camera in multiple directional dimensions; determine the first coefficient according to the exposure time of the first image set and the collection time of the angular velocity data; and determine the first curvature according to the product of the first coefficient and the third curvature.

[0174] In some embodiments, the first motion information includes a first motion amplitude of multiple pixels; the first determination module 1002 is configured to determine the corresponding first optical flow information based on the i-th frame first image in the first image set and the j-th frame second image in the second image set; wherein i is greater than 0 and less than or equal to the number of images in the first image set; j is greater than 0 and less than or equal to the number of images in the second image set; the i-th frame first image corresponds to the j-th frame second image; the first motion amplitude of the n-th pixel is determined based on the first optical flow information corresponding to the n-th pixel in the i-th frame first image, the exposure time of the i-th frame first image and the exposure time of the j-th frame second image; wherein n is greater than 0 and less than or equal to the number of pixels in the i-th frame first image.

[0175] In some embodiments, the image quality parameters include the signal-to-noise ratio of multiple pixels; the second determination module 1003 is configured to obtain the noise parameters and sensitivity of the camera; determine the signal-to-noise ratio of the mth pixel based on the pixel value of the mth pixel of the jth frame of the second image in the second image set, the sensitivity and the noise parameters; wherein j is greater than 0 and less than or equal to the number of images in the second image set, and m is greater than 0 and less than or equal to the number of pixels in the jth frame of the second image.

[0176] In some embodiments, the image quality parameters include a signal-to-noise ratio of a plurality of pixels; the jitter information includes a first arc of jitter of the camera during shooting of the first image set; the first motion information includes a first motion amplitude of a plurality of pixels; the third determination module 1004 is configured to use the second image set as image data to be fused if the signal-to-noise ratio, and the first arc and / or the first motion amplitude satisfy a first condition; wherein the first condition includes: the proportion or number of the signal-to-noise ratio greater than or equal to a first threshold is greater than or equal to a second threshold, and the first arc is greater than or equal to a third threshold and / or the average value of the first motion amplitude is greater than or equal to a fourth threshold.

[0177] In some embodiments, the image quality parameter includes a signal-to-noise ratio of multiple pixels; the jitter information includes a first arc of the camera jittering during the shooting of the first image set; the first motion information includes a first motion amplitude of multiple pixels; a third determination module 1004 is configured to align the i-th frame first image in the first image set and the j-th frame second image in the second image set to obtain an unaligned area when the signal-to-noise ratio, the first arc and / or the first motion amplitude meet a second condition; wherein i is greater than 0 and less than or equal to the number of images in the first image set; j is greater than 0 and less than or equal to the number of images in the second image set; the i-th frame first image corresponds to the j-th frame second image; determine the second motion information of the unaligned area; determine the image data to be fused from the first image set and the second image set according to the signal-to-noise ratio of the unaligned area and the second motion information; wherein the second condition includes at least one of the following: the proportion or number of the signal-to-noise ratio greater than or equal to the first threshold is less than the second threshold, the first arc is less than the third threshold, and the average value of the first motion amplitude is less than the fourth threshold.

[0178] In some embodiments, the second motion information includes a second motion amplitude of multiple pixels; the third determination module 1004 is configured to cluster the misaligned areas to obtain at least one connected domain; determine the first optical flow information and grayscale change characteristics corresponding to the i-th frame first image; determine the second optical flow information corresponding to the at least one connected domain based on the first optical flow information and the grayscale change characteristics corresponding to the i-th frame first image; determine the second motion amplitude of the p-th pixel based on the second optical flow information corresponding to the p-th pixel in the at least one connected domain, the exposure time of the i-th frame first image and the exposure time of the j-th frame second image; wherein p is greater than 0 and less than or equal to the number of pixels in the at least one connected domain.

[0179] In some embodiments, the third determination module 1004 is configured to use the image data corresponding to the at least one connected domain in the second image set as the image data to be fused of the at least one connected domain when the signal-to-noise ratio of the misaligned area and the second motion amplitude satisfy a third condition; and determine the image data to be fused of the area other than the at least one connected domain from the first image set and the second image set based on the brightness and / or sharpness of the first image set and the second image set; wherein the third condition includes: the proportion of the signal-to-noise ratio of the misaligned area less than the fifth threshold or the number is less than the sixth threshold, and the proportion or number of the second motion amplitude greater than or equal to the seventh threshold is greater than or equal to the eighth threshold.

[0180] In some embodiments, the third determination module 1004 is configured to use the image data corresponding to the at least one connected domain in the first image set as the image data to be fused of the at least one connected domain when the signal-to-noise ratio and the second motion amplitude of the misaligned area meet a fourth condition; and determine the image data to be fused of the area other than the at least one connected domain from the first image set and the second image set based on the brightness and / or sharpness of the first image set and the second image set; wherein the fourth condition includes at least one of the following: the proportion or number of the misaligned areas whose signal-to-noise ratio is less than the fifth threshold is greater than or equal to the sixth threshold, and the proportion or number of the second motion amplitude is greater than or equal to the seventh threshold is less than the eighth threshold.

[0181] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0182] It should be noted that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit. It may also be implemented in the form of a combination of software and hardware.

[0183] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0184] An embodiment of the present application provides an electronic device, Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Fig.11 As shown, the electronic device 110 includes a memory 1101 and a processor 1102 , wherein the memory 1101 stores a computer program that can be run on the processor 1102 , and the processor 1102 implements the steps in the method provided in the above embodiment when executing the program.

[0185] It should be noted that the memory 1101 is configured to store instructions and applications executable by the processor 1102, and can also cache data to be processed or already processed by the processor 1102 and various modules in the electronic device 110 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0186] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the method provided in the above embodiment are implemented.

[0187] An embodiment of the present application provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps of the method provided in the above method embodiment.

[0188] It should be noted here that the description of the above electronic device, storage medium and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the electronic device, storage medium and computer program product embodiments of this application, please refer to the description of the method embodiment of this application for understanding.

[0189] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. The above description of each embodiment tends to emphasize the differences between the various embodiments, and the same or similarities can be referenced to each other. For the sake of brevity, this article will not repeat them.

[0190] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist at the same time, and object B exists alone.

[0191] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0192] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0193] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed on multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0194] In addition, all functional modules in the embodiments of the present application may be integrated into one processing unit, or each module may be a separate unit, or two or more modules may be integrated into one unit; the above-mentioned integrated modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0195] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0196] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0197] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0198] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0199] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0200] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a first image set and a second image set captured by a camera for a target scene; wherein the exposure time of images in the first image set is different from that of the second image set; Determining first motion information of the target scene in the first image set and / or jitter information of the camera during shooting of the first image set; determining an image quality parameter corresponding to the second image set; Determining image data to be fused from the first image set and the second image set according to the image quality parameter, the jitter information and / or the first motion information; The image data to be fused is subjected to fusion processing to obtain a target image of the target scene.

2. The method according to claim 1, characterized in that: The jitter information includes a first radian; The determining the camera shake information during the shooting of the first image set includes: determining angular velocity data collected by an angular velocity sensor during capturing of the first image set; Determine, according to the angular velocity data, a second arc of the camera in multiple direction dimensions respectively; The first arc is determined according to the second arc of the camera in multiple direction dimensions, the exposure time of the first image set and the acquisition time of the angular velocity data.

3. The method according to claim 2, characterized in that The determining the first curvature according to the second curvature of the camera in multiple direction dimensions, the exposure duration of the first image set, and the acquisition duration of the angular velocity data includes: Determine a third arc according to the second arc of the camera in multiple direction dimensions respectively; determining a first coefficient according to an exposure duration of the first image set and a collection duration of the angular velocity data; The first radian is determined according to the product of the first coefficient and the third radian.

4. The method according to claim 1, characterized in that: The first motion information includes first motion amplitudes of a plurality of pixels; The determining first motion information of the target scene in the first image set includes: Determine corresponding first optical flow information according to the i-th first image in the first image set and the j-th second image in the second image set; wherein i is greater than 0 and less than or equal to the number of images in the first image set; j is greater than 0 and less than or equal to the number of images in the second image set; the i-th first image corresponds to the j-th second image; Determine the first motion amplitude of the nth pixel point according to the first optical flow information corresponding to the nth pixel point in the i-th frame of the first image, the exposure time of the i-th frame of the first image, and the exposure time of the j-th frame of the second image; wherein n is greater than 0 and less than or equal to the number of pixels of the i-th frame of the first image.

5. The method according to claim 1, characterized in that The image quality parameter includes a signal-to-noise ratio of a plurality of pixels; and determining the image quality parameter corresponding to the second image set includes: Obtaining noise parameters and sensitivity of the camera; Determine the signal-to-noise ratio of the mth pixel in the jth frame of the second image in the second image set according to the pixel value of the mth pixel, the sensitivity and the noise parameter; wherein j is greater than 0 and less than or equal to the number of images in the second image set, and m is greater than 0 and less than or equal to the number of pixels in the jth frame of the second image.

6. The method according to any one of claims 1 to 5, characterized in that The image quality parameter includes a signal-to-noise ratio of a plurality of pixels; the jitter information includes a first arc of jitter of the camera during shooting of the first image set; the first motion information includes a first motion amplitude of a plurality of pixels; The determining, according to the image quality parameter, the jitter information and / or the first motion information, image data to be fused from the first image set and the second image set comprises: When the signal-to-noise ratio, and the first arc and / or the first motion amplitude satisfy a first condition, taking the second image set as image data to be fused; Among them, the first condition includes: the proportion or number of the signal-to-noise ratio greater than or equal to the first threshold is greater than or equal to the second threshold, and the first arc is greater than or equal to the third threshold and / or the average value of the first motion amplitude is greater than or equal to the fourth threshold.

7. The method according to any one of claims 1 to 5, characterized in that The image quality parameter includes a signal-to-noise ratio of a plurality of pixels; the jitter information includes a first arc of jitter of the camera during shooting of the first image set; the first motion information includes a first motion amplitude of a plurality of pixels; The determining, according to the image quality parameter, the jitter information and / or the first motion information, image data to be fused from the first image set and the second image set comprises: When the signal-to-noise ratio, and the first arc and / or the first motion amplitude satisfy a second condition, aligning the i-th first image in the first image set and the j-th second image in the second image set to obtain an unaligned area; wherein i is greater than 0 and less than or equal to the number of images in the first image set; j is greater than 0 and less than or equal to the number of images in the second image set; and the i-th first image corresponds to the j-th second image; determining second motion information of the misaligned area; determining image data to be fused from the first image set and the second image set according to the signal-to-noise ratio of the misaligned area and the second motion information; The second condition includes at least one of the following: the proportion or number of the signal-to-noise ratio greater than or equal to the first threshold is less than the second threshold, the first arc is less than the third threshold, and the average value of the first motion amplitude is less than the fourth threshold.

8. The method according to claim 7, characterized in that The second motion information includes second motion amplitudes of a plurality of pixels; The determining the second motion information of the misaligned area includes: Clustering the misaligned regions to obtain at least one connected domain; Determine first optical flow information and grayscale change characteristics corresponding to the i-th frame first image; Determine, according to the first optical flow information corresponding to the first image of the i-th frame and the grayscale change feature, the second optical flow information corresponding to the at least one connected domain; The second motion amplitude of the p-th pixel is determined according to the second optical flow information corresponding to the p-th pixel in the at least one connected domain, the exposure time of the i-th frame of the first image and the exposure time of the j-th frame of the second image; wherein p is greater than 0 and less than or equal to the number of pixels in the at least one connected domain.

9. The method according to claim 8, characterized in that The step of determining the image data to be fused from the first image set and the second image set according to the signal-to-noise ratio of the misaligned area and the second motion information includes: When the signal-to-noise ratio of the misaligned area and the second motion amplitude satisfy a third condition, using image data corresponding to the at least one connected domain in the second image set as image data to be fused of the at least one connected domain; and determining, according to the brightness and / or sharpness of the first image set and the second image set, image data to be fused in an area other than the at least one connected domain from the first image set and the second image set; The third condition includes: the proportion or number of the misaligned areas having a signal-to-noise ratio less than the fifth threshold is less than the sixth threshold, and the proportion or number of the second motion amplitude greater than or equal to the seventh threshold is greater than or equal to the eighth threshold.

10. The method according to claim 8, characterized in that The selecting a target data set from the first image set and the second image set according to the signal-to-noise ratio of the misaligned area and the second motion information comprises: When the signal-to-noise ratio of the misaligned area and the second motion amplitude satisfy a fourth condition, using image data corresponding to the at least one connected domain in the first image set as image data to be fused of the at least one connected domain; and determining, according to the brightness and / or sharpness of the first image set and the second image set, image data to be fused in an area other than the at least one connected domain from the first image set and the second image set; Among them, the fourth condition includes at least one of the following: the proportion or number of the misaligned areas whose signal-to-noise ratio is less than the fifth threshold is greater than or equal to the sixth threshold, and the proportion or number of the second motion amplitude greater than or equal to the seventh threshold is less than the eighth threshold.

11. An image processing device, characterized in that: The device comprises: An acquisition module, configured to acquire a first image set and a second image set captured by a camera for a target scene; wherein the exposure time of images in the first image set is different from that of the second image set; A first determination module, configured to determine first motion information of the target scene in the first image set and / or jitter information of the camera during shooting of the first image set; a second determination module, configured to determine an image quality parameter corresponding to the second image set; a third determination module, configured to determine image data to be fused from the first image set and the second image set according to the image quality parameter, the jitter information and / or the first motion information; The fusion module is configured to perform fusion processing on the image data to be fused to obtain a target image of the target scene.

12. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 9 is implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images