Image processing method, device and equipment and readable storage medium

Abstract

The invention discloses an image processing method, device and equipment and a readable storage medium, and belongs to the technical field of video processing. The method is executed by a terminal, and comprises the following steps: acquiring an original image, and an optical flow motion vector diagram and a registration image corresponding to the original image; determining different areas of a benchmark reference frame image according to the optical flow motion vector diagram; wherein different areas belong to different area types, and the area types comprise an exposure static area, a non-exposure static area, an exposure motion area and a non-exposure motion area; the benchmark reference frame image is a symmetric center image in the original image; determining a target strategy corresponding to each region according to the region type; and fusing the original image and the registration image according to the target strategy to obtain a fused image.

Description

Technical Field

[0001] The present application belongs to the field of video processing technology, and specifically relates to an image processing method, device, equipment and readable storage medium. Background Art

[0002] With the development of terminal devices, more users use mobile terminal devices to take photos and prefer to watch photos and preview videos on mobile terminal devices. In real scenes, the color of objects has a very high dynamic range. Most terminal devices can support 24 bits of information to represent colors, but the pixel value of a single channel is 8 bits of data. Therefore, the dynamic range that can be covered by a single photo taken by an ordinary terminal device is extremely limited. Using too short an exposure time will make the low-brightness parts of the scene dark and produce noise, while using too long an exposure time will cause the high-brightness parts of the scene to be overexposed and lose details. In order to make the image present rich color details and brightness levels, High Dynamic Range (HDR) technology that combines alternate exposure of multiple frames of pictures has become a mainstream solution.

[0003] Among them, the traditional HDR video synthesis technology directly fuses the alternating images of different exposure levels, and the obtained HDR video has the problem of poor image quality. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image processing method, apparatus, device and readable storage medium to solve the problem of poor quality of fused images.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0006] Acquire an original image, as well as an optical flow motion vector map and a registration image corresponding to the original image;

[0007] Determine different regions of a base reference frame image according to the optical flow motion vector map; wherein different regions belong to different region types, and the region types include an exposed static region, a non-exposed static region, an exposed motion region, and a non-exposed motion region; the base reference frame image is a symmetrical center image in the original image;

[0008] Determine, according to the area type, a target strategy corresponding to each area;

[0009] The original image and the registered image are fused according to the target strategy to obtain a fused image.

[0010] In a second aspect, an embodiment of the present application provides an image processing device, including:

[0011] A first processing module is used to obtain an original image, an optical flow motion vector map corresponding to the original image, and a registration image;

[0012] A second processing module is used to determine different regions of a base reference frame image according to the optical flow motion vector map; wherein different regions belong to different region types, and the region types include an exposed static region, a non-exposed static region, an exposed motion region, and a non-exposed motion region; and the base reference frame image is a symmetrical center image in the original image;

[0013] A third processing module is used to determine a target strategy corresponding to each of the areas according to the area type;

[0014] The fourth processing module is used to fuse the original image and the registered image according to the target strategy to obtain a fused image.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0018] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0019] In an embodiment of the present application, after acquiring the original image, the optical flow motion vector map corresponding to the original image, and the registered image corresponding to the original image, different regions of the baseline reference frame image can be determined based on the acquired optical flow motion vector map, and then the target strategy corresponding to each region of the baseline reference frame image can be determined according to the region types to which the different regions belong. Further, the determined target strategy can be used to fuse the original image and the configured image to obtain a fused image, thereby improving the quality of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a schematic diagram of the image processing flow of an embodiment of the present application;

[0021] Figure 2 is one of the schematic diagrams of the original image in the embodiment of the present application;

[0022] Figure 3 This is the second schematic diagram of the original image in the embodiment of the present application;

[0023] Figure 4 It is a schematic diagram of the application of the method of the embodiment of the present application;

[0024] Figure 5 It is a schematic diagram of the terminal hardware structure;

[0025] Figure 6 This is a schematic diagram of the HDR-MV-LITE module structure;

[0026] Figure 7 It is a schematic diagram of the terminal interface;

[0027] Figure 8 This is a schematic diagram of the HDR video processing flow;

[0028] Fig. 9 This is a schematic diagram of the offline HDR video processing process;

[0029] Fig.10 is a schematic diagram of the module structure of the image processing device according to an embodiment of the present application;

[0030] Fig.11 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application;

[0031] Fig.12 It is a schematic diagram of the structure of an electronic device according to another embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0033] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0034] The electronic device provided in the embodiment of the present application is described in detail below through specific embodiments and application scenarios in conjunction with the accompanying drawings.

[0035] like Figure 1 As shown, an image processing method in an embodiment of the present application is executed by a terminal, including:

[0036] Step 101: Acquire an original image, an optical flow motion vector map corresponding to the original image, and a registered image.

[0037] Among them, the original image can be a multi-frame image continuously captured by the image sensor, or it can be a multi-frame image obtained by processing the multi-frame images continuously captured by the image sensor through at least one fusion strategy (such as 2-in-1 half-frame rate output strategy, 2-in-1 equal frame rate output strategy, 3-in-1 reduced frame rate output strategy, 3-in-1 equal frame rate output strategy, etc.).

[0038] Optionally, the acquired original image includes an odd number of alternating long exposure images and short exposure images, for example, Figure 2 As shown, short exposure image 1 (exposure time (Exposure) is -2), long exposure image 2 (Exposure is 0), short exposure image 3 (Exposure is 2). Optionally, the acquired original image includes an odd number of images with increasing exposure, for example, Figure 3 As shown, image 1 has an Exposure of -5, image 2 has an Exposure of -3, image 3 has an Exposure of 0, image 4 has an Exposure of 3, and image 5 has an Exposure of 5.

[0039] The optical flow motion vector corresponding to the original image may be obtained by traditional optical flow estimation, or may be an optical flow motion vector determined by the region type to which each region belongs after the transmission channels of each pixel in the two images are divided into multiple regions based on the two images selected from the original image. For example, for the original image including short-exposure image 1, long-exposure image 2, and short-exposure image 3, four optical flow motion vectors are obtained, namely: optical flow motion vectors from short-exposure image 1 to short-exposure image 3 Figure 1 , the optical flow motion vector from short exposure image 3 to short exposure image 1 Figure 2 , the optical flow motion vector from short exposure image 1 to long exposure image 2 Figure 3 , the optical flow motion vector from short exposure image 3 to long exposure image 2 Figure 4 .

[0040] The warp image is obtained by performing an optical flow registration operation after obtaining the optical flow motion vector. For example, the optical flow registration operation can be to convert the optical flow motion vector Figure 2 The image warped to the middle frame (ie, long exposure image 2) is generated by combining with the short exposure image 1. For the original image including the short exposure image 1, the long exposure image 2 and the short exposure image 3, there are four registered images.

[0041] Step 102, determining different regions of a baseline reference frame image according to the optical flow motion vector map; wherein different regions belong to different region types, and the region types include an exposed still region, a non-exposed still region, an exposed motion region, and a non-exposed motion region; and the baseline reference frame image is a symmetrical center image in the original image.

[0042] Here, the reference frame image is also called an intermediate frame image. After the optical flow motion vector map is obtained in step 101, the reference frame image is further distinguished into an exposed static area, a non-exposed static area, an exposed motion area, and a non-exposed motion area.

[0043] Step 103: Determine a target strategy corresponding to each of the areas according to the area type.

[0044] In this step, the target strategy corresponding to each region is determined according to the region type to which each region belongs, so as to adopt an applicable strategy for different regions.

[0045] Optionally, based on a preset correspondence between different area types and policies, a target policy is searched for the area type through the correspondence.

[0046] Step 104: fuse the original image and the registered image according to the target strategy to obtain a fused image.

[0047] In this step, the target strategy determined in step 103 is used to fuse the original image and the configured image.

[0048] In this way, according to the above steps 101-104, the terminal executes the method of the embodiment of the present application. After obtaining the original image, the optical flow motion vector diagram corresponding to the original image, and the registered image corresponding to the original image, it can determine the different areas of the baseline reference frame image based on the obtained optical flow motion vector diagram, and then determine the target strategy corresponding to each area of ​​the baseline reference frame image according to the area type to which the different areas belong. Further, the determined target strategy can be used to fuse the original image and the configured image to obtain a fused image, thereby improving the quality of the fused image.

[0049] Optionally, in this embodiment, the original image is a RAW domain image. RAW domain images have a higher dynamic range, a wider range of light intensity, and a higher color depth, and can retain more highlights and shadows. In addition, each pixel on the RAW domain image is linear data, which is proportional to the exposure intensity. Compared with RGB domain images, HDR fusion in the RAW domain can generate higher quality HDR images.

[0050] Optionally, in this embodiment, the long exposure image is also called a long frame (i.e., long exposure frame) image, and the short exposure image is also called a short frame (i.e., short exposure frame) image. As an implementation method, in order to reduce power consumption, the long frame image acquisition size is full size, such as 1920*980; the short frame image acquisition size is half size, such as 960*490. Short frame images have high noise, and using bilinear interpolation to restore the short frame size to full size can reduce noise intensity and reduce power consumption of storing and transmitting short frame images.

[0051] Optionally, in this embodiment, determining different regions of the benchmark reference frame image according to the optical flow motion vector map includes:

[0052] Determining a moving area and a stationary area of ​​the benchmark reference frame image based on the optical flow motion vector diagram;

[0053] Based on a preset pixel threshold, an exposure area and a non-exposure area of ​​the benchmark reference frame image are determined.

[0054] That is, the obtained optical flow motion vector map is used to determine the moving area and the still area of ​​the baseline reference frame image; and a preset pixel threshold is used to determine whether the area to which the pixel point of the baseline reference frame image belongs is an exposed area or a non-exposed area, thereby obtaining the exposed still area, non-exposed still area, exposed moving area, and non-exposed moving area of ​​the baseline reference frame image.

[0055] As an optional implementation, the values ​​of U and V on the optical flow motion vector map are used to determine whether a certain pixel of the reference frame image is moving, so as to determine whether the pixel is in a moving area or a stationary area. U represents the displacement in a first direction (such as the x direction), and V represents the displacement in a second direction (such as the y direction).

[0056] As an optional implementation, the pixel value of a certain pixel point of the reference frame image is compared with a preset pixel threshold to determine whether the pixel point is in the exposure area or the non-exposure area. For example, the preset pixel threshold ε1, then

[0057]

[0058] Among them, ExposureMask k,i,j is the area of ​​pixel point (i, j) of the reference frame image, i and j are the coordinate values ​​of the pixel points, Value_Max is the maximum pixel value of the reference frame image (e.g., Value_Max of a 16-bit reference frame image is 65535), and I is the pixel value of pixel point (i, j). Among them, the low exposure area can be regarded as a non-exposure area, and the normal exposure area and the high exposure area can be regarded as an exposure area; or, the low exposure area and the normal exposure area can be regarded as a non-exposure area, and the high exposure area can be regarded as an exposure area.

[0059] Optionally, in this embodiment, fusing the original image and the registered image according to the target strategy to obtain a fused image includes:

[0060] Based on the target pixel and the target strategy corresponding to the area to which the target pixel belongs, calculation is performed according to the pixel and fusion weight indicated by the target strategy to obtain a fusion pixel value of the target pixel;

[0061] Wherein, the target pixel point is any pixel point in the benchmark reference frame image;

[0062] The target strategy is a first strategy, a second strategy, a third strategy or a fourth strategy, the first strategy corresponds to the exposed static area, the second strategy corresponds to the non-exposed static area, the third strategy corresponds to the exposed motion area, and the fourth strategy corresponds to the non-exposed motion area.

[0063] That is, in the embodiment of the present application, for the region type, the target strategy can indicate the pixel points and fusion weights to implement specific fusion calculations.

[0064] Optionally, the first strategy indicates that a pixel value of a first pixel point in the fused image is determined by a first pixel value and a first weight; wherein the first pixel value is a pixel value of the first pixel point in a first image, the first weight is a fusion weight of the first image at the first pixel point, and the first image includes one or more images selected from the original image;

[0065] The second strategy indicates that the pixel value of the second pixel point in the fused image is determined by a second pixel value and a second weight; wherein the second pixel value is the pixel value of the second pixel point in each of the original image and the registered image, and the second weight is the fusion weight of each of the original image and the registered image at the second pixel point;

[0066] The third strategy indicates that the pixel value of the third pixel point in the fused image is determined by a third pixel value and a third weight; wherein the third pixel value is the pixel value of the third pixel point in the second image, the third weight is the fusion weight of the second image at the third pixel point, and the second image includes one or more images selected from the registered image;

[0067] The fourth strategy indicates that the pixel value of the fourth pixel point in the fused image is determined by the pixel value of the fourth pixel point in the baseline reference frame image.

[0068] That is, for the first pixel point in the fused image, the first pixel point is in the exposure static area in the base reference frame image, and its pixel value is determined by the pixel value of the first pixel point in the first image (one or more images selected from the original image) and the fusion weight of the first image at the first pixel point according to the first strategy indication. Among them, the coordinates of the first pixel point (i, j) in the fused image in the first image remain unchanged and are also (i, j).

[0069] Optionally, the first image may be selected in a predefined manner. For example, the original image includes short exposure image 1, long exposure image 2, and short exposure image 3, and long exposure image 2 is the reference frame image. Then, the first image may be short exposure image 1 and short exposure image 3.

[0070] O i,j =I s1 i,j *norm(W s1 i,j )+I s2 i,j *norm(W s2 i,j )

[0071] Among them, O i,jRepresents the pixel value of the first pixel (i, j) in the fused image, I s1 i,j represents the value of the first pixel (i, j) in the short exposure image 1 (short frame); I s2 i,j represents the value of the first pixel (i, j) in the short exposure image 3 (short frame); W s1 i,j represents the fusion weight of the value of the short exposure image 1 (short frame) at the first pixel (i, j); W s2 i,j Represents the fusion weight of the value of the short exposure image 3 (short frame) at the first pixel (i, j).

[0072] P s1 i,j represents the pixel mean value of the area where the first pixel point (i, j) is located in the short exposure image 1 (short frame) (such as the 3*3 block where the first pixel point (i, j) is located); P s2 i,j represents the pixel mean value of the area where the first pixel point (i, j) is located in the short exposure image 3 (short frame) (such as the 3*3 block where the first pixel point (i, j) is located), Value_Mid is the pixel mean value of the reference frame image, and α1 is W s1 i,j The weight coefficient of W s2 i,j The weight coefficient, σ 2 is the pixel variance of the benchmark reference frame image, and α1 and α2 can be preset, such as set to 0.5.

[0073] For the second pixel point in the fused image, the second pixel point is in the non-exposed static area in the reference frame image, and its pixel value is determined according to the second strategy instruction by the pixel value of the second pixel point in each of the original image and the registered image, and the fusion weight of each of the original image and the registered image at the second pixel point. The coordinates of the second pixel point (i, j) in the fused image remain unchanged in each of the original image and the registered image, and are also (i, j).

[0074] Optional, Among them, O i,j Represents the pixel value of the second pixel (i, j) in the fused image, I k,i,j represents the pixel value of the second pixel (i, j) in the kth image of the original image and the registered image, W k,i,jrepresents the fusion weight of the kth image of the original image and the registered image at the second pixel (i, j), and N represents the total number of images of the original image and the registered image. The fusion weight used in this formula is weight-normalized, which can facilitate the weighted average calculation of pixels, that is, Of course, W can also be used directly in this formula u,i,j .

[0075] For the third pixel point in the fused image, the third pixel point is in the exposed motion area in the base reference frame image, and its pixel value is determined according to the third strategy indication by the pixel value of the third pixel point in one or more images (second images) selected from the registered image, and the fusion weight of the second image at the third pixel point. The coordinates of the third pixel point (i, j) in the fused image in the second image remain unchanged, and are also (i, j).

[0076] Optionally, the second image may be selected in a predefined manner. For example, the original image includes short exposure image 1, long exposure image 2, and short exposure image 3, and long exposure image 2 is the reference frame image. Then the second image may be an optical flow motion vector. Figure 1 Combined with the short exposure image 3, a warp to the intermediate frame is generated Figure 1 , and, the optical flow motion vector Figure 2 Combined with the short exposure image 1, it generates a warp to the middle frame Figure 2 ,

[0077] O i,j =I s1_warp i,j *norm(W s1_warp i,j )+I s2_warp i,j *norm(W s2_warp i,j )

[0078] Among them, O i,j Represents the pixel value of the third pixel (i, j) in the fused image, I s1_warp i,j express Figure 1 The pixel value of the third pixel (i, j) in s2_warp i,j express Figure 2 The pixel value of the third pixel (i, j) in s1_warp i,j express Figure 1 The fusion weight at the third pixel (i, j), W s2_warp i,j express Figure 2 The fusion weight at the third pixel (i, j).

[0079] P s1_warp i,j express Figure 1 The pixel mean of the area where the third pixel point (i, j) is located (such as the 3*3block where the third pixel point (i, j) is located); s2_warp i,j express Figure 2 The pixel mean value of the area where the third pixel point (i, j) is located (such as the 3*3 block where the third pixel point (i, j) is located), Value_Mid is the pixel mean value of the reference frame image, and α3 is W s1_warp i,j The weight coefficient of W s2_warp i,j The weight coefficient, σ 2 is the pixel variance of the benchmark reference frame image. α3 and α4 can be preset, such as written into the chip of the terminal according to requirements.

[0080] For the fourth pixel point in the fused image, the fourth pixel point is in the non-exposed motion area in the base reference frame image, and its pixel value is determined by the pixel value of the fourth pixel point in the base reference frame image according to the fourth strategy indication. The coordinates of the fourth pixel point (i, j) in the fused image in the base reference frame image remain unchanged, and are also (i, j).

[0081] Optionally, the pixel value of the fourth pixel point (i, j) in the fused image is equal to the pixel value of the fourth pixel point (i, j) in the benchmark reference frame image.

[0082] Furthermore, optionally, in this embodiment, the fusion weight is associated with a brightness weight, a saturation weight, and a contrast weight.

[0083] That is, for the kth image, W k,i,j =(C k,i,j )*(S k,i,j )*(E k,i,j ), where W k,i,j represents the fusion weight of pixel (i, j) in the kth image, C k,i,j represents the contrast weight of pixel (i, j) in the kth image, S k,i,j represents the saturation weight of pixel (i, j) in the kth image, E k,i,j Represents the brightness weight of pixel (i, j) in the kth image.

[0084] In this way, exposure intensity, long-short exposure ratio and motion area are taken into consideration in the weight design process. The weight value of each pixel in each image is comprehensively calculated through three indicators (gradient, saturation and brightness) to achieve the purpose of reducing fusion anomalies.

[0085] Optionally, in this embodiment, the brightness weight is determined by a channel brightness weight of the third image in each transmission channel;

[0086] The saturation weight is determined by a pixel standard deviation of the third image in each transmission channel;

[0087] The contrast weight is determined by the channel contrast weight of the third image in each transmission channel.

[0088] Thus, for the third image, i.e., the kth image, E k,i,j Based on the brightness information of a single pixel, the image is processed by channels. When the channel pixel value is higher than the maximum value corresponding to the image bit or close to 0, it is likely to be in a severely underexposed or overexposed area. When the channel pixel value is near the middle value of the maximum value corresponding to the image bit, it can often be considered to be a well-exposed area. The information of these areas should be retained as much as possible during the fusion process. Therefore,

[0089]

[0090] Among them, E x k,i,j represents the channel brightness weight of the pixel (i, j) in the kth image in the transmission channel x, I represents the value of the pixel in the kth image, Value_Mid is the maximum value of the image bit corresponding to the middle value, α5 x represents the first adjustable parameter in the transmission channel x, ε2 represents the second adjustable parameter, σ1 x2 represents the variance in the transmission channel x. Assuming that the kth image is a four-channel (RGGB) image, then E k,i,j =E R k,i,j *E G1 k,i,j *E G2 k,i,j *E B k,i,j , E R k,i,j represents the channel brightness weight of the pixel (i, j) in the kth image in the transmission channel R, E G1 k,i,j represents the channel brightness weight of the pixel (i, j) in the kth image in the transmission channel G1, E G2 k,i,j represents the channel brightness weight of the pixel (i, j) in the kth image in the transmission channel G2, E B k,i,j Represents the channel brightness weight of the pixel (i, j) in the kth image in the transmission channel B.

[0091] The saturation weight S k,i,j , for overexposure or severe underexposure, the saturation of both is close to 0, and the areas with high saturation are often the areas that need to be retained, measured by the pixel standard deviation of the channel. Assuming that the kth image is a four-channel image, then

[0092]

[0093] represents the pixel value of the pixel point (i, j) in the kth image in the transmission channel R, represents the pixel value of the pixel point (i, j) in the kth image in the transmission channel G1, represents the pixel value of pixel point (i, j) in the kth image in transmission channel G2, Represents the pixel value of pixel point (i, j) in the kth image in transmission channel B.

[0094] In addition, the contrast weight C k,i,j Mainly based on the texture details and contour features of the long and short exposure images, C k,i,j It can also be expressed as C k In overexposed images or severely underexposed areas, texture and contour features are basically zero, and these are also areas to be discarded in exposure fusion. Underexposed areas lack details and brightness. One way is to use the Laplacian operator to extract the gradient information (i.e., contrast) of the image. The smaller the gradient information, the smaller the proportion, and vice versa.

[0095] C x k =|Δgrad(I x k )|;C x k represents the channel contrast weight of the k-th image in the transmission channel x, I x k represents the pixel value of the kth image in the transmission channel x. Assuming that the kth image is a four-channel image, then C k =C R k *C G1 k *C G2 k *C B k , C R k represents the channel contrast weight of the k-th image in the transmission channel R, C G1 k represents the channel contrast weight of the k-th image in the transmission channel G1, C G2 krepresents the channel contrast weight of the k-th image in the transmission channel G2, C B k Represents the channel contrast weight of the k-th image in the transmission channel B.

[0096] Optionally, in this embodiment, the third image may be any one of the first image, the second image, the original image, and the registered image.

[0097] Below, the original image is Figure 4 As shown, short exposure image 1 (short frame 1), long exposure image 2 (long frame 2), and short exposure image 3 (short frame 3) are taken as examples to illustrate the application of the image processing method of the embodiment of the present application.

[0098] Optionally, before using the original image, brightness alignment preprocessing can be performed. Specifically, short frame 1 and short frame 3 are subjected to noise reduction and brightness adjustment, and long frame 2 can be directly subjected to optical flow estimation. In the pixel registration process, optical flow estimation is first performed to obtain an optical flow motion vector map, and then registration is performed. In this way, fusion processing can be performed subsequently, including area judgment (determining the exposed static area, non-exposed static area, exposed motion area, and non-exposed motion area of ​​the baseline reference frame image), fusion index conversion (determining brightness weight, saturation weight, and contrast weight), fusion weight determination (determined by brightness weight, saturation weight, and contrast weight), and fused image calculation (determining the pixel value of the pixel in the fused image).

[0099] It should be known that in this embodiment, in order to enhance the details and texture of the output image and filter out the abnormal points of the fusion weights during the fusion process, a pyramid decomposition synthesis method is used in the fusion process to generate the final fused image. Specifically, Laplace pyramid decomposition is used for the base reference frame image and the registered image, and Gaussian pyramid decomposition is used for the weight map (image corresponding to the fusion weight) in order to filter out the abnormal points of the fusion weights. The Laplace pyramid is decomposed from the base reference frame image and the registered image of different exposures, and the Gaussian pyramid is decomposed from the corresponding weight map, and fused at each scale to obtain the fused Laplace pyramid. Upsampling starts from the top layer of Laplace, superimposing the Laplace details of the same scale, and then upsampling and superimposing the details, and recursively calculating until the highest resolution HDR fused image.

[0100] The method of the embodiment of the present application is applicable to HDR video processing and can achieve:

[0101] 1) Calculate the fusion weight map of each channel based on the four-channel data of the RAW domain image RGGB;

[0102] 2) The exposure intensity, long-short exposure ratio and motion area are taken into consideration in the weight design process, and the fusion weight of each pixel in each image is comprehensively calculated through three indicators (gradient, saturation and brightness);

[0103] 3) Use Laplacian pyramid and Gaussian pyramid to decompose the fusion weight map and the image to be fused layer by layer from high level to bottom level, extract the detailed texture of the image to be fused at each layer, and filter out abnormal weight values ​​in the weight map. Then superimpose the fusion results layer by layer from bottom level to top level until the target HDR fusion size is restored;

[0104] 4) Make full use of the performance of the terminal device SoC CPU, SoC GPU, and SoC NPU, call the SoC CPU / GPU for weight calculation in the HDR fusion process, and use chip linkage to improve the real-time performance of HDR video output.

[0105] It should be noted that in this embodiment, the hardware components of the terminal shown are as follows Figure 5 As shown, it includes (a) audio / video system, such as: camera (image sensor), audio equipment, display equipment; (b) chip processing system, such as central processing unit (CPU), neural network processor (NPU), graphics processing unit (GPU); (c) memory, such as system memory, local cache, etc. (d) communication bus. Of course, other hardware parts such as communication interface can also be included. Software components include kernel layer, system layer, application framework layer, and application layer. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor executes instructions in the memory, combined with its hardware completion.

[0106] The camera converts the optical image into an electronic signal (image Raw data), and the processor (CPU, NPU, GPU) is used for system coordination, scheduling, management, image-related processing flow and AI neural network processing flow, providing systematic computing support for the algorithm model. The system memory is used to temporarily store the calculation data in the CPU and exchange cache data with the CPU. The local cache is a buffer for CPU data exchange, which is used to solve the speed difference problem between the CPU speed and the memory speed. The processor, memory, and communication structure are connected through a communication bus; the memory is used to store execution instructions, the processor executes calculation instructions, and the required previous and next frame image data. The display device is used to present the digital image signal on the display to ensure the normal operation of the LCD display. The chip processing system is responsible for processing the data captured by the camera in the HDR video (High Dynamic Range Video Module-Lite, HDR-VM-Lite) module to output HDR-fused RGB images and HDR videos. The details of the HDR-MV-LITE module are as follows: Figure 6 As shown in the figure, it mainly includes four submodules: (a) HDR video frame acquisition submodule; (b) HDR single frame synthesis submodule; (c) storage submodule; (d) HDR video single frame processing submodule. The HDR-MV-LITE module inputs continuous long and short exposure RAW domain images and outputs RAW domain HDR synthesis images. The HDR video acquisition strategy (the strategy used by the image sensor to acquire images) is located in the HDR video frame acquisition module.

[0107] The HDR-VM-Lite module is used for terminal photo taking, video recording, video calling, live broadcasting and other functions.

[0108] Among them, the HDR single-frame synthesis submodule includes (a) optical flow estimation unit 1, (b) long and short exposure registration unit, and (c) long and short exposure synthesis unit.

[0109] The optical flow estimation unit 1 is used to estimate the optical flow motion vector between the long and short exposure frames.

[0110] The long-short exposure registration unit is used to align the pixels of the long-short exposure images according to the optical flow motion information, and can be calculated using the following formula I(x',y')=I(x+Δx,y+Δy). Where I represents the pixel value, (x',y') represents the pixel position of the target image; (x,y) represents the pixel position of the original image, and (Δx,Δy) represents the optical flow motion vector calculated by the optical flow estimation unit 1.

[0111] The long and short exposure synthesis unit processes the long and short exposure images respectively and outputs a fusion weight map (fusion weight corresponding image). The fusion weight can be calculated based on factors such as exposure ratio, pixel brightness, regional similarity, and time similarity, and can also be output through an AI neural network. In this embodiment, the fusion weight is associated with the brightness weight, saturation weight, and contrast weight.

[0112] Among them, the HDR video single frame processing submodule includes an optical flow estimation unit 2, a preprocessing unit, an HDR multi-frame feature extraction unit, an HDR multi-frame feature association unit, an HDR single-frame feature generation unit, and an artifact elimination unit. The optical flow estimation unit 2 can use a method similar to that of the optical flow estimation unit 1. The optical flow estimation result is used for HDR multi-frame feature fusion. After extracting the HDR video frame features, an alignment operation is performed on the feature layer. Finally, a jitter-free, color-stable HDR video is output.

[0113] Among them, the HDR video frame acquisition submodule automatically adjusts the fusion strategy, acquisition interval, and sensitivity according to the light intensity of the external environment and the movement of objects to ensure the optimal front-end long and short frame input in different scenarios.

[0114] In one embodiment, the HDR-VM-Lite module is input as a RAW domain image, which is directly acquired by the front-end camera sensor. The pixel value is proportional to the exposure time, brightness, etc., and is linear data. Most traditional optical flow estimation is applied to RGG domain nonlinear feature images after demosaicing, and is not applicable to RAW domain linear feature images. In addition, compared with the RGB domain, the pixel arrangement distribution in the Raw domain is RGGB, so the Raw domain pixels have a higher value on the G (green) channel, and the pixel values ​​on the B (blue) and R (red) channels are relatively low. The image processing method of the embodiment of the present application is used to improve the accuracy of inter-frame optical flow estimation and reduce HDR video fusion anomalies on the basis of ensuring the real-time performance of the algorithm, thereby improving the user's terminal shooting and reading experience.

[0115] Specifically, when a user uses a terminal to enable HDR video mode, such as clicking Figure 7 The HDR mode function button shown in the figure, the terminal camera will capture video frame images with a fixed frame rate and alternating long and short exposures. After that, the HDR video processing flow is as follows Figure 8 As shown:

[0116] The terminal can determine at least one fusion strategy and input a group of images into the HDR single-frame synthesis submodule. The HDR single-frame synthesis submodule selects alternating long and short exposure frames from the DDR (a) memory for corresponding HDR single-frame fusion; at the same time, the optical flow estimation unit 1 in the HDR single-frame synthesis submodule determines the optical flow motion vector map, and the calculation result is stored in the DDR (c) memory; the HDR single-frame synthesis submodule or the long and short exposure synthesis unit applies the image processing method of the embodiment of the present application, and saves one copy of the fusion result (fused image) in the DDR (b) memory for subsequent HDR video single-frame processing submodule use, and directly inputs one copy into the HDR video single-frame processing submodule. The HDR video single-frame processing submodule reads the original long and short exposure data, optical flow data (such as optical flow motion vector map), HDR fusion picture, and finally outputs an HDR picture that is consistent with the texture, color, motion and other features of the historical frame as a frame in the HDR output video.

[0117] In the HDR video processing flow, optical flow calculation runs on the SoC GPU, and other functions of the HDR single-frame synthesis submodule are implemented on the SoC CPU; other functions of the HDR video single-frame processing submodule are implemented on the SoC NPU. The performance of each SoC chip is fully utilized to output HDR video data in real time while ensuring the stability of HDR video.

[0118] In this way, the real-time HDR video output function can be completed on the terminal side, combining the optical flow information of the HDR video frame and relying on the AI ​​neural network model to improve the front and back stability of the HDR video frame, so as to achieve real-time online HDR video output. This solution can be used to reduce the jitter phenomenon of HDR video, and HDR video truly reflects the color range of real scenes.

[0119] Or, specifically, after using the terminal for HDR shooting, the user can set the HDR offline processing function. After the setting is completed, the terminal background starts the HDR video synthesis effect when the current process task is not busy. At this time, the original long and short exposure data collected by the front-end camera has been cleared from the DDR, so it is necessary to perform frame insertion processing on the synthesized HDR video, and at the same time call the optical flow model to calculate the optical flow vector information of the front and back frames of the HDR video. The offline processing process is as follows Fig. 9 As shown:

[0120] The HDR video is first deframed, and the result is stored in the DDR (a) memory; the deframed image enters the optical flow estimation unit 1 to output the optical flow motion vector information (such as the optical flow motion vector diagram) between the previous and next frames; the HDR video single frame processing submodule is called, and the HDR video single frame processing submodule performs self-training on the HDR video. After the training is completed, the trained model is used to read the HDR deframe data to regenerate each frame in the HDR video and remove the HDR video jitter. In this way, the HDR video offline output function can be completed on the terminal side, and self-training can be performed on the HDR video to make the model more compatible with the HDR video content and generate a more stable HDR video.

[0121] It should be noted that the terminal uses the CPU to send a control strategy to allocate corresponding GPU resources for the optical flow calculation of each block, and parallel calculation accelerates the optical flow calculation results of the picture to predict the optical flow motion vector field between long and short exposures. Taking the use of Lucas-Kanade for optical flow estimation as an example, before calculating the optical flow motion information of long and short exposures (such as the optical flow motion vector map), the two images need to be converted into a unified grayscale image, and then the optical flow information of the two images is calculated according to the Lucas-Kanade method. After obtaining the optical flow motion vector information, the pixel point A of the previous frame can be moved to a new position according to the motion vector, and the pixel point B of the current frame can be moved to the next frame position according to the motion vector.

[0122] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, an image processing device executing the image processing method is taken as an example to illustrate the image processing device provided in the embodiment of the present application.

[0123] like Fig.10 As shown, an image processing device 1000 according to an embodiment of the present application includes:

[0124] A first processing module 1010 is used to obtain an original image, an optical flow motion vector map corresponding to the original image, and a registered image;

[0125] The second processing module 1020 is used to determine different regions of the base reference frame image according to the optical flow motion vector map; wherein different regions belong to different region types, and the region types include an exposed static region, a non-exposed static region, an exposed motion region, and a non-exposed motion region; the base reference frame image is a symmetrical center image in the original image;

[0126] The third processing module 1030 is used to determine a target strategy corresponding to each of the regions according to the region type;

[0127] The fourth processing module 1040 is used to fuse the original image and the registered image according to the target strategy to obtain a fused image.

[0128] Optionally, the second processing module is further used for:

[0129] Determining a moving area and a stationary area of ​​the benchmark reference frame image based on the optical flow motion vector diagram;

[0130] Based on a preset pixel threshold, an exposure area and a non-exposure area of ​​the benchmark reference frame image are determined.

[0131] Optionally, the fourth processing module is further used for:

[0132] Based on the target pixel and the target strategy corresponding to the area to which the target pixel belongs, calculation is performed according to the pixel and fusion weight indicated by the target strategy to obtain a fusion pixel value of the target pixel;

[0133] Wherein, the target pixel point is any pixel point in the benchmark reference frame image;

[0134] The target strategy is a first strategy, a second strategy, a third strategy or a fourth strategy, the first strategy corresponds to the exposed static area, the second strategy corresponds to the non-exposed static area, the third strategy corresponds to the exposed motion area, and the fourth strategy corresponds to the non-exposed motion area.

[0135] Optionally, the first strategy indicates that a pixel value of a first pixel point in the fused image is determined by a first pixel value and a first weight; wherein the first pixel value is a pixel value of the first pixel point in a first image, the first weight is a fusion weight of the first image at the first pixel point, and the first image includes one or more images selected from the original image;

[0136] The second strategy indicates that the pixel value of the second pixel point in the fused image is determined by the second pixel value and the second weight; wherein the second pixel point is the pixel value of the second pixel point in each of the original image and the registered image, and the second weight is the fusion weight of each of the original image and the registered image at the second pixel point;

[0137] The third strategy indicates that the pixel value of the third pixel point in the fused image is determined by a third pixel value and a third weight; wherein the third pixel value is the pixel value of the third pixel point in the second image, the third weight is the fusion weight of the second image at the third pixel point, and the second image includes one or more images selected from the registered image;

[0138] The fourth strategy indicates that the pixel value of the fourth pixel point in the fused image is determined by the pixel value of the fourth pixel point in the baseline reference frame image.

[0139] Optionally, the fusion weight is associated with a brightness weight, a saturation weight and a contrast weight.

[0140] Optionally, the brightness weight is determined by a channel brightness weight of the third image in each transmission channel;

[0141] The saturation weight is determined by a pixel standard deviation of the third image in each transmission channel;

[0142] The contrast weight is determined by the channel contrast weight of the third image in each transmission channel.

[0143] The image processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0144] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0145] The image processing device provided in the embodiment of the present application can achieve Figure 1-Figure 9 To avoid repetition, the various processes implemented in the embodiment of the image processing method are not described again here.

[0146] Alternatively, if Fig.11As shown, an embodiment of the present application further provides an electronic device 1100, including a processor 1101 and a memory 1102, wherein the memory 1102 stores programs or instructions that can be executed on the processor 1101, and when the program or instructions are executed by the processor 1101, the various steps of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0147] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0148] Fig.12 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0149] The electronic device 1200 includes but is not limited to: a radio frequency unit 1201, a network module 1202, an audio output unit 1203, an input unit 1204, a sensor 1205, a display unit 1206, a user input unit 1207, an interface unit 1208, a memory 1209, and a processor 1210 and other components.

[0150] Those skilled in the art will appreciate that the electronic device 1200 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1210 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.12 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0151] The processor 1210 is used to obtain an original image, an optical flow motion vector map and a registered image corresponding to the original image;

[0152] Determine different regions of a base reference frame image according to the optical flow motion vector map; wherein different regions belong to different region types, and the region types include an exposed static region, a non-exposed static region, an exposed motion region, and a non-exposed motion region; the base reference frame image is a symmetrical center image in the original image;

[0153] Determine, according to the area type, a target strategy corresponding to each area;

[0154] The original image and the registered image are fused according to the target strategy to obtain a fused image.

[0155] Optionally, the processor 1210 is further configured to:

[0156] Determining a moving area and a stationary area of ​​the benchmark reference frame image based on the optical flow motion vector diagram;

[0157] Based on a preset pixel threshold, an exposure area and a non-exposure area of ​​the benchmark reference frame image are determined.

[0158] Optionally, the processor 1210 is further configured to:

[0159] Based on the target pixel and the target strategy corresponding to the area to which the target pixel belongs, calculation is performed according to the pixel and fusion weight indicated by the target strategy to obtain a fusion pixel value of the target pixel;

[0160] Wherein, the target pixel point is any pixel point in the benchmark reference frame image;

[0161] The target strategy is a first strategy, a second strategy, a third strategy or a fourth strategy, the first strategy corresponds to the exposed static area, the second strategy corresponds to the non-exposed static area, the third strategy corresponds to the exposed motion area, and the fourth strategy corresponds to the non-exposed motion area.

[0162] Optionally, the first strategy indicates that a pixel value of a first pixel point in the fused image is determined by a first pixel value and a first weight; wherein the first pixel value is a pixel value of the first pixel point in a first image, the first weight is a fusion weight of the first image at the first pixel point, and the first image includes one or more images selected from the original image;

[0163] The second strategy indicates that the pixel value of the second pixel point in the fused image is determined by a second pixel value and a second weight; wherein the second pixel value is the pixel value of the second pixel point in each of the original image and the registered image, and the second weight is the fusion weight of each of the original image and the registered image at the second pixel point;

[0164] The third strategy indicates that the pixel value of the third pixel point in the fused image is determined by a third pixel value and a third weight; wherein the third pixel value is the pixel value of the third pixel point in the second image, the third weight is the fusion weight of the second image at the third pixel point, and the second image includes one or more images selected from the registered image;

[0165] The fourth strategy indicates that the pixel value of the fourth pixel point in the fused image is determined by the pixel value of the fourth pixel point in the baseline reference frame image.

[0166] Optionally, the fusion weight is associated with a brightness weight, a saturation weight and a contrast weight.

[0167] Optionally, the brightness weight is determined by a channel brightness weight of the third image in each transmission channel;

[0168] The saturation weight is determined by a pixel standard deviation of the third image in each transmission channel;

[0169] The contrast weight is determined by the channel contrast weight of the third image in each transmission channel.

[0170] It should be understood that in the embodiment of the present application, the input unit 1204 may include a graphics processor (Graphics Processing Unit, GPU) 12041 and a microphone 12042, and the graphics processor 12041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1206 may include a display panel 12061, and the display panel 12061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1207 includes a touch panel 12071 and at least one of other input devices 12072. The touch panel 12071 is also called a touch screen. The touch panel 12071 may include two parts: a touch detection device and a touch controller. Other input devices 12072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0171] The memory 1209 can be used to store software programs and various data. The memory 1209 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1209 may include a volatile memory or a non-volatile memory, or the memory 1209 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1209 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0172] The processor 1210 may include one or more processing units; optionally, the processor 1210 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1210.

[0173] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0174] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0175] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0176] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0177] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0178] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0179] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0180] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: Executed by the terminal, including: Acquire an original image, as well as an optical flow motion vector map and a registration image corresponding to the original image; Determine different regions of a base reference frame image according to the optical flow motion vector map; wherein different regions belong to different region types, and the region types include an exposed static region, a non-exposed static region, an exposed motion region, and a non-exposed motion region; the base reference frame image is a symmetrical center image in the original image; Determine, according to the area type, a target strategy corresponding to each area; The original image and the registered image are fused according to the target strategy to obtain a fused image.

2. The method according to claim 1, characterized in that Determining different regions of the benchmark reference frame image according to the optical flow motion vector map includes: Determining a moving area and a stationary area of ​​the benchmark reference frame image based on the optical flow motion vector diagram; Based on a preset pixel threshold, an exposure area and a non-exposure area of ​​the benchmark reference frame image are determined.

3. The method according to claim 1, characterized in that The fusing the original image and the registered image according to the target strategy to obtain a fused image includes: Based on the target pixel and the target strategy corresponding to the area to which the target pixel belongs, calculation is performed according to the pixel and fusion weight indicated by the target strategy to obtain a fusion pixel value of the target pixel; Wherein, the target pixel point is any pixel point in the benchmark reference frame image; The target strategy is a first strategy, a second strategy, a third strategy or a fourth strategy, the first strategy corresponds to the exposed static area, the second strategy corresponds to the non-exposed static area, the third strategy corresponds to the exposed motion area, and the fourth strategy corresponds to the non-exposed motion area.

4. The method according to claim 3, characterized in that The first strategy indicates that the pixel value of the first pixel point in the fused image is determined by a first pixel value and a first weight; wherein the first pixel value is the pixel value of the first pixel point in the first image, the first weight is the fusion weight of the first image at the first pixel point, and the first image includes one or more images selected from the original image; The second strategy indicates that the pixel value of the second pixel point in the fused image is determined by a second pixel value and a second weight; wherein the second pixel value is the pixel value of the second pixel point in each of the original image and the registered image, and the second weight is the fusion weight of each of the original image and the registered image at the second pixel point; The third strategy indicates that the pixel value of the third pixel point in the fused image is determined by a third pixel value and a third weight; wherein the third pixel value is the pixel value of the third pixel point in the second image, the third weight is the fusion weight of the second image at the third pixel point, and the second image includes one or more images selected from the registered image; The fourth strategy indicates that the pixel value of the fourth pixel point in the fused image is determined by the pixel value of the fourth pixel point in the baseline reference frame image.

5. The method according to claim 4, characterized in that The fusion weight is associated with a brightness weight, a saturation weight, and a contrast weight.

6. The method according to claim 5, characterized in that The brightness weight is determined by the channel brightness weight of the third image in each transmission channel; The saturation weight is determined by a pixel standard deviation of the third image in each transmission channel; The contrast weight is determined by the channel contrast weight of the third image in each transmission channel.

7. An image processing device, characterized in that: include: A first processing module is used to obtain an original image, an optical flow motion vector map corresponding to the original image, and a registration image; A second processing module is used to determine different regions of a base reference frame image according to the optical flow motion vector map; wherein different regions belong to different region types, and the region types include an exposed static region, a non-exposed static region, an exposed motion region, and a non-exposed motion region; and the base reference frame image is a symmetrical center image in the original image; A third processing module is used to determine a target strategy corresponding to each of the areas according to the area type; The fourth processing module is used to fuse the original image and the registered image according to the target strategy to obtain a fused image.

8. The device according to claim 7, characterized in that The second processing module is also used for: Determining a moving area and a stationary area of ​​the benchmark reference frame image based on the optical flow motion vector diagram; Based on a preset pixel threshold, an exposure area and a non-exposure area of ​​the benchmark reference frame image are determined.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 6 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image processing method according to any one of claims 1 to 6 are implemented.

Images