Image processing method and device, electronic equipment and computer readable storage medium

Abstract

The invention relates to an image processing method and device, electronic equipment, a storage medium and a computer program product. The method comprises the following steps: by taking a first motion area of a first image as a reference, carrying out local alignment on a second motion area of a second image to obtain a third motion area; the exposure duration of the first image is less than the exposure duration of the second image; and fusing the third motion region into the first motion region of the first image to obtain a first target image. By adopting the method, the accuracy of motion area fusion can be improved.

Description

Technical Field

[0001] This application relates to the field of imaging (Camera Technology), and particularly to an image processing method, apparatus, electronic device, and computer-readable storage medium. Background Art

[0002] With the development of imaging technology, in order to obtain clearer and better-quality images, imaging devices such as mobile phones and cameras usually capture at least two images simultaneously and fuse the at least two images to generate a better-quality image. For example, the imaging device can use the HDR (High-Dynamic Range) algorithm to fuse multiple images with different dynamic ranges to obtain a high-dynamic range image.

[0003] However, traditional image processing methods usually only perform simple stitching and fusion based on the exposed area and non-exposed area, resulting in inaccurate fusion of moving areas. Summary of the Invention

[0004] Embodiments of this application provide an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of moving area fusion.

[0005] In a first aspect, this application provides an image processing method. The method includes:

[0006] Taking the first moving area of the first image as a reference, locally aligning the second moving area of the second image to obtain a third moving area; the exposure duration of the first image is less than the exposure duration of the second image;

[0007] Fusing the third moving area into the first moving area of the first image to obtain a first target image.

[0008] In a second aspect, this application further provides an image processing apparatus. The apparatus includes:

[0009] An alignment module, configured to take the first moving area of the first image as a reference and locally align the second moving area of the second image to obtain a third moving area; the exposure duration of the first image is less than the exposure duration of the second image;

[0010] A fusion module, configured to fuse the third moving area into the first moving area of the first image to obtain a first target image.

[0011] In a third aspect, this application further provides an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0012] Taking the first motion area of the first image as a reference, locally align the second motion area of the second image to obtain a third motion area; the exposure duration of the first image is less than the exposure duration of the second image;

[0013] Fuse the third motion area into the first motion area of the first image to obtain a first target image.

[0014] Fourthly, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0015] Taking the first motion area of the first image as a reference, locally align the second motion area of the second image to obtain a third motion area; the exposure duration of the first image is less than the exposure duration of the second image;

[0016] Fuse the third motion area into the first motion area of the first image to obtain a first target image.

[0017] Fifthly, the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0018] Taking the first motion area of the first image as a reference, locally align the second motion area of the second image to obtain a third motion area; the exposure duration of the first image is less than the exposure duration of the second image;

[0019] Fuse the third motion area into the first motion area of the first image to obtain a first target image.

[0020] For the above image processing method, device, electronic device, computer-readable storage medium and computer program product, the electronic device takes the first motion area of the first image as a reference, locally aligns the second motion area of the second image to obtain a third motion area, wherein the exposure duration of the first image is less than the exposure duration of the second image, that is, the first image is a short exposure frame and the second image is a long exposure frame. The short exposure frame, i.e., the first image, can avoid the ghosting problem, and the long exposure frame, i.e., the second image, has better image quality. Then, fusing the third motion area into the first motion area of the first image can improve the image quality of the first motion area in the first image, so as to obtain a first target image that avoids the ghosting problem and has higher image quality, improving the accuracy of motion area fusion. Description of the Drawings

[0021] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0022] Figure 1 It is a flowchart of an image processing method in an embodiment;

[0023] Figure 2 It is a regional comparison diagram of fusing short-exposure frames and long-exposure frames in an embodiment;

[0024] Figure 3 It is a schematic diagram of an image obtained by using a traditional image processing method in an embodiment;

[0025] Figure 4 It is a schematic diagram of a first target image in an embodiment;

[0026] Figure 5 It is a schematic diagram of a target weight map in an embodiment;

[0027] Figure 6 It is a schematic diagram of a sigmoid function curve in an embodiment;

[0028] Figure 7 It is a schematic diagram of a confidence map before mapping processing using the sigmoid function in an embodiment;

[0029] Figure 8 It is a schematic diagram of a first weight map in an embodiment;

[0030] Figure 9 It is a schematic diagram of a second weight map in an embodiment;

[0031] Figure 10 It is a schematic diagram of a third weight map in an embodiment;

[0032] Figure 11 It is a schematic diagram of an image obtained by using a traditional image processing method in another embodiment;

[0033] Figure 12 It is a schematic diagram of a second target image in an embodiment;

[0034] Figure 13 It is an architecture diagram of an image processing method in an embodiment;

[0035] Figure 14 It is a schematic diagram of a first image in an embodiment;

[0036] Figure 15 Schematic diagram of a second image in an embodiment;

[0037] Figure 16 Schematic diagram of a first image after global alignment in an embodiment;

[0038] Figure 17 Schematic diagram of a fused image obtained by fusing a first image and a second image after global alignment in an embodiment;

[0039] Figure 18 Schematic diagram of a second image after local alignment in an embodiment;

[0040] Figure 19 Schematic diagram of a fused image obtained by fusing a first image after global alignment and a second image after local alignment in an embodiment;

[0041] Figure 20 Schematic diagram of a first target image in an embodiment;

[0042] Figure 21 Flowchart of image processing in another embodiment;

[0043] Figure 22 Block diagram of the structure of an image processing apparatus in an embodiment;

[0044] Figure 23 Internal structure diagram of an electronic device in an embodiment. Detailed implementation manners

[0045] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0046] In one embodiment, as Figure 1 shown, an image processing method is provided. In this embodiment, it is exemplified that the method is applied to an electronic device. The electronic device may be a terminal or a server. It can be understood that the method can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal may be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices may be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, smart cars, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server may be implemented by an independent server or a server cluster composed of multiple servers.

[0047] In this embodiment, the image processing method includes the following steps:

[0048] Step S102: Taking the first motion area of the first image as a reference, locally align the second motion area of the second image to obtain a third motion area; the exposure duration of the first image is less than that of the second image.

[0049] Among them, the exposure duration of the first image is less than that of the second image, that is, the first image is a short-exposure frame and the second image is a long-exposure frame. The brightness of the first image is less than that of the second image.

[0050] Optionally, both the first image and the second image can be input images with any bit width (bit). For example, 8-bit corresponds to a data range of (0-255), and 10-bit corresponds to a data range of (0-1023), etc. The first image can be an original short-exposure frame or an image obtained by downsampling the original short-exposure frame; similarly, the first image can be an original long-exposure frame or an image obtained by downsampling the original long-exposure frame, and it is not limited to this.

[0051] It can be understood that when the electronic device takes pictures through the camera module, the camera module performs exposure to generate an image. The longer the exposure duration, the more light the camera module obtains, and the brighter the image, that is, the exposure duration and the image brightness are positively correlated. However, when the camera module takes pictures of a moving object, if the exposure duration is long, the motion areas in the obtained image information are inconsistent, which may cause ghosting in the picture; if the exposure duration is short, less image information is obtained, which may result in poor image quality. As Figure 2 shown, the short-exposure frame is fused on the left side of the image, and the long-exposure frame is fused on the right side of the image. Therefore, the image quality of the left area of the image is poor, and the image quality of the right area of the image is good.

[0052] The motion area is the area where the moving object is located in the image. The first motion area is the area where the moving object is located in the first image, the second motion area is the area where the moving object is located in the second image, and the third motion area is the area where the moving object is located in the locally aligned second image.

[0053] Optionally, the electronic device performs exposure with different exposure parameters to obtain the first image and the second image; the exposure duration of the first image is less than that of the second image. Among them, the exposure parameters include the exposure duration, and can also include at least one of the light source and the sensitivity. The first image and the second image are adjacent in the shooting time sequence.

[0054] Local alignment (local warp, local registration) refers to the alignment between local areas of an image, that is, the fine alignment of the texture between the first motion area and the second motion area, which is used to improve the image quality and the accuracy of feature extraction.

[0055] The electronic device uses the first motion area of the first image as a reference to locally align the second motion area of the second image, and can perform operations such as displacement and distortion on the second motion area to align it to the first motion area, thereby obtaining a third motion area.

[0056] Using the first motion area of the first image as a reference to locally align the second motion area of the second image to obtain a third motion area includes: using the first motion area of the first image as a reference to determine the displacement information between the second motion area of the second image and the first motion area of the first image; based on the displacement information, locally align the second motion area of the second image to obtain a third motion area. This displacement information is the local displacement information.

[0057] Among them, the displacement information can be the optical flow information between the second motion area and the first motion area.

[0058] Optionally, through an artificial intelligence optical flow network (Ai flow network), determine the optical flow information between the second motion area of the second image and the first motion area of the first image.

[0059] The electronic device inputs the first image and the second image into the artificial intelligence optical flow network, calculates the optical flow information between the second motion area of the second image and the first motion area of the first image through the artificial intelligence optical flow network, and outputs this optical flow information.

[0060] Optionally, the electronic device displaces the pixels in the second motion area of the second image according to the displacement information to obtain a third motion area.

[0061] Step S104, fuse the third motion area into the first motion area of the first image to obtain a first target image.

[0062] Among them, the first target image is the image obtained by fusing the third motion area into the first image.

[0063] Optionally, the first motion area includes at least one of a first sub-area and a second sub-area; the brightness of the first sub-area is greater than a first brightness threshold, the brightness of the second sub-area is less than a second brightness threshold, and the first brightness threshold is greater than the second brightness threshold.

[0064] Among them, both the first brightness threshold and the second brightness threshold can be set as needed.

[0065] Optionally, the exposure amount of the first sub-area is greater than a first exposure amount threshold, the exposure amount of the second sub-area is less than a second exposure amount threshold, and the first exposure amount threshold is greater than the second exposure amount threshold, that is, the first sub-area is an overexposed area and the second sub-area is a non-overexposed area. Among them, both the first exposure amount threshold and the second exposure amount threshold can be set as needed.

[0066] It can be understood that if the first motion area includes a first sub - area and a second sub - area, it means that the first motion area is between the over - exposed area and the non - over - exposed area, and the first motion area contains both a part of the over - exposed area and a part of the non - over - exposed area; if the first motion area includes the first sub - area, it means that the first motion area is entirely in the over - exposed area; if the first motion area includes the second sub - area, it means that the first motion is entirely in the non - over - exposed area.

[0067] Optionally, the electronic device fuses the image information (texture information) of the third motion area into the first motion area of the first image to obtain a first target image.

[0068] Optionally, the electronic device fuses the third motion area into the first motion area of the first image based on the fusion weight corresponding to the third motion area to obtain a first target image.

[0069] Optionally, the electronic device performs an averaging process on the third motion area and the first motion area to obtain a fused motion area, and updates the first motion area of the first image based on the fused motion area to obtain a first target image.

[0070] In an alternative embodiment, the electronic device can cover the first motion area of the first image with the fused motion area to obtain a first target image. In another alternative embodiment, the electronic device can replace the first motion area of the first image with the fused motion area to obtain a first target image.

[0071] Optionally, the electronic device can also perform image signal processing (ISP, Image Signal Processor) on the first target image to output a final target image. Among them, the image signal processing includes processing such as brightening and tone mapping.

[0072] In the above - mentioned image processing method, the electronic device uses the first motion area of the first image as a reference to locally align the second motion area of the second image to obtain a third motion area. Among them, the exposure duration of the first image is less than that of the second image, that is, the first image is a short - exposure frame and the second image is a long - exposure frame. The short - exposure frame, that is, the first image, can avoid the ghosting problem, and the long - exposure frame, that is, the second image, has better image quality. Then, by fusing the third motion area into the first motion area of the first image, defects in the image (artifacts, such as abnormal textures like small breaks and small misalignments) can be removed to improve the image quality of the first motion area in the first image, so as to obtain a first target image that avoids the ghosting problem and has higher image quality, improving the accuracy of motion area fusion.

[0073] Moreover, since there may be overexposed areas in the second motion area of the second image with a longer exposure time, resulting in the entire second motion area being a white maximum value and thus unable to be accurately aligned, the electronic device uses the first motion area of the first image with a shorter exposure time as a reference to more accurately perform local alignment on the second motion area of the second image.

[0074] Moreover, the above image processing method can effectively improve the image completion rate for backlight scenes or non-backlight scenes, day scenes or night scenes, portrait scenes or non-portrait scenes, etc., and there are obvious benefits in image quality performance; backlight scenes or non-backlight scenes can include various shooting objects, such as people, animals, buildings or landscapes, etc.

[0075] In one embodiment, the number of first images can be one or at least two, and the number of second images can also be one or at least two.

[0076] Optionally, if the electronic device is in a night scene, the first image with the longest exposure time is determined from at least two first images as a reference, and the third motion area is fused into the first image with the longest exposure time to obtain a first target image.

[0077] It can be understood that usually less image information is obtained in a night scene, and as a short exposure frame, the first image has a poor signal-to-noise ratio. The first image with the longest exposure time is the one that obtains the most image information. Therefore, when the electronic device fuses the third motion area into the first image with the longest exposure time, a first target image with better image quality can be obtained.

[0078] Exemplarily, when the electronic device is in a night scene, ev0 (long exposure frame), ev12 (short exposure frame), and ev24 (even shorter exposure frame) are captured. The second motion area of ev0 is locally aligned with the first motion area of ev12 as a reference to obtain a third motion area; the third motion area is fused into the first motion area of ev12 to obtain a first target image.

[0079] Optionally, if the electronic device is in a day scene, one of the first images is determined from at least two first images as a reference for local alignment, and the second motion area of each second image is locally aligned respectively to obtain a third motion area; each third motion area is fused into the first motion area of the reference first image to obtain a first target image.

[0080] In an alternative implementation, the electronic device can fuse each third motion area into the first motion area of the reference first image on average, or fuse it into the first motion area of the reference first image according to different weights, which is not limited here.

[0081] In one embodiment, as Figure 3 shown, the electronic device uses a traditional image processing method, and the quality of the generated target image is poor and there is ghosting; as Figure 4 shown, for the same original image, the electronic device uses the above image processing method, and the generated first target image can eliminate ghosting and improve the image quality.

[0082] In one embodiment, it is characterized in that fusing the third motion area into the first motion area of the first image to obtain a first target image includes: obtaining a target weight map; the target weight map includes the fusion weights corresponding to the pixels in the third motion area; based on the fusion weights corresponding to the pixels in the third motion area, fusing the third motion area into the first motion area to obtain a first target image; the fusion weight corresponding to each pixel in the third motion area is positively correlated with the image information amount of the third motion area included in the corresponding pixel in the first target image.

[0083] Wherein, the target weight map is a weight map for fusing the third motion area and the first motion area. Each pixel in the target weight map represents the fusion weight corresponding to each pixel in the second image after local alignment. The target weight map includes the fusion weights corresponding to each pixel in the third motion area, and may also include the fusion weights corresponding to each pixel in the area other than the third motion area in the second image after local alignment. The larger the fusion weight, the more the image information amount of the third motion area included in the pixel corresponding to the fusion weight in the first target image.

[0084] Optionally, for each pixel pair in each third motion area and the first motion area, determine the target pixel of the pixel pair according to the fusion weight of the pixel pair in the third motion area and the difference weight of the pixel pair in the first motion area; generate the target motion area of the first target image according to each target pixel. Wherein, the sum of the fusion weight and the difference weight is 1.

[0085] Optionally, for each pixel pair in each third motion area and the first motion area, multiply the pixel information of the pixel pair in the third motion area by the fusion weight to obtain a first product; multiply the pixel information of the pixel pair in the first motion area by the difference weight to obtain a second product; add the first product and the second product to obtain the target pixel.

[0086] Optionally, the electronic device calculates the first target image using the following formula:

[0087]

[0088] Wherein, is the first target image, is the second image after local alignment, is the first image, is the target weight map.

[0089] In this embodiment, the electronic device will fuse the third motion area into the first motion area based on the fusion weights corresponding to each pixel in the third motion area to obtain the first target image. And the fusion weights corresponding to the pixels in the third motion area are positively correlated with the image information amount of the third motion area included in the corresponding pixels in the first target image, which can generate the first target image more accurately.

[0090] In one embodiment, obtaining the target weight map includes: obtaining at least two of the first weight map, the second weight map, and the third weight map; the first weight map characterizes the confidence of the displacement information between the first image and the second image, the second weight map characterizes the brightness information of the first image, and the third weight map characterizes the verification result of the motion area; generating the target weight map based on at least two of the first weight map, the second weight map, and the third weight map.

[0091] Among them, the first weight map, the second weight map, and the third weight map can all be normalized images, and the first weight in the first weight map, the second weight in the second weight map, and the third weight in the third weight map are between 0 and 1. The displacement information between the first image and the second image can be the optical flow information between the first image and the second image. The optical flow information can be an optical flow field.

[0092] Optionally, the electronic device multiplies at least two of the first weight map, the second weight map, and the third weight map to generate the target weight map.

[0093] In an alternative embodiment, the electronic device obtains the first weight map, the second weight map, and the third weight map, and multiplies the first weight map, the second weight map, and the third weight map to obtain the target weight map.

[0094] Exemplarily, the electronic device can generate the target weight map using the following formula: mask = (a) * (b) * (c); where mask is the target weight map, (a) is the first weight map, (b) is the second weight map, and (c) is the third weight map. As Figure 5 shown in the target weight map, the fusion weights in the white areas are larger, and the fusion weights in the black areas are smaller; that is, the whiter the color, the larger the corresponding fusion weight, and the blacker the color, the smaller the corresponding fusion weight. Figure 5 The fusion weight of the white portrait area in [description] is greater than 0.85, while the fusion weight of the arm area with relatively large motion is less than 0.1.

[0095] In another alternative embodiment, the electronic device can also obtain the first weight map and the second weight map, and multiply the first weight map and the second weight map to obtain the target weight map.

[0096] In another alternative embodiment, the electronic device may also obtain a second weight map and a third weight map, multiply the second weight map and the third weight map to obtain a target weight map.

[0097] Optionally, the electronic device may also generate the target weight map in other ways, such as adding or averaging at least two of the first weight map, the second weight map, and the third weight map, etc., and is not limited thereto.

[0098] In this embodiment, the electronic device obtains at least two of the first weight map, the second weight map, and the third weight map, and the first weight map, the second weight map, and the third weight map respectively represent different dimensions. By using at least two weight maps of different dimensions, the target weight map can be generated more accurately.

[0099] In one embodiment, the method for obtaining the first weight map includes: obtaining a confidence map of the displacement information between a first image and a second image; generating the first weight map based on the confidence map; the first weight in the first weight map is positively correlated with the corresponding confidence in the confidence map.

[0100] Wherein, the confidence in the confidence map (conf) represents the reliability of the displacement information between the first image and the second image. The displacement information between the first image and the second image includes the displacement information between the second motion area and the first motion area, that is, the displacement information of local alignment. The higher the confidence, the higher the reliability of the displacement information corresponding to the confidence.

[0101] It can be understood that due to moving objects, noise, texture occlusion, etc. in the image, when the electronic device fuses the third motion area into the first motion area of the first image, the confidence map can be used as an important reference to determine whether to give a large fusion weight based on the confidence map.

[0102] Optionally, the electronic device determines the displacement information between the second motion area of the second image and the first motion area of the first image, and the confidence map of the displacement information through an artificial intelligence optical flow network.

[0103] Optionally, the electronic device determines the first weight corresponding to each confidence in the confidence map according to the corresponding relationship between the confidence and the weight, and generates the first weight map based on each first weight. Wherein, the corresponding relationship between the confidence and the weight is a positive correlation.

[0104] Optionally, generating the first weight map based on the confidence map includes: respectively performing erosion processing, blurring processing, size adjustment, and mapping processing on the confidence map to generate the first weight map.

[0105] Among them, the erosion process (erode), which is also the morphological erosion operation, is used to eliminate noise in the image, disconnect connected objects, etc. The blur process is used to eliminate noise or details in the image, make the image smoother, and avoid sudden changes in data. The resizing can be to increase or decrease the size. The electronic device can increase the size by upsampling and decrease the size by downsampling.

[0106] Optionally, the mapping process can be performed using the sigmoid function. Among them, the sigmoid function can enhance the contrast of the confidence map. As Figure 6 shown in the sigmoid function curve graph. As Figure 7 shown in the confidence map before the electronic device performs the mapping process using the sigmoid function. After the electronic device performs the mapping process using the sigmoid function, the white and black areas in the confidence map can be enlarged, and the gray area can be reduced to enhance the contrast in the confidence map, obtaining the confidence map as Figure 8 , which is also the first weight map.

[0107] Optionally, the electronic device processes the confidence map in the following order:

[0108] erode cv::Size(7, 7)-->blur cv::Size(5, 5)-->resize-->sigmod

[0109] Among them, (7, 7) is the parameter of the erosion processor, and Size(5, 5) is the parameter of the blur processor.

[0110] In this embodiment, the electronic device obtains the confidence map of the displacement information between the first image and the second image. The confidence corresponding in this confidence map is positively correlated with the first weight in the first weight map. Therefore, based on the confidence map, the first weight map can be accurately generated. Further, by respectively performing the erosion process, blur process, resizing process, and mapping process on the confidence map, the first weight map can be generated more accurately.

[0111] In one embodiment, the method for obtaining the second weight map includes: obtaining the brightness information of the first image; generating the second weight map based on the brightness information of the first image; the second weight in the second weight map is negatively correlated with the corresponding brightness information in the first image.

[0112] Optionally, the electronic device determines the second weight corresponding to the brightness information of each pixel in the first image according to the correspondence between brightness and weight; and generates a second weight map based on each second weight. Wherein, the correspondence between brightness and weight is negatively correlated, that is, the greater the brightness of the pixel in the first image, the smaller the second weight corresponding to the pixel.

[0113] Optionally, the electronic device obtains a brightness image corresponding to the first image; and generates a second weight map based on the brightness image. As Figure 9 shown in the second weight map, if the image brightness of the first image is darker, the second weight of the corresponding area in the second weight map is larger, that is, the white area in the second weight map; if the image brightness is brighter, the second weight of the corresponding area in the second weight map is smaller, that is, the black area in the second weight map.

[0114] It can be understood that the greater the brightness of the pixel in the first image of the short exposure frame, the greater the brightness of the corresponding pixel in the second image after local alignment of the long exposure frame, which belongs to the overexposed area. For the overexposed area, more image information of the short exposure frame is used during image fusion, that is, the second weight of the overexposed area in the second image after local alignment is smaller.

[0115] Optionally, generating a second weight map based on the brightness information of the first image includes: using the second image after local alignment as a reference, performing brightness alignment on the brightness information of the first image to obtain the first image after brightness alignment; generating a second weight map based on the brightness information of the first image after brightness alignment; the second weight in the second weight map is negatively correlated with the corresponding brightness information in the first image after brightness alignment.

[0116] Optionally, based on the first image and the exposure ratio, perform brightness alignment on the brightness information of the first image to obtain the first image after brightness alignment; wherein, the exposure ratio represents the brightness relationship between the first image and the second image, and also represents the brightness relationship between the first image and the second image after local alignment.

[0117] Optionally, the electronic device multiplies the brightness information of the first image by the exposure ratio to obtain the first image after brightness alignment.

[0118] Optionally, the electronic device determines the second weight corresponding to the brightness information of each pixel in the first image after brightness alignment according to the correspondence between brightness and weight; and generates a second weight map based on each second weight. Wherein, the correspondence between brightness and weight is negatively correlated, that is, the greater the brightness of the pixel in the first image after brightness alignment, the smaller the second weight corresponding to the pixel.

[0119] Optionally, the electronic device uses the second image as a reference to globally align the first image to obtain the globally aligned first image; based on the luminance information of the globally aligned first image, a second weight map is generated.

[0120] Optionally, the electronic device performs thresholding and normalization on the luminance-aligned first image respectively to obtain a second weight map. For thresholding, for example, pixels with pixel values greater than 255 in the luminance-aligned first image are set to 255; normalization can set the pixel values in the luminance-aligned first image within the range of 0 - 1.

[0121] Optionally, the electronic device processes the globally aligned first image in the following order:

[0122] ev12*ratio-->thres-->Normalization

[0123] where ev12 is the globally aligned first image, ratio is the luminance relationship between the first image and the second image, and thres is the thresholding process.

[0124] Taking the case where both the first image and the second image are 12-bit data, with a maximum value of 4096 and a black point flat value of 256, when AE (Automatic Exposure) is completed, the exposure time (exp time) and exposure gain (exp gain) of the first image or the second image will be output. The electronic device determines the luminance relationship between the first image and the second image using the following formula:

[0125]

[0126] where is the luminance relationship between the first image and the second image, is the exposure time of the first image, is the exposure gain of the first image, is the exposure time of the second image, is the exposure gain of the second image.

[0127] Align the luminance of the first image with the second image as a reference:

[0128]

[0129]

[0130] where is the output second image, is the input second image, that is, the second image remains unchanged; is the output first image, is the input first image, is the black level value of the first image, is the luminance relationship between the first image and the second image.

[0131] In this embodiment, the electronic device acquires the luminance information of the first image. The corresponding luminance information in the first image and the second weight in the second weight map are negatively correlated. Then, based on the luminance information of the first image, the second weight map can be accurately generated. Further, taking the second image after local alignment as a reference, the luminance information of the first image is luminance-aligned to obtain the first image after luminance alignment, that is, the first image within the normal luminance range. Then, based on the luminance information of the first image after luminance alignment, the second weight map can be generated more accurately.

[0132] In one embodiment, the method for obtaining the third weight map includes: obtaining a difference image between the first image and the second image after local alignment; the difference pixels in the difference image are the differences between the pixels in the second image after local alignment and the corresponding pixels in the first image; generating a third weight map based on the difference image; the third weight in the third weight map is negatively correlated with the corresponding difference pixels in the difference image.

[0133] Optionally, the electronic device performs a difference process on the first image and the second image after local alignment to obtain a difference image between the first image and the second image after local alignment.

[0134] As Figure 10 shown in the third weight map, the black area represents the area with a large difference between the first image and the second image after local alignment, which is determined as the area of dynamic movement of the moving object (the movement existing in the moving object itself) or the occluded area, so the third weight is small; the white area represents the area with a small difference between the first image and the second image after local alignment, which is determined as the area of non-dynamic movement of the non-moving object or the non-occluded area, so the third weight is large.

[0135] Optionally, if the difference pixel is greater than the target difference threshold, it indicates that the movement area corresponding to the difference pixel is the area of dynamic movement of the moving object; if the difference pixel is less than or equal to the target difference threshold, it indicates that the movement area corresponding to the difference pixel is the area of non-dynamic movement.

[0136] Among them, the target difference threshold can be set as needed.

[0137] It can be understood that if the difference pixel is greater than the target difference threshold, it indicates that the movement in the area where the difference pixel between the first image and the second image after local alignment is located is large. And the second image after local alignment is locally aligned based on the first image. That is to say, this difference pixel is the area where the moving object moves dynamically, that is, the area where the moving object itself moves, and the movement cannot be eliminated through local alignment. If the difference pixel is less than or equal to the target difference threshold, it indicates that the movement in the area where the difference pixel between the first image and the second image after local alignment is located is small. It can be considered that the area where this moving pixel is located is the area where the non-moving object moves dynamically, that is, the area of non-dynamic movement.

[0138] Further, if the difference pixel is large, it indicates that the difference between the first image and the second image after local alignment at the corresponding position of this difference pixel is large. If the pixel corresponding to this difference pixel in the second image after local alignment is fused into the first image, a ghosting problem will occur. Therefore, the third weight corresponding to this difference pixel is small. Similarly, if the difference pixel is small, it indicates that the difference between the first image and the second image after local alignment at the corresponding position of this difference pixel is small. The pixel corresponding to this difference pixel in the second image after local alignment can be fused into the first image without generating a ghosting problem, and the third weight corresponding to this difference pixel is large. That is to say, the third weight in the third weight map is negatively correlated with the corresponding difference pixel in the difference image.

[0139] Optionally, the electronic device performs thresholding, size adjustment, and normalization on the difference image respectively to obtain the third weight map.

[0140] Optionally, the electronic device processes the difference image in the following order:

[0141] abs diff-->thres-->resize-->Normalization

[0142] Among them, abs diff is the difference image, thres is thresholding, and resize is size adjustment.

[0143] In this embodiment, the electronic device obtains the difference image between the first image and the second image after local alignment. Then, based on this difference image, the third weight map can be generated more accurately.

[0144] In one embodiment, the above method further includes: generating a second target image according to at least one of the second image and the second image after local alignment, and the first target image.

[0145] Optionally, the electronic device fuses at least one of the second image and the second image after local alignment, and the first target image to generate the second target image.

[0146] Optionally, the electronic device performs HDR fusion on at least one of the second image and the locally aligned second image, and the first target image to generate a second target image. The second target image is an HDR image.

[0147] It can be understood that both the second image and the locally aligned second image are short exposure frames, while the first target image is a long exposure frame. The electronic device fuses the long exposure frame and the short exposure frame, and the obtained second target image contains both the image information of the short exposure frame and the image information of the long exposure frame, expanding the dynamic range of the second target image.

[0148] Optionally, the electronic device can also perform image signal processing on the second target image to output a final target image. The image signal processing includes processing such as brightening and tone mapping.

[0149] In this embodiment, the electronic device can generate a second target image with richer image information and a wider dynamic range based on at least one of the second image and the locally aligned second image, and the first target image.

[0150] In one embodiment, as Figure 11 shown, when the electronic device uses traditional image processing methods, the quality of the generated target image is poor and there are ghosts; as Figure 12 shown, for the same original image, when the electronic device uses the above image processing method, the generated second target image can eliminate ghosts, improve the image quality, and can increase the dynamic range of the image, containing more image information with a wider dynamic range.

[0151] In one embodiment, before using the first motion area of the first image as a reference to locally align the second motion area of the second image to obtain a third motion area, it further includes: using the second image as a reference to globally align the first image to obtain a globally aligned first image; using the first motion area of the first image as a reference to locally align the second motion area of the second image to obtain a third motion area, including: using the first motion area of the globally aligned first image as a reference to locally align the second motion area of the second image to obtain a third motion area.

[0152] Global alignment (glocal warp, global registration) refers to the alignment between the globals of two images, that is, the rough alignment of the textures between the first image and the second image.

[0153] Optionally, the electronic device uses the second image as a reference to perform brightness alignment on the first image to obtain a brightness-aligned first image; using the second image as a reference to perform global alignment on the brightness-aligned first image to obtain a globally aligned first image.

[0154] Optionally, the electronic device removes the black level of the first image to obtain the first image after black level removal; determines the brightness relationship between the first image and the second image, and aligns the brightness of the first image to the second image based on this brightness relationship to obtain the first image with aligned brightness. Here, the first image and the second image can be original RAW data, and the brightness relationship between the first image and the second image can be the exposure ratio between the first image and the second image.

[0155] Optionally, the electronic device downsamples the second image and the first image with aligned brightness to obtain the second image and the first image with a target size; inputs the second image and the first image with the target size into an artificial intelligence optical flow network to output the global displacement information (CV optical flow information) between the second image and the first image; takes the second image as a reference, and globally aligns the first image according to this global displacement information to obtain the first image after global alignment. Here, the target size is the size required for the artificial intelligence optical flow network to process the image, usually a smaller size, which can save computing resources.

[0156] Optionally, the electronic device takes the second image as a reference to determine the global displacement information between the second image and the first image; based on the global displacement information, globally aligns the first image to obtain the first image after global alignment. Here, the global displacement information can be the optical flow information (cv optical flow) between the second image and the first image.

[0157] Optionally, through an artificial intelligence optical flow network (Ai flow network), determine the local displacement information between the second motion area of the second image and the first motion area of the first image after global alignment; take the first motion area of the first image after global alignment as a reference, and locally align the second motion area of the second image according to the local displacement information to obtain a third motion area. Here, the local displacement information can be the optical flow information between the second motion area of the second image and the first motion area of the first image after global alignment.

[0158] Optionally, fusing the third motion area into the first motion area of the first image to obtain a first target image includes: fusing the third motion area into the first motion area of the first image after global alignment to obtain a first target image.

[0159] Optionally, the electronic device downsamples the first image after global alignment to obtain the downsampled first image; downsamples the second image to obtain the downsampled second image; takes the first motion area of the downsampled first image as a reference, and locally aligns the second motion area of the downsampled second image to obtain a third motion area.

[0160] In this embodiment, the electronic device uses the second image as a reference to globally align the first image to obtain the globally aligned first image, and then uses the first motion area of the globally aligned first image as a reference to locally align the second motion area of the second image, so as to accurately obtain the third motion area.

[0161] In one embodiment, as Figure 13 shown is the architecture diagram of the image processing method. The electronic device uses the second image as a reference to globally align the first image to obtain the globally aligned first image; if the electronic device fuses the second image and the globally aligned first image, an image without ghosting can be obtained by fusion.

[0162] The electronic device inputs the globally aligned first image and the second image into an artificial intelligence optical flow network to determine the local displacement information between the first motion area of the globally aligned first image and the second motion area of the second image, and the confidence map of the local displacement information; uses the globally aligned first image as a reference to locally align the second motion area of the second image according to the local displacement information to obtain the third motion area; obtains the brightness information of the first image and the difference image between the first image and the locally aligned second image; generates a first weight map based on the confidence map, generates a second weight map based on the brightness information of the first image, and generates a third weight map based on the difference image; generates a target weight map based on the first weight map, the second weight map, and the third weight map, and the target weight map includes the fusion weight corresponding to each pixel in the third motion area; fuses the third motion area into the first motion area of the globally aligned first image based on the fusion weight corresponding to each pixel in the third motion area of the locally aligned second image to obtain the first target image.

[0163] In one embodiment, referring to Figures 14 to 20 , the electronic device captures a first image as Figure 14 shown, and captures a second image as Figure 15 shown. The exposure duration of the first image is less than that of the second image. There is a height difference in the positions of people in the first image and the second image, indicating global motion, and there are differences in the arms of people, indicating local motion; when the electronic device captures images, the left side of the vertical line in the first image and the second image represents the overexposed area, and the right side of the vertical line is the non-overexposed area. During image fusion, the overexposed area is incorporated into the short-exposure frame, and the non-overexposed area is fused with the long-exposure frame.

[0164] The electronic device uses the second image as a reference to globally align the first image, that is, aligns the position of the person in the first image to the position of the person in the second image to obtain the globally aligned first image, that is Figure 16; however, there is local movement in the arm area of the first image after global alignment compared to the second image. If the electronic device fuses the first image and the second image after global alignment, the resulting fused image is as shown in Figure 17 as follows. Figure 17 In the fused image of, there will be a ghost problem in the area of the human arm with local movement.

[0165] The electronic device uses the first motion area of the first image after global alignment as a reference to determine the displacement information between the second motion area of the second image and the first motion area of the first image after global alignment; based on the displacement information, the second motion area of the second image is locally aligned to obtain the second image after local alignment, and the second image after local alignment includes a third motion area. The second image after local alignment is as shown in Figure 18 as follows. If the electronic device fuses the first image after global alignment and the second image after local alignment, the resulting fused image is as shown in Figure 19 as follows; since the human arms in the first image after global alignment and the second image after local alignment are already aligned, Figure 19 the ghost problem in the human arm area can be removed from the fused image of.

[0166] In the Figure 19 fused image, in the circular range where the motion area of the arm is located, to solve the ghost problem, a short exposure frame (the first image after global alignment) is fused, thereby reducing the image quality of the motion area of the arm in the Figure 19 fused image. Therefore, the electronic device fuses the long exposure frame (the third motion area of the second image after local alignment) into the first motion area of the first image after global alignment to obtain a first target image; the first target image is as shown in Figure 20 as follows. The first image not only eliminates the ghost problem generated in image fusion but also avoids the problem of poor image quality caused by only the first motion area of the first image in the motion area of the arm, improving the image quality of the image motion area.

[0167] In one embodiment, as shown in Figure 21As shown, the electronic device exposes with a first exposure duration to obtain a first image, and exposes with a second exposure duration to obtain a second image, where the first exposure duration is less than the second exposure duration; using the second image as a reference, optical flow registration is performed on the first image and the second image to calculate the global displacement information between the first image and the second image; according to this global alignment information, the first image is globally aligned to obtain the globally aligned first image; the globally aligned first image is luminance-aligned to obtain the luminance-aligned first image; the second image and the luminance-aligned first image are used to calculate the displacement information of the moving region and the confidence map of the displacement information; the second image is locally aligned to obtain the locally aligned second image; based on the luminance-aligned first image and the locally aligned second image, a target weight map is calculated, which is obtained based on the confidence map, the luminance information of the luminance-aligned first image, and the difference image between the first image and the locally aligned second image, and the target weight map includes the fusion weight corresponding to each pixel in the third moving region; based on the fusion weight corresponding to each pixel in the third moving region, the third moving region of the locally aligned second image is fused into the first moving region of the luminance-aligned first image to obtain a first target image; the first target image and the second image are HDR-fused to obtain a second target image.

[0168] In one embodiment, another image processing method is also provided, which is applied to an electronic device. The image processing method includes the following steps:

[0169] Step A1, using the second image as a reference, globally align the first image to obtain the globally aligned first image; the exposure duration of the first image is less than that of the second image.

[0170] Step A2, using the first moving region of the globally aligned first image as a reference, determine the displacement information between the second moving region of the second image and the first moving region of the first image; based on the displacement information, locally align the second moving region of the second image to obtain a third moving region; the first moving region includes at least one of a first sub-region and a second sub-region, the luminance of the first sub-region is greater than a first luminance threshold, the luminance of the second sub-region is less than a second luminance threshold, and the first luminance threshold is greater than the second luminance threshold.

[0171] The electronic device executes at least two of Step A3, Step A4, and Step A5 to obtain at least two of a first weight map, a second weight map, and a third weight map; the first weight map characterizes the confidence of the displacement information between the globally aligned first image and the second image, the second weight map characterizes the luminance information of the globally aligned first image, and the third weight map characterizes the verification result of the moving region.

[0172] Step A3: Obtain a confidence map of the displacement information between the first image and the second image after global alignment; perform erosion processing and blurring processing on the confidence map respectively to generate a first weight map; the first weight in the first weight map is positively correlated with the corresponding confidence in the confidence map.

[0173] Step A4: Obtain the luminance information of the first image after global alignment; using the second image after local alignment as a reference, perform luminance alignment on the luminance information of the first image after global alignment to obtain the first image after luminance alignment; based on the luminance information of the first image after luminance alignment, generate a second weight map; the second weight in the second weight map is negatively correlated with the corresponding luminance information in the first image after luminance alignment.

[0174] Step A5: Obtain a difference image between the first image after global alignment and the second image after local alignment; the difference pixels in the difference image are the differences between the pixels in the second image after local alignment and the corresponding pixels in the first image after global alignment; based on the difference image, generate a third weight map; the third weight in the third weight map is negatively correlated with the corresponding difference pixels in the difference image; if the difference pixel is greater than the target difference threshold, it indicates that the motion area corresponding to the difference pixel is the area where the moving object moves dynamically; if the difference pixel is less than or equal to the target difference threshold, it indicates that the motion area corresponding to the difference pixel is a non-dynamic motion area.

[0175] The electronic device continues to execute steps A6 to A8.

[0176] Step A6: Generate a target weight map based on at least two of the first weight map, the second weight map, and the third weight map; the target weight map includes the fusion weights corresponding to each pixel in the third motion area.

[0177] Step A7: Based on the fusion weights corresponding to each pixel in the third motion area, fuse the third motion area into the first motion area of the first image after global alignment to obtain a first target image; the fusion weights corresponding to each pixel in the third motion area are positively correlated with the image information amount of the third motion area included in the corresponding pixel in the first target image.

[0178] Step A8: Generate a second target image according to at least one of the second image and the second image after local alignment, and the first target image.

[0179] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0180] Based on the same inventive concept, an embodiment of the present application further provides an image processing apparatus for implementing the above-mentioned image processing method. The solution provided by this apparatus to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the following image processing apparatus can refer to the limitations on the image processing method in the above text, and will not be repeated here.

[0181] In one embodiment, as Figure 22 shown, an image processing apparatus is provided, including: an alignment module 2202 and a fusion module 2204, where:

[0182] The alignment module 2202 is configured to locally align the second motion region of the second image with the first motion region of the first image to obtain a third motion region; the exposure duration of the first image is less than the exposure duration of the second image.

[0183] The fusion module 2204 is configured to fuse the third motion region into the first motion region of the first image to obtain a first target image.

[0184] For the above-mentioned image processing apparatus, the electronic device locally aligns the second motion region of the second image with the first motion region of the first image to obtain a third motion region, where the exposure duration of the first image is less than the exposure duration of the second image, that is, the first image is a short exposure frame and the second image is a long exposure frame. The short exposure frame, that is, the first image, can avoid the ghosting problem, and the long exposure frame, that is, the second image, has better image quality. Then, fusing the third motion region into the first motion region of the first image can improve the image quality of the first motion region in the first image, so as to obtain a first target image that avoids the ghosting problem and has higher image quality, and improve the accuracy of motion region fusion.

[0185] In one embodiment, the above-mentioned fusion module 2204 is further configured to obtain a target weight map; the target weight map includes the fusion weights corresponding to the pixels in the third motion area; based on the fusion weights corresponding to the pixels in the third motion area, fuse the third motion area into the first motion area to obtain a first target image.

[0186] In one embodiment, the above-mentioned fusion module 2204 is further configured to obtain at least two of a first weight map, a second weight map, and a third weight map; the first weight map represents the confidence of the displacement information between the first image and the second image, the second weight map represents the brightness information of the first image, and the third weight map represents the verification result of the motion area; based on at least two of the first weight map, the second weight map, and the third weight map, generate a target weight map.

[0187] In one embodiment, the above-mentioned fusion module 2204 is further configured to obtain a confidence map of the displacement information between the first image and the second image; based on the confidence map, generate a first weight map.

[0188] In one embodiment, the above-mentioned fusion module 2204 is further configured to perform erosion processing, blurring processing, size adjustment, and mapping processing on the confidence map respectively to generate a first weight map.

[0189] In one embodiment, the above-mentioned fusion module 2204 is further configured to obtain the brightness information of the first image; based on the brightness information of the first image, generate a second weight map.

[0190] In one embodiment, the above-mentioned fusion module 2204 is further configured to use the locally aligned second image as a reference to perform brightness alignment on the brightness information of the first image to obtain the first image with brightness alignment; based on the brightness information of the first image with brightness alignment, generate a second weight map.

[0191] In one embodiment, the above-mentioned fusion module 2204 is further configured to obtain a difference image between the first image and the locally aligned second image; based on the difference image, generate a third weight map.

[0192] In one embodiment, if the difference pixel is greater than the target difference threshold, it indicates that the motion area corresponding to the difference pixel is the area where the moving object moves dynamically; if the difference pixel is less than or equal to the target difference threshold, it indicates that the motion area corresponding to the difference pixel is the area of non-dynamic motion.

[0193] In one embodiment, the above-mentioned fusion module 2204 is further configured to generate a second target image according to at least one of the second image and the locally aligned second image, and the first target image.

[0194] In one embodiment, the alignment module 2202 is further configured to globally align the first image with the second image as a reference to obtain the globally aligned first image; and locally align the second motion region of the second image with the first motion region of the globally aligned first image as a reference to obtain a third motion region.

[0195] In one embodiment, the alignment module 2202 is further configured to determine displacement information between the second motion region of the second image and the first motion region of the first image with the first motion region of the first image as a reference; and locally align the second motion region of the second image based on the displacement information to obtain a third motion region.

[0196] In one embodiment, the first motion region includes at least one of a first sub-region and a second sub-region; the brightness of the first sub-region is greater than a first brightness threshold, the brightness of the second sub-region is less than a second brightness threshold, and the first brightness threshold is greater than the second brightness threshold.

[0197] Each module in the above image processing device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor in the electronic device in the form of hardware or be independent of the processor, or can be stored in the memory in the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0198] In one embodiment, an electronic device is provided. The electronic device can be a terminal, and its internal structure diagram can be as Figure 23As shown in the figure. The electronic device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and external devices. The communication interface of the electronic device is used to communicate with external terminals in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an image processing method. The display unit of the electronic device is used to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the electronic device, or an external keyboard, touchpad, or mouse, etc.

[0199] Those skilled in the art can understand that Figure 23 the structure shown in the figure is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the electronic device to which the solution of this application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0200] The embodiment of this application also provides a computer-readable storage medium. One or more non-volatile computer-readable storage media containing computer-executable instructions, when the computer-executable instructions are executed by one or more processors, cause the processors to execute the steps of the image processing method.

[0201] The embodiment of this application also provides a computer program product containing instructions, which when run on a computer, causes the computer to execute the image processing method.

[0202] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0203] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0204] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0205] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. An image processing method, characterized in that, Including: Taking the first motion area of the first image as a reference, locally aligning the second motion area of the second image to obtain a third motion area; the exposure duration of the first image is less than the exposure duration of the second image; Fusing the third motion area into the first motion area of the first image to obtain a first target image.

2. The method according to claim 1, wherein The step of fusing the third motion area into the first motion area of the first image to obtain a first target image includes: Obtaining a target weight map; the target weight map includes the fusion weights corresponding to the pixels in the third motion area; Based on the fusion weights corresponding to the pixels in the third motion area, fusing the third motion area into the first motion area to obtain a first target image.

3. The method according to claim 2, wherein The step of obtaining a target weight map includes: Obtaining at least two of a first weight map, a second weight map, and a third weight map; the first weight map represents the confidence of the displacement information between the first image and the second image, the second weight map represents the brightness information of the first image, and the third weight map represents the verification result of the motion area; Generating a target weight map based on at least two of the first weight map, the second weight map, and the third weight map.

4. The method according to claim 3, characterized in that The method for obtaining the first weight map includes: Obtaining a confidence map of the displacement information between the first image and the second image; Generating a first weight map based on the confidence map.

5. The method according to claim 4, characterized in that, The step of generating a first weight map based on the confidence map includes: Performing erosion processing, blurring processing, size adjustment, and mapping processing on the confidence map respectively to generate a first weight map.

6. The method according to claim 3, characterized in that, The method for obtaining the second weight map includes: Obtaining the brightness information of the first image; Generating a second weight map based on the brightness information of the first image.

7. The method according to claim 6, wherein The step of generating a second weight map based on the brightness information of the first image includes: Taking the locally aligned second image as a reference, performing brightness alignment on the brightness information of the first image to obtain the first image with brightness alignment; Generating a second weight map based on the brightness information of the first image with brightness alignment.

8. The method according to claim 3, characterized in that, The method for obtaining the third weight map includes: Obtaining a difference image between the first image and the locally aligned second image; Generating a third weight map based on the difference image.

9. The method according to claim 8, characterized in that If the difference pixel is greater than the target difference threshold, it indicates that the motion area corresponding to the difference pixel is the area where the moving object moves dynamically; if the difference pixel is less than or equal to the target difference threshold, it indicates that the motion area corresponding to the difference pixel is the area of non-dynamic motion.

10. The method according to any one of claims 1 to 9, characterized in that, The method further includes: Generating a second target image according to at least one of the second image and the locally aligned second image, and the first target image.

11. The method according to any one of claims 1 to 9, characterized in that, Before taking the first motion area of the first image as a reference and locally aligning the second motion area of the second image to obtain a third motion area, it further includes: Taking the second image as a reference and globally aligning the first image to obtain the globally aligned first image; Using the first motion area of the first image as a reference, locally aligning the second motion area of the second image to obtain a third motion area, including: Using the first motion area of the first image after global alignment as a reference, locally aligning the second motion area of the second image to obtain a third motion area.

12. The method according to any one of claims 1 to 9, characterized in that Using the first motion area of the first image as a reference, locally aligning the second motion area of the second image to obtain a third motion area, including: Using the first motion area of the first image as a reference, determining the displacement information between the second motion area of the second image and the first motion area of the first image; Based on the displacement information, locally aligning the second motion area of the second image to obtain a third motion area.

13. The method according to any one of claims 1 to 9, characterized in that The first motion area includes at least one of a first sub-area and a second sub-area; the brightness of the first sub-area is greater than a first brightness threshold, the brightness of the second sub-area is less than a second brightness threshold, and the first brightness threshold is greater than the second brightness threshold.

14. An image processing apparatus, characterized in that, Including: An alignment module, configured to use the first motion area of the first image as a reference, locally align the second motion area of the second image to obtain a third motion area; The exposure duration of the first image is less than the exposure duration of the second image; A fusion module, configured to fuse the third motion area into the first motion area of the first image to obtain a first target image.

15. An electronic device, including a memory and a processor, wherein a computer program is stored in the memory, characterized in that, When the computer program is executed by the processor, the processor is caused to execute the steps of the image processing method according to any one of claims 1 to 13.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images