Image processing method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses an image processing method and device, equipment and a storage medium, and the method comprises the steps: obtaining a first image and a second image which are collected by a camera module at the same time, and enabling the first image and the second image to correspond to different exposures respectively; the first image is used for generating an initial preview image. The first image and the second image are aligned to determine a first cutting area, and the first cutting area is an overlapped image area between the first image and the second image; and cutting the initial preview image according to the first cutting area to obtain a target preview image. And fusing the first image and the second image to obtain an initial fused image. And cutting the initial fusion image according to the first cutting area to obtain a target fusion image. According to the embodiment of the invention, the first cutting area is determined by aligning the first image and the second image, and the target preview image and the target fusion image are obtained by cutting according to the first cutting area, so that the field angles of the two images are ensured to be consistent.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of imaging technology, including but not limited to an image processing method, apparatus, device, and storage medium. Background Art

[0002] Currently, in some image acquisition scenarios, when an electronic device takes a photo, it will first display a preview image to the user, and then process the preview image through an image processing algorithm to obtain a target image for replacement display. In the related art, during the image display process, due to the inconsistent field of view angles of the finally displayed target image and the preview image, there will be jumps or mirror image problems in the image. Summary of the Invention

[0003] The image processing method, apparatus, device, and storage medium provided by the embodiments of the present application can ensure that the field of view angles of the preview image and the generated target fusion image are consistent, and there will be no problems of jumps or mirror images. The image processing method, apparatus, device, and storage medium provided by the embodiments of the present application are implemented as follows:

[0004] On the one hand, an embodiment of the present application provides an image processing method, including:

[0005] Obtaining a first image and a second image simultaneously collected by an imaging module, where the first image and the second image correspond to different exposure amounts respectively; the first image is used to generate an initial preview image;

[0006] Aligning the first image and the second image to determine a first cropping area, where the first cropping area is the overlapping image area between the first image and the second image;

[0007] Cropping the initial preview image according to the first cropping area to obtain a target preview image;

[0008] Fusing the first image and the second image to obtain an initial fusion image;

[0009] Cropping the initial fusion image according to the first cropping area to obtain a target fusion image.

[0010] On the other hand, an embodiment of the present application further provides an image processing apparatus, including:

[0011] An image acquisition module, configured to obtain a first image and a second image simultaneously collected by an imaging module, where the first image and the second image correspond to different exposure amounts respectively; the first image is used to generate an initial preview image;

[0012] A cropping area determination module, configured to align the first image and the second image to determine a first cropping area, where the first cropping area is the overlapping image area between the first image and the second image;

[0013] The first cropping module is configured to crop the initial preview image according to the first cropping area to obtain a target preview image;

[0014] The image fusion module is configured to fuse the first image and the second image to obtain an initial fused image;

[0015] The second cropping module is configured to crop the initial fused image according to the first cropping area to obtain a target fused image.

[0016] The computer device provided by the embodiments of the present application includes a memory and a processor. The memory stores a computer program that can run on the processor, and when the processor executes the program, the method described in the embodiments of the present application is implemented.

[0017] The computer-readable storage medium provided by the embodiments of the present application stores a computer program thereon, and when the computer program is executed by a processor, the method provided by the embodiments of the present application is implemented.

[0018] The computer program product provided by the embodiments of the present application includes a computer program, and when the computer program is executed by a processor in an electronic device, the processor implements the method provided by the embodiments of the present application.

[0019] For the image processing method, device, computer device, and computer-readable storage medium provided by the embodiments of the present application, the first image and the second image simultaneously collected by the imaging module are obtained. The first image and the second image respectively correspond to different exposure amounts. The first image is used to generate an initial preview image. The first image and the second image are aligned to determine the first cropping area, and the first cropping area is the overlapping image area between the first image and the second image. The initial preview image is cropped according to the first cropping area to obtain a target preview image. The first image and the second image are fused to obtain an initial fused image. The initial fused image is cropped according to the first cropping area to obtain a target fused image. In this way, the image processing method can determine the first cropping area by aligning the first image and the second image, and crop the target preview image and the target fused image according to the first cropping area, ensuring that the field of view angles of the two images are consistent and avoiding problems such as jumping or mirroring of the target preview image and the target fused image displayed before and after. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0021] Figure 1A schematic diagram showing the change in the viewing angle field of a displayed image according to an embodiment of the present application;

[0022] Figure 2 A flowchart showing an image processing method according to an embodiment of the present application;

[0023] Figure 3 A schematic diagram showing a first cropping area according to an embodiment of the present application;

[0024] Figure 4 A schematic diagram showing a second cropping area or a third cropping area according to an embodiment of the present application;

[0025] Figure 5 A flowchart showing another image processing method according to an embodiment of the present application;

[0026] Figure 6 A schematic diagram showing an image processing process according to an embodiment of the present application;

[0027] Figure 7 A schematic diagram showing an image processing device according to an embodiment of the present application;

[0028] Figure 8 A schematic diagram showing an electronic device according to an embodiment of the present application. Detailed implementation manners

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will further describe the specific technical solutions of the present application in detail with reference to the accompanying drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application but are not intended to limit the scope of the present application.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0031] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with each other without conflict.

[0032] It should be noted that the terms "first / second / third" involved in the embodiments of the present application are used to distinguish similar or different objects and do not represent a specific order for the objects. It can be understood that "first / second / third" can be interchanged with a specific order or sequence when permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0033] The image processing method of the embodiment of the present application can be executed by an electronic device, which may include but is not limited to a mobile phone, a wearable device (such as a smart watch, a smart bracelet, smart glasses, etc.), a tablet computer, a laptop computer, a vehicle terminal, a PC (Personal Computer), etc. The function implemented by the method can be implemented by calling a program code by a processor in the electronic device, and of course the program code can be stored in a computer storage medium. It can be seen that the electronic device at least includes a processor and a storage medium.

[0034] The image processing method of the embodiment of the present application can be used in any application scenario where different images with the same image content are displayed before and after image processing. For example, during image acquisition, due to excessive ambient light (such as direct sunlight or highly reflective scenes), improper camera settings (such as slow shutter, large aperture, high ISO or high exposure compensation), insufficient sensor dynamic range and inability to take into account light and dark details, the captured image may be overexposed, resulting in poor quality of the captured image. In the above case, the electronic device first selects an original image as a preview image for front-end display, and replaces the current preview image with the processed image obtained after image processing of the original image, and updates the application scenario displayed on the front end.

[0035] Alternatively, for images with defects such as insufficient dynamic range, motion blur, and insufficient depth of field, the electronic device can first select an original image as a preview image for front-end display, and then replace the current preview image with the processed image after performing image processing on the original image, and update the display on the front end.

[0036] Optionally, the electronic device may process images with problems such as overexposure, insufficient dynamic range, motion blur, and insufficient depth of field by performing image fusion processing on a corrected image captured simultaneously with the original image, so as to correct the defective original image and obtain a target image with better image quality and visual effect.

[0037] Figure 1 FIG. 1 is a schematic diagram showing a change in the viewing angle field of a display image according to an embodiment of the present application. Figure 1 As shown, the left side is the preview image initially displayed by the electronic device, and the right side is the target image obtained by the electronic device after image processing. In some image processing scenarios, the image position will be offset relative to the canvas, resulting in blank areas in the canvas. Currently, the blank areas are usually filled in a mirroring manner, resulting in a difference in the viewing field between the target image obtained by image processing and the preview image, resulting in image jumps or the appearance of mirrored areas.

[0038] Exemplarily, in the case where the image processing method is image fusion, there is a certain motion difference between the original image to be fused and the corresponding corrected image. During the process of image fusion by the electronic device, it is necessary to align the corrected image and the preview image before fusion. In this way, in the target image obtained after fusion, there will be a mirror problem in the area where the corrected image and the preview image do not overlap, resulting in different viewing fields of the target image and the original preview image.

[0039] Based on the above technical problems, the image processing method according to the embodiments of the present application can, before fusing the preview image and the corrected image, first determine a first cropping area according to the area where mirroring will occur after alignment and fusion, and then crop the preview image according to the first cropping area for preview display. Then, perform image fusion processing, and crop the fused image according to the first cropping area to obtain the target fused image for display, ensuring that the viewing angles of the images displayed twice before and after are consistent.

[0040] Figure 2 Shows a flowchart of an image processing method according to an embodiment of the present application. As Figure 2 shown, the job parsing method according to the embodiments of the present application may include the following steps S20 - S24.

[0041] For ease of description, the image processing method according to the embodiments of the present application is described with an electronic device as the execution subject. It should be understood that the execution subject for executing the embodiments of the present application may also be a processor or a chip in the electronic device, etc., and the embodiments of the present application do not make any limitations.

[0042] Step S20: The electronic device acquires a first image and a second image simultaneously collected by the camera module.

[0043] In a possible implementation manner, the electronic device may include a camera module for image acquisition. Among them, during each image acquisition process, the camera module may acquire multiple candidate images with the same exposure amount. And the camera module in the embodiments of the present application may also simultaneously acquire images with different exposure amounts.

[0044] In some embodiments, when the exposure quality of the images acquired by the camera module is poor, such as overexposure, underexposure, or highlight overflow, etc., the camera module may automatically acquire a reference image corresponding to the candidate image with an exposure amount different from that of the candidate image while acquiring each candidate image. The reference image is used for image processing of the candidate image with poor exposure quality. That is, when the exposure quality of the images acquired by the camera module is poor, the camera module may acquire multiple groups of images during the image acquisition process, and each group of images includes a candidate image and a reference image acquired simultaneously.

[0045] Optionally, when the camera module captures multiple sets of images during image acquisition, and each set of images includes a candidate image and a reference image captured simultaneously. The electronic device can screen out a candidate image with the highest image quality from each set of images as the first image, and use the reference image corresponding to the first image as the second image to perform image processing on the first image based on the second image. Among them, the first image and the second image correspond to different exposure amounts respectively.

[0046] In some possible implementation manners, the exposure value corresponding to the first image is 0, and the exposure value corresponding to the second image is less than 0. Among them, the first image can be an image with an Exposure Value 0 (EV0) exposure value of 0. The second image can be an ambient light exposure value image (EVm). Due to different exposure durations, the exposure duration of the first image is relatively longer than that of the second image, and the brightness value of the first image will be higher than that of the second image. Therefore, in the embodiment of the present application, when the first image is overexposed, the second image can be used to correct the first image, and the overexposure problem of the first image can be corrected to improve the image quality.

[0047] In some other embodiments, there is also a case where the exposure duration of the first image is relatively shorter than that of the second image, that is, the brightness value of the first image is lower than that of the second image. In this case, in the embodiment of the present application, when the first image is underexposed, the second image can be used to fill light for the first image, and the underexposure problem of the first image can be corrected to improve the image quality.

[0048] In some embodiments, the electronic device further includes a display module for image display. The first image is used as the basis for the display image, that is, the electronic device can generate an initial preview image according to the first image, and then further process it to obtain a target preview image and then display it through the display module. Among them, the electronic device can crop the first image according to the zoom information selected by the user during image acquisition to obtain the initial preview image.

[0049] Exemplarily, when the zoom factor included in the zoom information is 1 without zooming, or a value greater than 0 and less than 1 such as 0.5 that magnifies the original image acquisition canvas, the size of the first image captured by the electronic device is substantially the same as the initial preview image serving as the display basis, that is, it can be directly determined that the first image is the initial preview image. Alternatively, after performing preset image processing such as image noise reduction, image domain conversion, and stylization on the first image, the initial preview image can be obtained. When the zoom factor included in the zoom information is a value greater than 1 such as 2 or 3 that reduces the original image acquisition canvas, the first image captured by the electronic device is substantially also the image captured without zooming the original image acquisition canvas. The first image can be further magnified by the corresponding multiple according to the zoom size and then cropped based on the image size corresponding to the first image to obtain the initial preview image. Alternatively, after performing preset image processing such as image noise reduction, image domain conversion, and stylization on the first image, the first image can be further magnified by the corresponding multiple according to the zoom size and then cropped based on the image size corresponding to the first image to obtain the initial preview image.

[0050] In some other embodiments, when the exposure quality of the image captured by the camera module is high, that is, there are no exposure problems during the shooting process such as overexposure, underexposure, or highlight overflow. The electronic device can determine that no image processing is required, and the camera module only captures multiple candidate images with the same exposure amount. The electronic device can select a candidate image with the highest image quality as the first image, and based on the zoom information selected by the user during image acquisition, crop the first image directly to obtain the target image, and display the target image through the display module. The target image is the image displayed by the display module after image acquisition and is not replaced after display.

[0051] Step S21: Align the first image and the second image to determine the first cropping region.

[0052] In a possible implementation manner, after determining the first image and the second image, the electronic device can determine the first cropping region by aligning the first image and the second image. Among them, the first cropping region is the overlapping image region after the first image and the second image are aligned.

[0053] Optionally, the process of aligning the first image and the second image in the embodiments of the present application is used to perform geometric matching on a specific area between the two images to make them spatially consistent. Among them, the above alignment method can be pre-alignment between images, that is, not really performing the image alignment action, but only used to determine the expected overlapping image area between the two aligned images after image alignment. Or, the alignment method can also be the actual overlapping image area between the two aligned images after the image alignment action has been performed.

[0054] In some embodiments, the process of aligning the first image and the second image by the electronic device in the embodiments of the present application can be implemented by any image alignment method such as image alignment by identifying image feature points and image alignment based on template matching.

[0055] Exemplarily, the electronic device can perform image alignment by calculating the global motion offset information between the images. That is, the electronic device can calculate the global motion offset information between the second image and the first image. Then, the first image and the second image are aligned according to the global motion offset information, and the first cropping area is determined according to the alignment result. Among them, the global motion offset information between the second image and the first image is used to describe the position change of the second image relative to the first image, and can represent the overall displacement or motion trend of all pixels in the second image relative to the first image in space, usually caused by global geometric transformations such as translation, rotation, and scaling.

[0056] The above method of performing image alignment through global motion offset information requires a small number of parameters to be calculated during the image alignment process, can quickly solve the alignment result, and ensure the real-time nature of the calculation process. At the same time, based on the global offset, the consistency of the overall motion of the image is ensured, effectively filtering out the interference of local motion, and having better robustness.

[0057] In some embodiments, the electronic device can determine the global motion offset information between the second image and the first image by calculating the affine matrix. That is, the electronic device first determines the affine matrix between the first image and the second image. Then, the global motion offset information between the second image and the first image is determined according to the affine matrix. Optionally, the affine matrix can be calculated by various methods such as the optical flow method, the feature point matching method, or the phase correlation method, and represents a global geometric transformation relationship for describing how to map the pixel points in the second image to the corresponding positions in the first image. Thus, the embodiments of the present application can simply, efficiently, and accurately determine the global motion offset information between the second image and the first image based on the affine matrix.

[0058] Exemplarily, the affine matrix in the embodiments of the present application can be calculated by the optical flow method. That is, the electronic device can determine at least one first corner point in the first image. Then, the dense optical flow from the second image to the first image is calculated, and the dense optical flow includes the motion offsets corresponding to each pixel point in the second image. The second corner point corresponding to each first corner point in the second image is determined according to the dense optical flow. The affine matrix between the first image and the second image is determined according to at least one first corner point and the second corner point corresponding to each first corner point.

[0059] Thus, in the embodiments of the present application, by analyzing the motion of the corner points between the first image and the second image to calculate the affine matrix and further estimate the global motion offset between the two images, complex motions can be processed more robustly while maintaining relatively high computational efficiency.

[0060] In a possible implementation manner, the form of the global motion offset information between the first image and the second image in the embodiments of the present application can be the global optical flow. That is, after calculating the affine matrix between the first image and the second image, the electronic device can generate the global optical flow between the first image and the second image based on the affine matrix between the two images. This process is essentially to convert the affine transformation parameters into a dense motion field covering the entire image (i.e., the displacement vector of each pixel).

[0061] In a possible implementation manner, after calculating the global motion offset information between the first image and the second image, the electronic device can align the first image and the second image to determine the overlapping image area after alignment of the two images as the first cropping area.

[0062] In some embodiments, during the image acquisition process, when the image scaling ratio represented by the selected scaling information is 1 or less than 1, the image displayed by the display device of the electronic device is not enlarged relative to the first image. Therefore, the images displayed by the display device of the electronic device are all images with the same content as the first image, and the first image and the second image can be directly aligned according to the global motion offset information to determine the overlapping image area between the first alignment area and the second alignment area, and the overlapping image area is used as the first cropping area.

[0063] In other embodiments, during the image acquisition process, when the image scaling ratio represented by the selected scaling information is 1 or less than 1, the image displayed by the display device of the electronic device is enlarged relative to the first image. Therefore, the images displayed by the display device of the electronic device are all images of the content of a local area in the first image. The electronic device needs to determine the area for display in the first image as the first alignment area, and the area at the corresponding position in the second image as the second alignment area, and then determine the first cropping area based on the first alignment area and the second alignment area.

[0064] Exemplarily, the electronic device may first determine a first alignment region corresponding to the first image and a second alignment region corresponding to the second image according to the scaling information. Then, the first alignment region and the second alignment region are aligned according to the global motion offset information, an overlapping image region between the first alignment region and the second alignment region is determined, and the overlapping image region is used as the first cropping region. The cropping method of the first cropping region can be determined based on the region to be displayed. Considering that only the region to be displayed is aligned during the subsequent fusion process, the region where the two images match after image fusion (i.e., the overlapping region after alignment) can be accurately estimated, and an accurate first cropping region can be obtained.

[0065] For example, when the scaling information is 2, the electronic device can magnify the first image by 2 times starting from the center point and crop the first alignment region based on the original canvas size. At the same time, the second image is magnified by 2 times starting from the center point, and the second alignment region is cropped based on the original canvas size. Then, the first alignment region and the second alignment region are aligned to obtain the first cropping region.

[0066] Figure 3 A schematic diagram of a first cropping region according to an embodiment of the present application is shown. As Figure 3 shown, in the embodiment of the present application, the electronic device determines the first alignment region and the second alignment region of the first image according to the scaling information. Among them, when the scaling ratio indicated by the scaling information is greater than 0 and less than 1, the first alignment region is the first image, and the second alignment region is the second image. When the scaling ratio indicated by the scaling information is greater than 1, the first alignment region is a local region in the first image, and the second alignment region is a local region in the second image.

[0067] The electronic device may further align the first alignment region and the second alignment region according to the global motion offset information between the first image and the second image, and then determine the overlapping region after alignment as the first cropping region.

[0068] Thus, in the embodiment of the present application, the electronic device can calculate the global motion offset information based on the images of the original size, which can ensure that the obtained global motion offset information is more accurate. At the same time, the accurate first cropping region is obtained by obtaining the local regions of the images to be displayed according to the scaling information for image alignment.

[0069] Step S22: Crop the initial preview image according to the first cropping region to obtain a target preview image.

[0070] In a possible implementation, after determining the first cropping area, the electronic device may crop the initial preview image determined according to the first image based on the first cropping area to obtain a target preview image. The target preview image is the image initially displayed to the user by the display device of the electronic device after image acquisition.

[0071] In some embodiments, the size of the image within the first cropping area does not conform to the canvas ratio selected by the user. The canvas ratio is the canvas ratio selected by the user during image acquisition, such as 4:3, 3:4, 16:9, 3:2, etc. After obtaining the initial preview image, the electronic device may determine a second cropping area according to the canvas ratio of the initial preview image and the first cropping area. Crop the initial preview image based on the second cropping area to obtain a target preview image. The size of the image within the second cropping area conforms to the canvas ratio of the initial preview image. After cropping based on the second cropping area, the image area of the initial preview image within the second cropping area is retained, and the image area of the initial preview image outside the second cropping area is removed. Thus, the embodiments of the present application can ensure that the canvas ratios of the target preview image and the initial preview image obtained after image cropping are the same.

[0072] After completing the image cropping to obtain the target preview image, the electronic device may directly display the target preview image to the user through the display device first to ensure the real-time acquisition of the image by the user.

[0073] Figure 4 Shows a schematic diagram of a second cropping area or a third cropping area according to an embodiment of the present application. As Figure 4 shown, when the size ratio within the first cropping area is different from the canvas ratio of the initial preview image, the electronic device needs to adjust the first cropping area according to the size of the first cropping area and the canvas ratio of the initial preview image to obtain a second cropping area with a size ratio the same as the canvas ratio of the initial preview image.

[0074] Exemplarily, when the size ratio of the first cropping area is narrower than the canvas ratio, the first cropping area may be shortened longitudinally to obtain a second cropping area with a shortened size ratio the same as the canvas ratio. Optionally, when the size ratio of the first cropping area is wider than the canvas ratio, the first cropping area may be shortened transversely to obtain a second cropping area with a shortened size ratio the same as the canvas ratio.

[0075] Step S23: Fuse the first image and the second image to obtain an initial fused image.

[0076] In a possible implementation, the electronic device may perform image fusion processing on the first image and the second image to obtain an initial fused image. Among them, the process of image fusion processing is used to correct the first image through the second image to improve the quality of the first image and ensure that the finally displayed image to the user has a better visual effect.

[0077] Optionally, in the image fusion process of the embodiments of the present application, the first image and the second image may be directly fused to obtain an initial fused image. However, in the actual application process, in order to further improve the image quality of the first image, at least one of the first image and the second image needs to be subjected to other related image processing during the image fusion process. If the entire image is fused, the computational amount is large and the computational efficiency is low.

[0078] Therefore, in order to further improve the efficiency of the image fusion process, the electronic device may consider the image area that needs to be finally displayed for image fusion, that is, only fuse the image areas that need to be displayed in the first image and the second image, without considering other areas. In this way, unnecessary calculation processes can be avoided in some scenarios, the computational amount of the image fusion process can be reduced, and the efficiency of the image fusion process can be improved.

[0079] In some embodiments, during the image acquisition process, when the image scaling ratio represented by the selected scaling information is 1 or less than 1, the image displayed by the display device of the electronic device is not enlarged relative to the first image. Therefore, the image displayed by the display device of the electronic device is an image with the same content as the first image, and there is no area that does not need to be calculated in the image, and the first image and the second image can be directly fused to obtain an initial fused image.

[0080] In other embodiments, during the image acquisition process, when the image scaling ratio represented by the selected scaling information is 1 or less than 1, the image displayed by the display device of the electronic device is enlarged relative to the first image. Therefore, the images displayed by the display device of the electronic device are all images of the content of the local area in the first image. In addition to the local area that needs to be displayed in the image, the fusion process will also introduce useless areas that do not need to be calculated. Therefore, the local area that needs to be displayed can be extracted first and then image fusion can be performed.

[0081] Exemplarily, in the above case, the way for the electronic device to perform image fusion can be as follows: determine the first image to be fused according to the scaling information and the first image; determine the second image to be fused according to the scaling information and the second image; fuse the first image to be fused and the second image to be fused to obtain an initial fused image. Among them, the way to determine the first image to be fused can be to enlarge the first image to the scaling multiple indicated by the scaling information, and then intercept the area within the image frame where the original canvas is located to obtain the first image to be fused. The way to determine the second image to be fused can be to enlarge the second image to the scaling multiple indicated by the scaling information, and then intercept the area within the image frame where the original canvas is located to obtain the second image to be fused. Among them, the first image and the second image can be enlarged according to a preset enlargement rule, that is, enlarged along the preset enlargement direction from the preset enlargement reference point.

[0082] Taking the preset enlargement rule of using the center point of the image as the enlargement reference point for image scaling as an example for illustration. When the scaling information is 2, the electronic device can enlarge the first image 2 times from the center point to the surrounding, and perform cropping based on the image frame where the original canvas is located, and retain the first image area inside the image frame to obtain the first image to be fused. At the same time, the electronic device also enlarges the second image 2 times from the center point to the surrounding, and performs cropping based on the image frame where the original canvas is located, and retains the second image area inside the image frame to obtain the second image to be fused.

[0083] In some embodiments, after determining the first image to be fused and the second image to be fused, the electronic device can fuse the first image to be fused and the second image to be fused to obtain an initial fused image. Among them, the process of image fusion is used to combine the information of the first image to be fused and the second image to be fused into one image to enhance the visual effect or extract more valuable information. The image fusion process in the embodiments of the present application can adopt any fusion method such as a pixel-based fusion method, a multi-resolution pyramid fusion method, and a region-of-interest-based fusion method.

[0084] During the image fusion process, the electronic device first aligns the first image to be fused and the second image to be fused considering the global offset information between the first image and the second image, and then performs image fusion based on the aligned first image to be fused and second image to be fused to obtain an initial fused image. This fusion process takes the first image to be fused as the reference, that is, the first image to be fused does not move during the alignment process, and the second image to be fused moves relative to the canvas. After the second image to be fused is aligned with the first image to be fused, some areas will be moved out of the canvas, and corresponding blank areas will appear. The electronic device will perform mirror processing on the blank areas that appear within the canvas to fill the blank areas.

[0085] Optionally, before performing image fusion processing, the electronic device may also perform image processing such as image enhancement, image noise reduction, and image repair on at least one of the first image to be fused and the second image to be fused. Alternatively, after performing image fusion processing, the electronic device may also perform image processing such as image enhancement, image noise reduction, and image repair on the obtained initial fused image. The image processing before and after image fusion can further improve the quality of the obtained initial fused image.

[0086] In some embodiments, the process of the image fusion processing may be executed after processing and displaying the initial preview image, or executed simultaneously with the process of processing and displaying the initial preview image, which is not limited herein. This execution order is used to quickly display the target preview image before the image fusion is completed, ensuring the timeliness of the display after the image is captured.

[0087] Step S24: Crop the initial fused image according to the first cropping region to obtain the target fused image.

[0088] In a possible implementation manner, after the electronic device determines the first cropping region and performs fusion processing on the first image and the second image to obtain the initial fused image, it may crop the initial fused image based on the first cropping region to obtain the target fused image. Among them, the target fused image is the image that the display device of the electronic device replaces and displays to the user after image acquisition and image processing. That is, after determining the target fused image, the electronic device replaces the currently displayed target preview image in the display device with the target fused image.

[0089] Optionally, the target fused image determined in the embodiments of the present application is a high dynamic range image (High Dynamic Range Imaging, HDR). This type of image is a technology that captures a wider range of brightness (from the darkest shadow to the brightest highlight) by synthesizing multiple photos with different exposures. Compared with ordinary low dynamic range images, HDR can retain more details, especially in scenes with strong contrast between light and dark (such as backlight, mixed indoor and outdoor lighting), the image details are richer and the visual effect is better. Therefore, the replaced target fused image has better quality and visual effect than the target preview image before replacement.

[0090] In some embodiments, the electronic device needs to align the first image to be fused and the second image to be fused for image fusion, and then perform image fusion based on the aligned first image to be fused and the second image to be fused to obtain an initial fused image. Due to certain differences between the two images, there are some non-overlapping regions in the two images after alignment. Usually, mirror processing is performed on this region and then image fusion is carried out. In this way, there will be some mirror regions in the obtained initial fused image. The electronic device can crop the initial fused image with the first cropping region to obtain a target fused image with the mirror region removed.

[0091] In the above cropping process, the first cropping region is used to determine other regions in the initial fused image except for the mirror region. However, the size of the image within the first cropping region does not conform to the canvas ratio selected by the user. This canvas ratio is the canvas ratio selected by the user during image acquisition, such as 4:3, 3:4, 16:9, 3:2, etc. After obtaining the initial fused image, the electronic device can determine a third cropping region according to the canvas ratio of the initial fused image and the first cropping region. Crop the initial fused image based on the third cropping region to obtain a target fused image. Among them, the size of the image within the third cropping region conforms to the canvas ratio of the initial fused image. After cropping based on the third cropping region, the image region of the initial fused image within the third cropping region is retained, and the image region of the initial fused image outside the third cropping region is removed. Thus, the embodiments of the present application can ensure that the canvas ratio of the target fused image obtained after image cropping is the same as that of the initial fused image.

[0092] Figure 4 Shows a schematic diagram of a second cropping region or a third cropping region according to an embodiment of the present application. As Figure 4 shown, when the size ratio within the first cropping region is different from the canvas ratio of the initial fused image, the electronic device needs to adjust the first cropping region according to the size of the first cropping region and the canvas ratio of the initial fused image to obtain a third cropping region with a size ratio the same as the canvas ratio of the initial fused image.

[0093] Exemplarily, when the size ratio of the first cropping region is narrower than the canvas ratio, the first cropping region can be shortened longitudinally to obtain a third cropping region with a shortened size ratio the same as the canvas ratio. Optionally, when the size ratio of the first cropping region is wider than the canvas ratio, the first cropping region can be shortened transversely to obtain a third cropping region with a shortened size ratio the same as the canvas ratio.

[0094] After completing the image cropping to obtain the target fusion image, the electronic device can directly replace and display the target fusion image to the user through the display device to ensure the quality of the image obtained by the user. Since the target preview image and the target fusion image displayed before and after by the display device are images obtained by cropping the mirror area in the same area, they have the same field of view angle and there is no display jump or mirror content.

[0095] Figure 5 The flowchart showing another image processing method according to an embodiment of the present application is as follows. Figure 5 As shown, after the electronic device determines the first image and the second image, the first image can be used as the reference image and the second image can be used as the reference image for correcting the first image for image processing.

[0096] Exemplarily, the specific image processing process can be to first perform step S50 to extract corner points from the first image to obtain a plurality of first corner points. And perform step S51 to calculate the dense optical flow between the first image and the second image to obtain the dense optical flow between the first image and the second image. Further, based on the first corner points and the dense optical flow, perform step S52 for corner point matching to determine the corresponding second corner points of each first corner point in the second image. Based on the matching relationship between the plurality of first corner points and the second corner points, perform step S53 to determine the affine matrix between the first image and the second image, and then perform step S54 according to the affine matrix to determine the global motion offset information between the first image and the second image.

[0097] Optionally, the electronic device can determine two new cropping regions, namely the second cropping region and the third cropping region, based on the global motion offset information and the scaling information for indicating the cropping region of the original image. The cropping positions corresponding to the second cropping region and the third cropping region are the same, and they are only used to crop different images.

[0098] After determining the second cropping region, the electronic device can perform image cropping by executing step S56, crop the initial preview image determined from the first image based on the second cropping region to obtain the target preview image, and display the target preview image first.

[0099] Optionally, the electronic device can perform image fusion processing on the first image and the second image by executing step S55 based on the scaling information and the global motion offset information to obtain the initial fusion image. In the embodiment of the present application, the process of this image fusion processing can be implemented by the image algorithm processing module of the electronic device.

[0100] After determining the third cropping region and the initial fused image, the electronic device can perform image cropping by executing step S57, crop the target fused image from the initial fused image based on the third cropping region, and replace the currently displayed target preview image with the target fused image for display. The process of this image cropping process can be implemented by the image post-processing module of the electronic device.

[0101] Figure 6 A schematic diagram showing an image processing process according to an embodiment of the present application. As Figure 6 shown, in the image processing process of the embodiment of the present application, after the electronic device obtains the global motion offset information after image alignment of the first image and the second image, the electronic device's module sends the scaling information to the cropping region calculation module, and the cropping region calculation module determines the first cropping region based on the global motion offset information and the scaling information together. Then, the scheduling module crops the target preview image according to the first cropping region. At the same time, the scheduling module also sends the offset information to the image algorithm processing module to perform image fusion processing to obtain the initial fused image. The scheduling module sends the first cropping region to the image post-processing module to crop the initial fused image to obtain the target fused image.

[0102] Based on the above technical features, the embodiment of the present application can pre-calculate the region where mirroring will occur during the image fusion process after selecting the first image as the display basis, obtain the cropping region, and first crop and then display the preview image based on this cropping region to ensure the timeliness of display after image acquisition. At the same time, the image currently displayed is optimized through algorithm processing, and the optimized image is also cropped and replaced for display through the same cropping region, avoiding the problems of mirroring or jumping of the preview image and the target image displayed before and after, and ensuring the consistency of the field of view of the images before and after.

[0103] It should be understood that although each step in the above flowcharts is displayed sequentially according to the arrow indication, these steps are not necessarily executed sequentially according to the arrow indication. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above flowcharts may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

[0104] Based on the foregoing embodiments, an embodiment of the present application provides an image processing apparatus. The apparatus includes each module included therein, and each unit included in each module, and can be implemented by a processor; of course, it can also be implemented by specific logic circuits. During implementation, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0105] Figure 7 FIG. shows a schematic diagram of an image processing apparatus 60 according to an embodiment of the present application. As Figure 7 shown, the image processing apparatus 60 according to an embodiment of the present application may include:

[0106] An image acquisition module 61, configured to acquire a first image and a second image simultaneously collected by an imaging module, where the first image and the second image respectively correspond to different exposure amounts; the first image is used to generate an initial preview image;

[0107] A cropping area determination module 62, configured to align the first image and the second image to determine a first cropping area, where the first cropping area is an overlapping image area between the first image and the second image;

[0108] A first cropping module 63, configured to crop the initial preview image according to the first cropping area to obtain a target preview image;

[0109] An image fusion module 64, configured to fuse the first image and the second image to obtain an initial fusion image;

[0110] A second cropping module 65, configured to crop the initial fusion image according to the first cropping area to obtain a target fusion image.

[0111] In a possible implementation manner, the cropping area determination module 62 is further configured to:

[0112] Calculate the global motion offset information between the second image and the first image;

[0113] Align the first image and the second image according to the global motion offset information, and determine the first cropping area according to the alignment result.

[0114] In a possible implementation manner, the cropping area determination module 62 is further configured to:

[0115] Determine a first alignment area corresponding to the first image and a second alignment area corresponding to the second image according to the scaling information;

[0116] Align the first alignment region and the second alignment region according to the global motion offset information, determine the overlapping image region between the first alignment region and the second alignment region, and use the overlapping image region as the first cropping region.

[0117] In a possible implementation, the cropping region determination module 62 is further configured to:

[0118] Determine the affine matrix between the first image and the second image;

[0119] Determine the global motion offset information between the second image and the first image according to the affine matrix.

[0120] In a possible implementation, the cropping region determination module 62 is further configured to:

[0121] Determine at least one first corner point in the first image;

[0122] Calculate the dense optical flow from the second image to the first image, where the dense optical flow includes the motion offset corresponding to each pixel point in the second image;

[0123] Determine the corresponding second corner point in the second image for each first corner point according to the dense optical flow;

[0124] Determine the affine matrix between the first image and the second image according to at least one first corner point and the corresponding second corner point for each first corner point.

[0125] In a possible implementation, the first cropping module 63 is further configured to:

[0126] Determine the second cropping region according to the canvas ratio of the initial preview image and the first cropping region;

[0127] Crop the initial preview image based on the second cropping region to obtain the target preview image.

[0128] In a possible implementation, the image fusion module 64 is further configured to:

[0129] Determine the first image to be fused according to the scaling information and the first image;

[0130] Determine the second image to be fused according to the scaling information and the second image;

[0131] Fuse the first image to be fused and the second image to be fused to obtain the initial fused image.

[0132] In a possible implementation, the second cropping module 65 is further configured to:

[0133] Determine the third cropping region according to the canvas ratio of the initial fused image and the first cropping region;

[0134] Crop the initial fused image based on the third cropping region to obtain the target fused image.

[0135] In a possible implementation, the exposure value corresponding to the first image is 0, and the exposure value corresponding to the second image is less than 0.

[0136] The description of the above device embodiments is similar to that of the above method embodiments and has similar beneficial effects to the method embodiments. For the technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0137] It should be noted that in the embodiments of the present application Figure 7 The division of modules by the image processing device shown is schematic, only a logical function division, and there may be other division methods in actual implementation. In addition, each functional unit in the various embodiments of the present application may be integrated in a processing unit, may exist separately physically, or two or more units may be integrated in one unit. The above integrated unit may be implemented in the form of hardware, or in the form of a software functional unit, or in the form of a combination of software and hardware.

[0138] It should be noted that in the embodiments of the present application, if the above method is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the related technology, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device to execute all or part of the methods described in the various embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), magnetic disks, or optical discs that can store program codes. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0139] Figure 8 Shows a schematic diagram of an electronic device according to an embodiment of the present application. As Figure 8 shown, an embodiment of the present application provides an electronic device, which may be a server, and its internal structure diagram may be as Figure 7As shown in the figure. The electronic device includes a processor 720, a memory, and a transceiver 740 connected via a system bus 710. Among them, the processor 720 of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium 731 and an internal memory 732. The non-volatile storage medium 731 stores an operating system, a computer program, and a database. The internal memory 732 provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium 731. The database of the electronic device is used to store data. The transceiver 740 of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor 720, the above method is implemented.

[0140] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor 720, the steps in the method provided in the above embodiment are implemented.

[0141] An embodiment of the present application provides a computer program product containing instructions. When it runs on a computer, it causes the computer to execute the steps in the method provided in the above method embodiment.

[0142] Those skilled in the art can understand that Figure 8 the structure shown in the figure is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0143] In one embodiment, the job parsing device provided by the present application can be implemented in the form of a computer program, and the computer program can run on an electronic device as shown in Figure 8 the figure. Each program module constituting the above device can be stored in the memory of the electronic device. The computer program composed of each program module causes the processor 720 to execute the steps in the methods of various embodiments of the present application described in this specification.

[0144] It should be noted here that the descriptions of the above storage medium and device embodiments are similar to those of the above method embodiments, and have beneficial effects similar to those of the method embodiments. For the technical details not disclosed in the storage medium, storage medium, and device embodiments of the present application, please refer to the descriptions of the method embodiments of the present application for understanding.

[0145] It should be understood that the "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the appearances of "in one embodiment" or "in an embodiment" or "in some embodiments" throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics may be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the magnitude of the serial numbers of the above processes does not mean the sequence of execution. The execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application. The serial numbers of the embodiments of the present application above are only for description and do not represent the advantages or disadvantages of the embodiments. The above descriptions of the various embodiments tend to emphasize the differences between the various embodiments, and their similarities or similarities can be referred to each other. For the sake of brevity, they will not be elaborated herein.

[0146] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, object A and / or object B can represent: the situation where object A exists alone, the situation where object A and object B exist simultaneously, and the situation where object B exists alone.

[0147] It should be noted that in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, the element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0148] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The above-described embodiments are only illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling or communication connection between the components shown or discussed with each other can be through some interfaces, and the indirect coupling or communication connection of the devices or modules can be electrical, mechanical or other forms.

[0149] The modules described above as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network elements; some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0150] In addition, in each embodiment of this application, all the functional modules may be integrated in one processing unit, or each module may be a separate unit alone, or two or more modules may be integrated in one unit; the above integrated modules may be implemented in the form of hardware or in the form of a combination of hardware and software functional units.

[0151] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the above method embodiments; and the foregoing storage medium includes: removable storage devices, read-only memory (ROM), magnetic disks, or optical disks and other various media that can store program codes.

[0152] Alternatively, if the above integrated unit of this application is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiments of this application essentially or the part that contributes to the related technology can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing an electronic device to execute all or part of the methods described in the various embodiments of this application. And the foregoing storage medium includes: removable storage devices, ROM, magnetic disks, or optical disks and other various media that can store program codes.

[0153] The methods disclosed in the several method embodiments provided in this application can be combined arbitrarily without conflict to obtain new method embodiments.

[0154] The features disclosed in the several product embodiments provided in this application can be combined arbitrarily without conflict to obtain new product embodiments.

[0155] The features disclosed in the several method or device embodiments provided in this application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

[0156] As described above, it is only the implementation mode of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed by this application can easily think of changes or substitutions, which should all be covered within the protection scope of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the said claims.

Claims

1. An image processing method, characterized in that, The method includes: Obtaining a first image and a second image simultaneously collected by an imaging module, where the first image and the second image respectively correspond to different exposure amounts; the first image is used to generate an initial preview image; Aligning the first image and the second image to determine a first cropping region, where the first cropping region is the overlapping image region between the first image and the second image; Cropping the initial preview image according to the first cropping region to obtain a target preview image; Fusing the first image and the second image to obtain an initial fused image; Cropping the initial fused image according to the first cropping region to obtain a target fused image.

2. The method according to claim 1, wherein The aligning the first image and the second image to determine a first cropping region includes: Calculating global motion offset information between the second image and the first image; Aligning the first image and the second image according to the global motion offset information, and determining a first cropping region according to the alignment result.

3. The method according to claim 2, characterized in that The aligning the first image and the second image according to the global motion offset information, and determining a first cropping region according to the alignment result includes: Determining a first alignment region corresponding to the first image and a second alignment region corresponding to the second image according to scaling information; Aligning the first alignment region and the second alignment region according to the global motion offset information, determining the overlapping image region between the first alignment region and the second alignment region, and using the overlapping image region as the first cropping region.

4. The method according to claim 2, characterized in that, The calculating global motion offset information between the second image and the first image includes: Determining an affine matrix between the first image and the second image; Determining global motion offset information between the second image and the first image according to the affine matrix.

5. The method according to claim 4, wherein The determining an affine matrix between the first image and the second image includes: Determining at least one first corner point in the first image; Calculating the dense optical flow from the second image to the first image, where the dense optical flow includes the motion offset amounts corresponding to each pixel point in the second image; Determining a second corner point corresponding to each first corner point in the second image according to the dense optical flow; Determining an affine matrix between the first image and the second image according to the at least one first corner point and the second corner point corresponding to each first corner point.

6. The method according to claim 1, wherein The cropping the initial preview image according to the first cropping region to obtain a target preview image includes: Determining a second cropping region according to the canvas ratio of the initial preview image and the first cropping region; Cropping the initial preview image based on the second cropping region to obtain a target preview image.

7. The method according to claim 1, characterized in that, The fusing the first image and the second image to obtain an initial fused image includes: Determining a first image to be fused according to the scaling information and the first image; Determining a second image to be fused according to the scaling information and the second image; Fusing the first image to be fused and the second image to be fused to obtain an initial fused image.

8. The method according to claim 1, characterized in that, Cropping the initial fused image according to the first cropping area to obtain a target fused image, including: Determining a third cropping area according to the canvas ratio of the initial fused image and the first cropping area; Cropping the initial fused image based on the third cropping area to obtain a target fused image.

9. The method according to any one of claims 1 to 7, characterized in that, The exposure value corresponding to the first image is 0, and the exposure value corresponding to the second image is less than 0.

10. An image processing apparatus, characterized in that, The device includes: An image acquisition module, configured to acquire a first image and a second image simultaneously collected by an imaging module, the first image and the second image corresponding to different exposure amounts respectively; the first image is used to generate an initial preview image; A cropping area determination module, configured to align the first image and the second image to determine a first cropping area, the first cropping area being an overlapping image area between the first image and the second image; A first cropping module, configured to crop the initial preview image according to the first cropping area to obtain a target preview image; An image fusion module, configured to fuse the first image and the second image to obtain an initial fused image; A second cropping module, configured to crop the initial fused image according to the first cropping area to obtain a target fused image.

11. An electronic device, comprising a memory and a processor, the memory storing a computer program that can run on the processor, characterized in that, When the processor executes the program, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images