Image processing method, device, electronic device and storage medium

Abstract

This application provides an image processing method, apparatus, electronic device, and storage medium, relating to the field of exposure imaging. The method includes: acquiring a first standard dynamic range (SDR) image and a second SDR image with different exposure times for the same scene; acquiring a motion vector map, a motion region mask map, and a first high dynamic range (HDR) image based on the first SDR image and the second SDR image; acquiring a second HDR image based on the first SDR image, the second SDR image, and the motion vector map; and acquiring a target HDR image based on the motion region mask map, the first HDR image, and the second HDR image. This application can selectively fuse images with different exposures based on moving pixels, thereby achieving noise reduction in HDR images, reducing the impact of moving objects on HDR images, avoiding resolution degradation in moving areas, and improving the quality of HDR images and the accuracy of image processing.

Description

Technical Field

[0001] The present application relates to the field of exposure imaging, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] Compared with Standard Dynamic Range (SDR) images, High-Dynamic Range (HDR) images can provide more dynamic range and image details. In related technologies, SDR images with different exposure times of the same scene are generally fused to generate HDR images, where only SDR images with short exposure frame (SEF) are used for moving areas, and then spatial noise reduction is performed on the moving areas to improve the signal-to-noise ratio (SNR) of the image.

[0003] However, since the SDR image SNR of SEF is extremely low, using weaker spatial noise reduction cannot effectively improve the SNR, while using stronger spatial noise reduction will cause the resolution of the moving area to be seriously reduced. Therefore, how to improve the quality of HDR images has become one of the important research directions. Summary of the invention

[0004] The purpose of this application is to solve one of the technical problems in the above technology at least to a certain extent.

[0005] The first aspect of the present application provides an image processing method, comprising:

[0006] Acquire a first standard dynamic range SDR image and a second SDR image with different exposure times under the same scene;

[0007] According to the first SDR image and the second SDR image, a motion vector map, a motion area mask map and a first high dynamic range HDR image are obtained;

[0008] Acquire a second HDR image based on the first SDR image, the second SDR image, and the motion vector graph;

[0009] A target HDR image is acquired based on the motion region mask image, the first HDR image, and the second HDR image.

[0010] The first aspect of the present application provides an image processing method, which also has the following technical features, including:

[0011] According to one embodiment of the present application, the process of acquiring a motion vector diagram includes: for any first pixel point on a first SDR image, acquiring a second pixel point of a second SDR image that matches any first pixel point; acquiring respective coordinate values ​​of any first pixel point and the matched second pixel point; and acquiring a motion vector diagram based on respective coordinate values ​​of any first pixel point and the matched second pixel point.

[0012] According to one embodiment of the present application, obtaining a second pixel point of a second SDR image that matches any first pixel point includes: obtaining any first pixel point and a corresponding candidate matching area on the second SDR image; obtaining a first pixel value of the first pixel point and a candidate second pixel value within the candidate matching area; obtaining an absolute value of a first difference between the first pixel value and the candidate second pixel value; and confirming a second pixel point that matches the first pixel point from the candidate matching area based on the absolute value of the first difference.

[0013] According to an embodiment of the present application, a motion vector diagram is obtained based on the respective coordinate values ​​of any first pixel point and the matching second pixel point, including: obtaining the coordinate difference between any first pixel point and the matching second pixel point to determine the horizontal displacement and the vertical displacement; obtaining the first motion vector based on the horizontal displacement, and obtaining the second motion vector based on the vertical displacement; and determining the motion vector diagram based on the first motion vector and the second motion vector.

[0014] According to an embodiment of the present application, a process of acquiring a first HDR image includes: acquiring a first exposure ratio of a first SDR image and a second SDR image; and fusing the first SDR image and the second SDR image according to the first exposure ratio to acquire the first HDR image.

[0015] According to an embodiment of the present application, based on the first SDR image, the second SDR image and the motion vector diagram, a second HDR image is obtained, including: obtaining a second exposure ratio of the first SDR image and the second SDR image; and fusing the first SDR image and the second SDR image according to the second exposure ratio and the motion vector diagram to obtain the second HDR image.

[0016] According to one embodiment of the present application, the process of acquiring the motion area mask map includes: acquiring a third pixel value of a third pixel point in the second SDR image that is at the same position as the first pixel point; acquiring the absolute value of the second difference between the first pixel value and the third pixel value; and acquiring the motion area mask map based on the absolute value of the second difference.

[0017] According to an embodiment of the present application, a target HDR image is acquired based on a motion area mask image, a first HDR image, and a second HDR image, including: normalizing the values ​​of the pixels in the motion area mask image to acquire a fusion ratio; and fusing the first HDR image and the second HDR image according to the fusion ratio to acquire the target HDR image.

[0018] A second aspect of the present application provides an image processing device, comprising:

[0019] A first acquisition module is used to acquire a first standard dynamic range SDR image and a second SDR image with different exposure times under the same scene;

[0020] A second acquisition module is used to acquire a motion vector map, a motion area mask map and a first high dynamic range HDR image according to the first SDR image and the second SDR image;

[0021] A third acquisition module is used to acquire a second HDR image based on the first SDR image, the second SDR image and the motion vector diagram;

[0022] The fourth acquisition module is used to acquire a target HDR image based on the motion region mask image, the first HDR image and the second HDR image.

[0023] The second aspect of the present application provides an image processing device, which also has the following technical features, including:

[0024] According to one embodiment of the present application, the second acquisition module is also used to: for any first pixel point on the first SDR image, obtain a second pixel point of the second SDR image that matches any first pixel point; obtain the respective coordinate values ​​of any first pixel point and the matched second pixel point; and obtain a motion vector diagram based on the respective coordinate values ​​of any first pixel point and the matched second pixel point.

[0025] According to one embodiment of the present application, the second acquisition module is also used to: acquire any first pixel point and the corresponding candidate matching area on the second SDR image; acquire the first pixel value of the first pixel point and the candidate second pixel value within the candidate matching area; acquire the absolute value of the first difference between the first pixel value and the candidate second pixel value; and confirm the second pixel point that matches the first pixel point from the candidate matching area based on the absolute value of the first difference.

[0026] According to one embodiment of the present application, the second acquisition module is also used to: obtain the coordinate difference between any first pixel point and the matching second pixel point to determine the horizontal displacement and the vertical displacement; obtain the first motion vector based on the horizontal displacement, and obtain the second motion vector based on the vertical displacement; determine the motion vector map based on the first motion vector and the second motion vector.

[0027] According to an embodiment of the present application, the second acquisition module is further used to: obtain a first exposure ratio of the first SDR image and the second SDR image; and fuse the first SDR image and the second SDR image according to the first exposure ratio to obtain a first HDR image.

[0028] According to one embodiment of the present application, the third acquisition module is further used to: obtain a second exposure ratio of the first SDR image and the second SDR image; and fuse the first SDR image and the second SDR image according to the second exposure ratio and the motion vector diagram to obtain a second HDR image.

[0029] According to one embodiment of the present application, the second acquisition module is also used to: obtain a third pixel value of a third pixel point in the second SDR image that is at the same position as the first pixel point; obtain the absolute value of the second difference between the first pixel value and the third pixel value; and obtain a motion area mask map based on the absolute value of the second difference.

[0030] According to an embodiment of the present application, the fourth acquisition module is further used to: normalize the values ​​of the pixel points in the motion area mask image to obtain a fusion ratio; and fuse the first HDR image and the second HDR image according to the fusion ratio to obtain a target HDR image.

[0031] The third aspect embodiment of the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image processing method provided in the first aspect of the present application.

[0032] The fourth aspect of the present application provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the image processing method provided in the first aspect of the present application.

[0033] The fifth aspect of the present application provides a computer program product. When the instruction processor in the computer program product executes, the image processing method provided in the first aspect of the present application is executed.

[0034] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application.

[0035] The embodiments of the present application can fuse images with different exposures in a targeted manner according to the pixels that are moving, so as to achieve the purpose of denoising the HDR image, reduce the impact of moving objects on the HDR image, avoid the reduction of resolution in the moving area, and improve the quality of the HDR image and the accuracy of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0037] Figure 1 A flowchart of an image processing method according to an embodiment of the present application;

[0038] Figure 2 A flowchart of an image processing method according to an embodiment of the present application;

[0039] Figure 3 A schematic diagram of an image processing method according to an embodiment of the present application;

[0040] Figure 4 A flowchart of an image processing method according to an embodiment of the present application;

[0041] Figure 5 A schematic diagram of an image processing method according to an embodiment of the present application;

[0042] Figure 6 A schematic diagram of an image processing method according to an embodiment of the present application;

[0043] Figure 7 A flowchart of an image processing method according to an embodiment of the present application;

[0044] Figure 8 A flowchart of an image processing method according to an embodiment of the present application;

[0045] Fig. 9 A schematic diagram of an image processing method according to an embodiment of the present application;

[0046] Fig.10 A flowchart of an image processing method according to an embodiment of the present application;

[0047] Fig.11 A schematic diagram of an image processing method according to an embodiment of the present application;

[0048] Fig.12 A schematic diagram of an image processing method according to an embodiment of the present application;

[0049] Fig.13 A schematic diagram of the structure of an image processing device according to an embodiment of the present application;

[0050] Fig.14 It is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0051] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0052] The image processing method, device, electronic device and storage medium of the embodiments of the present application are described below with reference to the accompanying drawings.

[0053] Figure 1 is a flow chart of an image processing method according to an embodiment of the present application, as shown in Figure 1 As shown, the method includes:

[0054] S11, acquiring a first SDR image and a second SDR image with different exposure times under the same scene.

[0055] An image acquisition device is used to shoot the same scene multiple times with different exposure degrees to obtain a first SDR image and a second SDR image with different exposure times.

[0056] Optionally, the image acquisition device may be a terminal device, a tablet computer, etc. having a camera, and this application does not impose any limitation on this.

[0057] S12, acquiring a motion vector map, a motion region mask map and a first HDR image according to the first SDR image and the second SDR image.

[0058] Optionally, during implementation, the image acquisition device may shake and objects in the scene may move, so the position of the moving object may change between the first SDR image and the second SDR image.

[0059] In some implementations, the first SDR image is fused with the second SDR image to obtain a first HDR image. During the implementation, since the position of the moving object will change, the first HDR image may appear "ghosting" and the image quality is not high. Therefore, in an embodiment of the present application, the area where the moving object is located in the first SDR image and the second SDR image, that is, the moving area, is analyzed, and a motion vector map is obtained based on the displacement of any pixel point to facilitate the subsequent improvement of the SNR of the image.

[0060] In some implementations, the first SDR image and the second SDR image are analyzed in combination to obtain the confidence that any pixel at the corresponding position belongs to the motion area, and then the confidence is used as the value of the pixel to generate a motion area mask map.

[0061] S13, acquiring a second HDR image based on the first SDR image, the second SDR image and the motion vector map.

[0062] The motion vector map can reflect the displacement of the moving area. According to the motion vector of any pixel point in the first SDR image, its matching point in the second SDR image is obtained, so as to fuse the moving pixels in a targeted manner. In other words, the first SDR image and the second SDR image are fused in a targeted manner to achieve the purpose of image noise reduction and obtain the second HDR image.

[0063] S14, acquiring a target HDR image based on the motion region mask image, the first HDR image, and the second HDR image.

[0064] In the embodiment of the present application, the value of the pixel in the motion area mask map reflects the confidence that the pixel at that position belongs to the motion area. The fusion ratio is obtained based on the motion area mask map, and then the first HDR image and the second HDR image are fused to obtain the target HDR image.

[0065] The fusion ratio can show the probability of motion in the image, and then control the weight ratio of the image with moving objects during high dynamic range fusion according to the probability of motion, thereby reducing the impact of moving objects on the HDR image.

[0066] In the embodiment of the present application, a motion vector map, a motion region mask map and a first HDR image are obtained based on the first SDR image and the second SDR image, a second HDR image is obtained based on the first SDR image, the second SDR image and the motion vector map, and a target HDR image is obtained based on the motion region mask map, the first HDR image and the second HDR image. The embodiment of the present application can fuse images of different exposures in a targeted manner according to pixels where motion occurs, thereby achieving the purpose of denoising the HDR image, reducing the impact of moving objects on the HDR image, avoiding the reduction of resolution in the moving area, and improving the quality of the HDR image and the accuracy of image processing.

[0067] Figure 2 is a flow chart of an image processing method according to an embodiment of the present application, as shown in Figure 2 As shown, based on the above embodiment, the process of acquiring the first HDR image includes:

[0068] S21, obtaining a first exposure ratio between the first SDR image and the second SDR image.

[0069] For illustration, the exposure time of the first SDR image is shorter than that of the second SDR image. In the embodiment of the present application, a first exposure ratio of the first SDR image to the second SDR image is obtained as Ratio, that is, the brightness of the second SDR image is Ratio times that of the first SDR image.

[0070] S22, fusing the first SDR image and the second SDR image according to the first exposure ratio to obtain a first HDR image.

[0071] like Figure 3 As shown, the coordinate value of the pixel point of the first HDR image can be obtained by using the following formula:

[0072] PX h =(Px s +Ratio×PX l ) / (1+Ratio)

[0073] Among them, Px s is the pixel value of the pixel with coordinates (X0, Y0) in the first SDR image, PX l is the pixel value of the pixel with coordinates (X0, Y0) in the second SDR image, PX h is the pixel value of the pixel point with coordinates (X0, Y0) in the fused first HDR image. Similarly, the pixel value of each pixel point in the first HDR image can be obtained to obtain the first HDR image.

[0074] The embodiments of the present application can fuse images with different exposures in a targeted manner according to the pixels that are moving, so as to achieve the purpose of denoising the HDR image, reduce the impact of moving objects on the HDR image, avoid the reduction of resolution in the moving area, and improve the quality of the HDR image and the accuracy of image processing.

[0075] Figure 4 is a flow chart of an image processing method according to an embodiment of the present application, as shown in Figure 4 As shown, the exposure time of the first SDR image is shorter than the exposure time of the second SDR image. The process of obtaining the motion vector diagram includes:

[0076] S41, for any first pixel point on the first SDR image, obtain a second pixel point of the second SDR image that matches any first pixel point.

[0077] Get a first sliding window centered on any first pixel point and a corresponding candidate matching area on the second SDR image, such as Figure 5 As shown, in the embodiment of the present application, 3*3 pixels centered at any first pixel are used as the first sliding window 510, and 33*33 pixels centered at the same position pixel on the second SDR image are used as the candidate matching area 520 as an example for introduction. Figure 6 As shown, the first pixel value PATCH at position [i] [j] in the first sliding window 510 is obtained. s[i][j], and the second pixel value PATCH at position [i][j] in any candidate second sliding window 530 in the candidate matching area 520 t [i][j], the size of the first sliding window 510 is the same as that of the candidate second sliding window 530; obtaining the absolute value of the first difference between the first pixel value and the second pixel value. In the embodiment of the present application, the absolute value of the first difference can be obtained by using the following formula:

[0078] SAD t =SUM(ABS(PATCH t [i][j]-PATCH s [i][j]))

[0079] Among them, SAD t represents the absolute value of the first difference, SUM(...) represents the summation operation, ABS(...) represents the absolute value operation, i represents the horizontal coordinate value of the pixel point in the sliding window, j represents the vertical coordinate value of the pixel point in the sliding window. In the embodiment of the present application, since the sliding window is 3*3 pixels, i can take the values ​​of 0, 1, and 2 respectively; in addition, j can also take the values ​​of 0, 1, and 2 respectively.

[0080] With a step size of 1 pixel, the candidate second sliding window 530 is used to traverse the entire candidate matching area 520, the absolute value of each first difference is obtained, and the second sliding window that matches the first sliding window 510 is confirmed from the candidate second sliding window 530 based on the absolute value of the first difference. Optionally, in an embodiment of the present application, the candidate second sliding window 530 corresponding to the smallest first difference absolute value is used as the second sliding window that matches the first sliding window 510, and the central pixel point of the second sliding window is confirmed to be the second pixel point of the second SDR image matched by any first pixel point.

[0081] S42, obtaining respective coordinate values ​​of any first pixel point and a matching second pixel point.

[0082] Obtain the coordinate values ​​of any first pixel point and the matching second pixel point in their respective image coordinate systems. In other words, obtain the coordinate value of any first pixel point in the first SDR image, and obtain the coordinate value of any first pixel point and the matching second pixel point in the second SDR image.

[0083] S43, obtaining a motion vector map according to the coordinate values ​​of any first pixel point and the matched second pixel point.

[0084] Obtain the coordinate difference between any first pixel point and the matching second pixel point to determine the lateral displacement and the longitudinal displacement. Obtain a first motion vector based on the lateral displacement, and obtain a second motion vector based on the longitudinal displacement. Determine a motion vector diagram based on the first motion vector and the second motion vector. In the embodiment of the present application, the coordinate value of the first pixel point is (X0, Y0), and the coordinate value of the second pixel point matching the first pixel point is (X1, Y1) as an example for illustration. It can be determined that the lateral displacement dx=X1-X0 and the longitudinal displacement dy=Y1-Y0. Optionally, in the embodiment of the present application, dx is used as the first motion vector and dy is used as the second motion vector to determine the motion vector diagram.

[0085] Optionally, the first motion vector may be substituted for the value of the second pixel of the second SDR image to obtain a first motion vector map, and the second motion vector may be substituted for the value of the second pixel of the second SDR image to obtain a second motion vector map. In other words, the first motion vector is used as the value of the pixel of (X1, Y1), and so on, until the first motion vector map is obtained. Similarly, the second motion vector is used as the value of the pixel of (X1, Y1), and so on, until the second motion vector map is obtained.

[0086] The embodiments of the present application can fuse images with different exposures in a targeted manner according to the pixels that are moving, so as to achieve the purpose of denoising the HDR image, reduce the impact of moving objects on the HDR image, avoid the reduction of resolution in the moving area, and improve the quality of the HDR image and the accuracy of image processing.

[0087] Figure 7 is a flow chart of an image processing method according to an embodiment of the present application, as shown in Figure 7 As shown, based on the above embodiment, the process of obtaining the motion region mask map includes:

[0088] S71, obtaining a third pixel value of a third pixel point at the same position as the first pixel point in the second SDR image.

[0089] Obtain a third pixel value of a third sliding window in the second SDR image at the same position as the first sliding window. In the embodiment of the present application, the first sliding window is a sliding window centered on the first pixel, the third sliding window is a sliding window centered on the third pixel, and the first sliding window and the third sliding window are the same size; the coordinate value of the first pixel in the first SDR image is (X0, Y0), and the coordinate value of the third pixel in the second SDR image is (X0, Y0).

[0090] S72, obtaining an absolute value of a second difference between the first pixel value and the third pixel value.

[0091] The absolute value SAD of the second difference between the first pixel value and the third pixel value is obtained by the following formula:m :

[0092] SAD m =SUM(ABS(PATCH l -[i][j]-PATCH s [i][j]))

[0093] PATCH l [i][j] is the third pixel value at position [i][j] in the third sliding window.

[0094] S73, obtaining a motion region mask map according to the absolute value of the second difference.

[0095] SAD m As the pixel value of the pixel point at the coordinate value (X0, Y0) in the motion region mask map, similarly, the pixel value of each pixel point in the motion region mask map can be obtained, and then the motion region mask map can be obtained.

[0096] The embodiments of the present application can fuse images with different exposures in a targeted manner according to the pixels that are moving, so as to achieve the purpose of denoising the HDR image, reduce the impact of moving objects on the HDR image, avoid the reduction of resolution in the moving area, and improve the quality of the HDR image and the accuracy of image processing.

[0097] Figure 8 is a flow chart of an image processing method according to an embodiment of the present application, as shown in Figure 8 As shown, on the basis of the above embodiment, based on the first SDR image, the second SDR image and the motion vector diagram, obtaining the second HDR image includes:

[0098] S81, obtaining a second exposure ratio between the first SDR image and the second SDR image.

[0099] In some implementations, the second exposure ratio of the first SDR image and the second SDR image is directly obtained. In some implementations, in order to improve the accuracy of image processing, multiple second exposure ratios can be obtained based on the first SDR image and the second SDR image to facilitate the subsequent fusion of the first SDR image and the second SDR image.

[0100] Assume that the exposure time of the first SDR image is shorter than that of the second SDR image. For any first pixel point of the first SDR image, a second pixel point of the second SDR image matching the first pixel point can be located according to the motion vector diagram. For example, for a pixel point with coordinates (X0, Y0) in the first SDR image, the coordinate value of the matching pixel point in the second SDR image is (X0+dx, Y0+dy). Fig. 9As shown, in the embodiment of the present application, a pixel point area MatchArea centered at (X0+dx, Y0+dy) in the second SDR image is obtained. l , optionally, the pixel area MatchArea l The range can be 11*11 pixels, and a fourth sliding window with the same size as the first sliding window is established for the pixel area MatchArea l Traversing, in some implementations, the following formula is used to generate a second exposure ratio for each fourth sliding window:

[0101] SAD nr =SUM(ABS(PATCH nr [i][j]-PATCH s [i][j]))

[0102] Among them, PATCH nr [i][j] is the fourth pixel value at position [i][j] in the fourth sliding window, SAD nr is the second exposure ratio corresponding to the central pixel of the fourth sliding window. Similarly, the pixel area MatchArea can be obtained. l A second exposure ratio for each pixel.

[0103] S82, merging the first SDR image and the second SDR image according to the second exposure ratio and the motion vector diagram to obtain a second HDR image.

[0104] The description will be continued by taking the example of obtaining a plurality of second exposure ratios according to the first SDR image and the second SDR image.

[0105] MatchArea based on pixel area l The second exposure ratio SAD of the pixel at position [m][n] nr [m][n], get the reciprocal WT of the second exposure ratio nr [m][n]=1 / SAD nr [m][n], where m represents the center pixel of the fourth sliding window in the pixel area MatchArea l The horizontal coordinate value in n represents the center pixel of the fourth sliding window in the pixel area MatchArea l In the embodiment of the present application, due to the vertical coordinate value of the pixel point area MatchArea l It is 11*11 pixels, so m can take values ​​of 0, 1, ..., 10 respectively; in addition, n can also take values ​​of 0, 1, ..., 10 respectively.

[0106] The pixel value PX of the pixel point with coordinates (X0, Y0) in the second HDR image is obtained using the following formula: nr :

[0107] PX nr =SUM(WT nr [m][n]×MatchArea l [m][n]) / SUM(WT nr [m][n])

[0108] Among them, MatchArea l [m][n] is the pixel area MatchArea l Similarly, the pixel value of each pixel in the second HDR image can be obtained, thereby obtaining the second HDR image.

[0109] The embodiments of the present application can fuse images with different exposures in a targeted manner according to the pixels that are moving, so as to achieve the purpose of denoising the HDR image, reduce the impact of moving objects on the HDR image, avoid the reduction of resolution in the moving area, and improve the quality of the HDR image and the accuracy of image processing.

[0110] Fig.10 is a flow chart of an image processing method according to an embodiment of the present application, as shown in Fig.10 As shown, based on the above embodiment, the process of obtaining the motion vector map includes:

[0111] S101, acquiring a first standard dynamic range SDR image and a second SDR image with different exposure times under the same scene.

[0112] S102, acquiring a motion vector map, a motion region mask map and a first high dynamic range HDR image according to the first SDR image and the second SDR image.

[0113] S103, acquiring a second HDR image based on the first SDR image, the second SDR image and the motion vector map.

[0114] For the introduction of steps S101 to S103, please refer to the relevant introduction in the above embodiment, which will not be introduced here again.

[0115] S104, normalizing the values ​​of the pixels in the motion region mask image to obtain a fusion ratio of any pixel.

[0116] In an embodiment of the present application, after normalizing the values ​​of the pixels in the motion area mask image, the value of the fusion ratio of any pixel is obtained to be in the interval [0,1]. The fusion ratio can show the probability of motion occurring in the image, and then control the weight ratio of the image with moving objects during high dynamic range fusion according to the probability of motion occurring, thereby reducing the impact of moving objects on the high dynamic range image.

[0117] S105: Fusing the first HDR image and the second HDR image according to a fusion ratio to obtain a target HDR image.

[0118] The following formula is used to obtain the pixel value PX of the pixel point with coordinates (X0, Y0) in the target HDR image: o :

[0119] PX o =PX h ×(1.0-f)+PX nr ×f

[0120] Among them, f is the fusion ratio obtained after the value of the pixel with the pixel coordinate (X0, Y0) in the motion area mask image is normalized. Similarly, the pixel value of each pixel in the target HDR image can be obtained, and then the target HDR image can be obtained.

[0121] The embodiments of the present application can fuse images with different exposures in a targeted manner according to the pixels that are moving, so as to achieve the purpose of denoising the HDR image, reduce the impact of moving objects on the HDR image, avoid the reduction of resolution in the moving area, and improve the quality of the HDR image and the accuracy of image processing.

[0122] Fig.11 Flow chart of an image processing system according to an embodiment of the present application. Fig.11 As shown, the image processing system 110 includes an HDR fusion module M0, a motion compensation noise reduction module M1, a motion vector detection module M2, a motion area detection module M3, and a fusion module M4, wherein:

[0123] The HDR fusion module M0 is used to fuse the first SDR image and the second SDR image to obtain a first HDR image, and send it to the fusion module M4.

[0124] The motion vector detection module M2 is used to detect moving objects in the first SDR image and the second SDR image, obtain the motion vectors of the moving pixels, and generate a motion vector map to send to the motion compensation denoising module M1. The motion compensation denoising module M1 fuses the first SDR image with the second SDR image according to the motion vector map, obtains the second HDR image and sends it to the fusion module M4.

[0125] The motion region detection module M3 is used to detect the motion regions of the first SDR image and the second SDR image, and obtain the confidence that each pixel in the first SDR image belongs to the motion region.

[0126] The fusion module M4 is used to fuse the first HDR image and the second HDR image using the confidence as the fusion ratio to generate a target HDR image.

[0127] The embodiments of the present application can fuse images with different exposures in a targeted manner according to the pixels that are moving, so as to achieve the purpose of denoising the HDR image, reduce the impact of moving objects on the HDR image, avoid the reduction of resolution in the moving area, and improve the quality of the HDR image and the accuracy of image processing.

[0128] Fig.12 Flow chart of an image processing system according to an embodiment of the present application. Fig.12 As shown, the exposure time of the first SDR image is shorter than the exposure time of the second SDR image, and the current coordinate is (X0, Y0). In the embodiment of the present application, the current coordinate is (X0, Y0), and then the pixel value Px of the first pixel point with the coordinate (X0, Y0) in the first SDR image is converted to s The pixel value PX of the third pixel point with coordinates (X0, Y0) in the second SDR image l Perform fusion to obtain the pixel value PX of the pixel point with coordinates (X0, Y0) in the first HDR image h ; Get the first sliding window PATCH centered at (X0, Y0) in the first SDR image s And the candidate matching area centered at (X0, Y0) in the second SDR image, the range of the sliding window can be 3*3 pixels, the range of the candidate matching area can be 33*33 pixels, according to the first sliding window PATCH s The absolute value of the first difference between the first pixel value in the candidate matching area and each second pixel value in the candidate matching area is used to confirm the coordinates (X1, Y1) of the second pixel point that matches the first pixel point in the candidate matching area, and then obtain the first motion vector dx=X1-X0 and the second motion vector dy=Y1-Y0, and in the second SDR image, obtain the pixel point area MatchArea centered on (X0+dx, Y0+dy). l , pixel area MatchArea l The range can be 11*11 pixels, using the second exposure ratio and MatchArea l The pixel value Px of the first pixel sPerform noise reduction and obtain the pixel value PX of the pixel point with coordinates (X0, Y0) in the second HDR image nr ; Get the third sliding window PATCH centered at (X0, Y0) in the second SDR image l , according to the first sliding window PATCH s The first pixel value in the third sliding window PATCH l The third pixel value in the HDR image is obtained, the absolute value of the second difference is obtained, and the absolute value of the second difference is used as the fusion ratio. The pixel value PX of the pixel point with coordinates (X0, Y0) in the first HDR image is h The pixel value PX of the pixel point with coordinates (X0, Y0) in the second HDR image nr Perform fusion to obtain the pixel value PX of the pixel point with coordinates (X0, Y0) in the target HDR image o .

[0129] The embodiments of the present application can fuse images with different exposures in a targeted manner according to the pixels that are moving, so as to achieve the purpose of denoising the HDR image, reduce the impact of moving objects on the HDR image, avoid the reduction of resolution in the moving area, and improve the quality of the HDR image and the accuracy of image processing.

[0130] It should be noted that, in other embodiments, the sliding window, the pixel area MatchArea l The candidate matching area can also be other ranges, where the sliding window range is smaller than the pixel area MatchArea l Range, pixel area MatchArea l The range of is smaller than the range of the candidate matching area, which is not limited in this embodiment of the present application.

[0131] Fig.13 FIG. 1 is a schematic diagram of the structure of an image processing device according to an embodiment of the present application. Fig.13 As shown, the image processing device 1300 includes:

[0132] A first acquisition module 1310 is used to acquire a first standard dynamic range SDR image and a second SDR image with different exposure times under the same scene;

[0133] A second acquisition module 1320 is used to acquire a motion vector map, a motion region mask map and a first high dynamic range HDR image according to the first SDR image and the second SDR image;

[0134] A third acquisition module 1330 is used to acquire a second HDR image based on the first SDR image, the second SDR image and the motion vector diagram;

[0135] The fourth acquisition module 1340 is configured to acquire a target HDR image based on the motion region mask image, the first HDR image, and the second HDR image.

[0136] The image processing device 1300 provided in the present application also has the following technical features, including:

[0137] According to one embodiment of the present application, the second acquisition module 1320 is also used to: for any first pixel point on the first SDR image, obtain a second pixel point of the second SDR image that matches any first pixel point; obtain the respective coordinate values ​​of any first pixel point and the matched second pixel point; and obtain a motion vector diagram based on the respective coordinate values ​​of any first pixel point and the matched second pixel point.

[0138] According to one embodiment of the present application, the second acquisition module 1320 is also used to: acquire any first pixel point and the corresponding candidate matching area on the second SDR image; acquire the first pixel value of the first pixel point and the candidate second pixel value within the candidate matching area; acquire the absolute value of the first difference between the first pixel value and the candidate second pixel value; and confirm the second pixel point that matches the first pixel point from the candidate matching area based on the absolute value of the first difference.

[0139] According to one embodiment of the present application, the second acquisition module 1320 is also used to: obtain the coordinate difference between any first pixel point and the matching second pixel point to determine the horizontal displacement and the vertical displacement; obtain the first motion vector based on the horizontal displacement, and obtain the second motion vector based on the vertical displacement; determine the motion vector map based on the first motion vector and the second motion vector.

[0140] According to an embodiment of the present application, the second acquisition module 1320 is further used to: obtain a first exposure ratio of the first SDR image and the second SDR image; and fuse the first SDR image and the second SDR image according to the first exposure ratio to obtain a first HDR image.

[0141] According to an embodiment of the present application, the third acquisition module 1330 is further used to: obtain a second exposure ratio of the first SDR image and the second SDR image; and fuse the first SDR image and the second SDR image according to the second exposure ratio and the motion vector diagram to obtain a second HDR image.

[0142] According to one embodiment of the present application, the second acquisition module 1320 is also used to: obtain a third pixel value of a third pixel point in the second SDR image that is at the same position as the first pixel point; obtain the absolute value of the second difference between the first pixel value and the third pixel value; and obtain a motion area mask map based on the absolute value of the second difference.

[0143] According to an embodiment of the present application, the fourth acquisition module 1340 is further used to: normalize the values ​​of the pixels in the motion area mask image to obtain a fusion ratio; and fuse the first HDR image and the second HDR image according to the fusion ratio to obtain a target HDR image.

[0144] The embodiments of the present application can fuse images with different exposures in a targeted manner according to the pixels that are moving, so as to achieve the purpose of denoising the HDR image, reduce the impact of moving objects on the HDR image, avoid the reduction of resolution in the moving area, and improve the quality of the HDR image and the accuracy of image processing.

[0145] To achieve the above embodiments, the present application also provides an electronic device, a computer-readable storage medium and a computer program product.

[0146] Fig.14 A block diagram of an electronic device according to an embodiment of the present application. Fig.14 The electronic device shown can be implemented to perform Figures 1 to 10 An image processing method of an embodiment.

[0147] like Fig.14 As shown, the electronic device 1400 includes: a memory 1410 and a processor 1420, a bus 1430 connecting different components (including the memory 1410 and the processor 1420), the memory 1410 stores a computer program, and when the processor 1420 executes the program, the method for identifying the device identification jump described in the embodiment of the present disclosure is implemented.

[0148] Bus 1430 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0149] The electronic device 1400 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by the electronic device 1400, including volatile and non-volatile media, removable and non-removable media.

[0150] The memory 1410 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 1440 and / or cache memory 1450. The electronic device 1400 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 1460 may be used to read and write non-removable, non-volatile magnetic media ( Fig.14 not shown, usually called a "hard drive"). Although Fig.14 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 1430 via one or more data medium interfaces. Memory 1410 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present disclosure.

[0151] A program / utility 1480 having a set (at least one) of program modules 1470 may be stored, for example, in the memory 1410, such program modules 1470 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 1470 generally perform the functions and / or methods of the embodiments described in the present disclosure.

[0152] The electronic device 1400 may also communicate with one or more external devices 1490 (e.g., keyboard, pointing device, display 1491, etc.), one or more devices that enable a user to interact with the electronic device 1400, and / or any device that enables the electronic device 1400 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through an input / output (I / O) interface 1492. Furthermore, the electronic device 1400 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 1493. Fig.14 As shown, the network adapter 1493 communicates with other modules of the electronic device 1400 via the bus 1430. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 1400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0153] The processor 1420 executes various functional applications and data processing by running the programs stored in the memory 1410 .

[0154] In order to implement the above embodiments, the present application also provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute Figures 1 to 10 An image processing method of an embodiment.

[0155] In order to implement the above embodiment, the present application also provides a computer program product. When the instruction processor in the computer program product executes, Figures 1 to 10 An image processing method of an embodiment.

[0156] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0157] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0158] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0159] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0160] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0161] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0162] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0163] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. An image processing method, characterized in that: include: Acquire a first standard dynamic range SDR image and a second SDR image with different exposure times under the same scene; Acquire a motion vector map, a motion region mask map, and a first high dynamic range HDR image according to the first SDR image and the second SDR image; Acquire a second HDR image based on the first SDR image, the second SDR image, and the motion vector graph; Acquire a target HDR image based on the motion region mask image, the first HDR image, and the second HDR image; The process of obtaining the motion vector diagram includes: For any first pixel point on the first SDR image, obtain a second pixel point of the second SDR image that matches the any first pixel point; Obtaining respective coordinate values ​​of any first pixel point and the matched second pixel point; Acquire the motion vector map according to the coordinate values ​​of each of the first pixel point and the matched second pixel point; The acquiring a second HDR image based on the first SDR image, the second SDR image and the motion vector diagram includes: Acquire a second exposure ratio of the first SDR image to the second SDR image; fusing the first SDR image with the second SDR image according to the second exposure ratio and the motion vector graph to obtain a second HDR image; The acquiring a target HDR image based on the motion region mask image, the first HDR image, and the second HDR image includes: Normalizing the values ​​of the pixels in the motion region mask image to obtain a fusion ratio; The first HDR image and the second HDR image are fused according to the fusion ratio to obtain the target HDR image.

2. The method according to claim 1, characterized in that The obtaining a second pixel point of the second SDR image matched by any first pixel point includes: Obtain any first pixel point and a corresponding candidate matching area on the second SDR image; Obtaining a first pixel value of the first pixel point and a candidate second pixel value within the candidate matching area; Obtaining an absolute value of a first difference between the first pixel value and the candidate second pixel value; The second pixel point that matches the first pixel point is confirmed from the candidate matching area according to the absolute value of the first difference.

3. The method according to claim 2, characterized in that The acquiring the motion vector map according to the coordinate values ​​of each of the any first pixel point and the matched second pixel point includes: Obtaining a coordinate difference between any first pixel point and the matched second pixel point to determine a lateral displacement and a longitudinal displacement; According to the lateral displacement, a first motion vector is obtained, and according to the longitudinal displacement, a second motion vector is obtained; The motion vector map is determined based on the first motion vector and the second motion vector.

4. The method according to claim 1, characterized in that: The process of acquiring the first HDR image includes: Acquire a first exposure ratio between the first SDR image and the second SDR image; The first SDR image and the second SDR image are fused according to the first exposure ratio to obtain the first HDR image.

5. The method according to claim 2, characterized in that: The process of acquiring the motion region mask map includes: Obtain a third pixel value of a third pixel point in the second SDR image that is located at the same position as the first pixel point; Obtaining an absolute value of a second difference between the first pixel value and the third pixel value; The motion region mask map is acquired according to the absolute value of the second difference.

6. An image processing device, characterized in that: include: A first acquisition module is used to acquire a first standard dynamic range SDR image and a second SDR image with different exposure times under the same scene; A second acquisition module, configured to acquire a motion vector map, a motion region mask map, and a first high dynamic range HDR image according to the first SDR image and the second SDR image; A third acquisition module, configured to acquire a second HDR image based on the first SDR image, the second SDR image and the motion vector map; a fourth acquisition module, configured to acquire a target HDR image based on the motion region mask image, the first HDR image, and the second HDR image; and the second acquisition module is further configured to: For any first pixel point on the first SDR image, obtain a second pixel point of the second SDR image that matches the any first pixel point; Obtaining respective coordinate values ​​of any first pixel point and the matched second pixel point; Acquire the motion vector map according to the coordinate values ​​of each of the first pixel point and the matched second pixel point; The third acquisition module is further used for: Acquire a second exposure ratio of the first SDR image to the second SDR image; fusing the first SDR image with the second SDR image according to the second exposure ratio and the motion vector graph to obtain a second HDR image; The fourth acquisition module is further used for: Normalizing the values ​​of the pixels in the motion region mask image to obtain a fusion ratio; The first HDR image and the second HDR image are fused according to the fusion ratio to obtain the target HDR image.

7. The device according to claim 6, characterized in that The second acquisition module is further used for: Obtain any first pixel point and a corresponding candidate matching area on the second SDR image; Obtaining a first pixel value of the first pixel point and a candidate second pixel value within the candidate matching area; Obtaining an absolute value of a first difference between the first pixel value and the candidate second pixel value; The second pixel point that matches the first pixel point is confirmed from the candidate matching area according to the absolute value of the first difference.

8. The device according to claim 7, characterized in that The second acquisition module is further used for: Obtaining a coordinate difference between any first pixel point and the matched second pixel point to determine a lateral displacement and a longitudinal displacement; According to the lateral displacement, a first motion vector is obtained, and according to the longitudinal displacement, a second motion vector is obtained; The motion vector map is determined based on the first motion vector and the second motion vector.

9. The device according to claim 6, characterized in that The second acquisition module is further used for: Acquire a first exposure ratio between the first SDR image and the second SDR image; The first SDR image and the second SDR image are fused according to the first exposure ratio to obtain the first HDR image.

10. The device according to claim 7, characterized in that The second acquisition module is further used for: Obtain a third pixel value of a third pixel point in the second SDR image that is located at the same position as the first pixel point; Obtaining an absolute value of a second difference between the first pixel value and the third pixel value; The motion region mask map is acquired according to the absolute value of the second difference.

11. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.

Images