Image processing method, device, electronic device and storage medium

Abstract

Provided are an image processing method, apparatus, electronic device, and storage medium. The image processing method includes: acquiring a first image and a second image captured for the same scene, wherein a first exposure value of the first image is less than a second exposure value of the second image; determining a first image block in the first image; searching the second image for a candidate second image block corresponding to the first image block; and fusing the first image and the second image based on a similarity between the first image block and the corresponding candidate second image block to obtain a target image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, an image processing device, an electronic device, and a computer-readable storage medium. Background Art

[0002] To obtain high dynamic range (HDR) images, multiple images acquired under different exposure conditions can be fused. However, in practical applications, if there are some fast-moving objects in the scene, directly fusing multiple images may produce ghosting.

[0003] The approaches described in this section are not necessarily approaches that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any approach described in this section is prior art simply by virtue of its inclusion in this section. Similarly, unless otherwise indicated, the issues raised in this section should not be considered as having been recognized in any prior art. Summary of the Invention

[0004] The present disclosure provides an image processing method, an electronic device, and a storage medium to achieve high-quality and efficient image fusion.

[0005] According to one aspect of the present disclosure, there is provided an image processing method, comprising: acquiring a first image and a second image captured for the same scene, wherein a first exposure value of the first image is less than a second exposure value of the second image; determining a first image block in the first image; searching the second image for a candidate second image block corresponding to the first image block; and fusing the first image and the second image based on a similarity between the first image block and the corresponding candidate second image block to obtain a target image.

[0006] According to another aspect of the present disclosure, an image processing apparatus is provided, comprising: an acquisition unit configured to acquire a first image and a second image captured for the same scene, wherein a first exposure amount of the first image is less than a second exposure amount of the second image; a first image block determination unit configured to determine a first image block in the first image; a candidate image block determination unit configured to search the second image for a candidate second image block corresponding to the first image block; and a fusion unit configured to fuse the first image and the second image based on a similarity between the first image block and the corresponding candidate second image block to obtain a target image.

[0007] According to another aspect of the present disclosure, an electronic circuit is provided, comprising: a circuit configured to execute the steps of the above method.

[0008] According to another aspect of the present disclosure, an electronic device is provided, including: a processor; and a memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the above method.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing a program is provided. The program includes instructions that, when executed by a processor of an electronic device, cause the electronic device to perform the above method.

[0010] According to another aspect of the present disclosure, a computer program product is provided, which includes a computer program, and when executed by a processor, the computer program implements the above method.

[0011] According to the embodiments of the present disclosure, by matching image blocks in a low-exposure image with image blocks in multiple transformed high-exposure images, efficient matching between images of different exposure levels can be conveniently achieved, thereby efficiently determining the matching image blocks that need to be fused to avoid ghosting in the fused image.

[0012] These and other aspects of the disclosure will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings illustrate exemplary embodiments and constitute a part of the specification. Together with the description of the specification, they serve to explain exemplary implementation of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals designate similar, but not necessarily identical, elements.

[0014] Figure 1 A flowchart showing an exemplary process of an image processing method according to an embodiment of the present disclosure is shown;

[0015] Figures 2A-2D An example of a first image block and a target second image block obtained according to an embodiment of the present disclosure is shown;

[0016] Figure 3A-Figure 3E The exemplary effect of the image fusion method according to the present disclosure is shown;

[0017] Figure 4 An exemplary block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown in FIG; and

[0018] Figure 5 is a block diagram illustrating an example of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] In this disclosure, unless otherwise specified, the use of terms such as "first" and "second" to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements. Such terms are only used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of the element, while in some cases, based on the context of the description, they may also refer to different instances.

[0020] The terms used in the descriptions of the various examples described in this disclosure are for the purpose of describing specific examples only and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element may be one or more. In addition, the term "and / or" used in this disclosure encompasses any one and all possible combinations of the listed items.

[0021] In related technologies, the location of moving objects in the scene can be determined by detecting mismatched areas in images captured at different exposure levels, so that the areas with moving objects can be specially processed when fusing images, such as reducing the weight of the areas involved in the fusion, thereby achieving the purpose of eliminating ghosting.

[0022] However, in the related art, since the exposure response of the sensor is nonlinear, it is difficult to accurately and quickly perform matching calculations on images with different exposure amounts.

[0023] In order to solve the above problems in the related art, the present disclosure provides a new image processing method to achieve fast matching of images under different exposure levels. The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0024] Figure 1 A flowchart illustrating an exemplary process of an image processing method according to an embodiment of the present disclosure is shown.

[0025] In step S102 , a first image and a second image captured for the same scene may be acquired, wherein a first exposure value of the first image is smaller than a second exposure value of the second image.

[0026] In step S104 , a first image block in the first image may be determined.

[0027] In step S106 , a candidate second image block corresponding to the first image block may be searched from the second image.

[0028] In step S108 , based on the similarity between the first image block and the corresponding candidate second image block, the first image and the second image may be fused to obtain a target image.

[0029] By utilizing the image processing method provided by the embodiments of the present disclosure, by transforming the candidate image blocks in the second image with high exposure, it is possible to solve the problem of inaccurate similarity calculation caused by the different exposure amounts of the first image and the second image, and to achieve rapid calculation of similarity and thus rapid matching of image blocks, thereby further improving the image fusion effect.

[0030] The various steps of method 100 are described in detail below.

[0031] In step S102, a first image and a second image captured for the same scene may be acquired, wherein a first exposure value of the first image may be smaller than a second exposure value of the second image.

[0032] The first image and the second image may have the same image area and the same image size.

[0033] In some embodiments, the first and second images include nearly identical objects. Because the first exposure used to acquire the first image differs from the second exposure used to acquire the second image, the first and second images have different dynamic ranges. When the first and second images are acquired at different times, there may be moving objects in the scene, resulting in different positions of the same object in the first and second images. In this case, directly fusing the first and second images will result in ghosting in the fused image.

[0034] In step S104 , a first image block in the first image may be determined.

[0035] In most cases, since the first and second images are captured relatively quickly, the objects in the images don't move much. Therefore, matching can be performed on smaller first image blocks in the first image, reducing the computational effort required for matching and improving matching accuracy.

[0036] In some embodiments, the size of the first image block may be 2s×2s. Those skilled in the art may set the size of s (e.g., 10 pixels) according to actual conditions. Different first image blocks may be extracted from the first image by traversing the first image with a step size of s.

[0037] In step S106 , a candidate second image block corresponding to the first image block may be searched from the second image.

[0038] In some embodiments, for the first image block determined in step S104, a search may be conducted within a range centered on the first image block to determine r×r candidate second image blocks for matching with the first image block. Persons skilled in the art may determine the value of r based on practical circumstances. Each candidate second image block may have the same size as the first image block, 2s×2s. In some implementations, the search step size for the second image block may be equal to the side length of the first image block. In other implementations, the search step size for the second image block may be smaller than the side length of the first image block. For example, the side length of the first image block may be an integer multiple of the search step size for the second image block, such as 2 or 4 times. Persons skilled in the art may set the search step size based on practical circumstances. By setting the search step size for the second image block to a value smaller than the side length of the first image block, overlapping image regions may exist between adjacent second image blocks. Using this method, image block search can be performed using larger image block information, while simultaneously facilitating the acquisition of smaller image blocks for image fusion.

[0039] In step S108 , based on the similarity between the first image block and the corresponding candidate second image block, the first image and the second image may be fused to obtain a target image.

[0040] In some embodiments, determining the similarity between the first image block and the corresponding candidate second image block may include determining a first similarity between the candidate second image block and the first image block based on a transformation of the candidate second image block.

[0041] As mentioned above, the exposure when acquiring the second image is greater than the exposure when acquiring the first image. Therefore, in order to more accurately match the image blocks in the second image with the image blocks in the first image, the pixel values ​​in the second image can be transformed to eliminate the problem of exposure nonlinearity of the sensor when acquiring the image.

[0042] In some embodiments, parameters for the transformation of the candidate second image block may be determined based on the first value of each first pixel in the first image block and the second value of a second pixel at a corresponding position in the candidate second image block. By determining the parameters for transforming the candidate second image block using the first value of each first pixel in the first image block and the second value of a second pixel at a corresponding position in the candidate second image block, transformation can be conveniently implemented.

[0043] In some embodiments, the transformation performed on the candidate second image block may be a linear transformation. It is understood that those skilled in the art may also perform other types of transformations on the candidate second image block, such as a logarithmic transformation, based on practical circumstances. In this disclosure, the principles of the present disclosure will be described using a linear transformation as an example, but the scope of the present disclosure is not limited thereto.

[0044] In some embodiments, the first similarity may be determined based on a difference between a second value of a second pixel in the transformed candidate second image block and a first value of a first pixel at a corresponding position in the first image block. By transforming the second pixel so that the value of the transformed second pixel is as close as possible to the value of the first pixel in the first image block, the accuracy of the image block in the second image and the image block in the first image can be improved.

[0045] The first value of the first pixel may be a brightness value determined based on the pixel values ​​of the first image, and the second value may be a brightness value determined based on the pixel values ​​of the second image. For example, the brightness value corresponding to the pixel value may be obtained by performing a logarithmic transformation on the pixel values ​​of the first image. Similarly, the brightness value corresponding to the pixel value may be obtained by performing a logarithmic transformation on the pixel values ​​of the second image. Logarithmic transformation can conveniently transform the pixel values ​​of an image into a brightness domain because the characteristics of the response curve of the sensor used to capture the image approximate the characteristics of an exponential function. In other examples, other functions may be used to process the pixel values ​​of the image to obtain the corresponding brightness value. Using this method, the similarity between images with different exposure levels can be calculated without obtaining the corresponding sensor curve.

[0046] In the case where the transformation for the candidate second image block is a linear transformation, the first similarity can be determined based on the difference between the second value of the second pixel in the candidate second image block after the linear transformation and the first value of the first pixel at the corresponding position in the first image block.

[0047] The following describes an exemplary method for determining transformation parameters by taking the transformation of the second image block as an example in which the transformation is a linear transformation.

[0048] The linear transformation of the candidate second image block is determined based on formula (1):

[0049] B LE_LOG ′=α*B LE_LOG +β (1)

[0050] Among them, B LE_LOG ′ represents the value of the second pixel in the candidate second image block after transformation, B LE_LOG represents the second value of the second pixel in the candidate second image block, and α and β are parameters of the linear transformation. By linearly mapping the candidate second image block, the problem of sensor exposure nonlinearity can be eliminated without destroying the texture structure of the image block itself.

[0051] A first similarity between the first image block and the candidate second image block may be determined based on formula (2), wherein the first similarity may be determined based on a difference between a second value of a second pixel in the candidate second image block after linear transformation and a first value of a first pixel at a corresponding position in the first image block:

[0052] D=∑ Ω ||α*B LE_LOG +β-B SE_LOG || 2 (2)

[0053] Among them, B SE_LOG is the first value of the first pixel in the first image block, B LE_LOG is the second value of the second pixel at the corresponding position in the candidate second image block, Ω is the set of all pixel positions in the image block, and α and β are parameters of the transformation performed on the candidate second image block.

[0054] The value of the first similarity D may indicate the degree of similarity between the first image block and the candidate second image block. The smaller the value of D is, the higher the degree of similarity between the first image block and the candidate second image block is.

[0055] As mentioned above, in order to obtain the most accurate similarity, it is necessary to make the transformed second image block and the first image block as similar as possible. Therefore, the parameters α and β that minimize the value of the first similarity D (i.e., the transformed second image block and the first image block are most similar) can be obtained by performing convex optimization on formula (3).

[0056]

[0057] The parameters α and β obtained by convex optimization of formula (3) can be expressed by formulas (4) and (5):

[0058]

[0059]

[0060] Among them, B SE_LOG is the first value of the first pixel in the first image block, B LE_LOG is the second value of the second pixel at the corresponding position in the candidate second image block, N is the number of pixels in the first image block, and Ω is the set of all pixel positions in the image block. For example, if the size of the first image block and the candidate second image block is 2s × 2s, N = 2s × 2s.

[0061] It can be seen that for the similarity defined based on formula (2), the parameters α and β obtained by formulas (4) and (5) can be used for calculation. Among them, the values ​​of the parameters α and β are only related to the values ​​of the pixels of the first image block and the second image block (such as the first value and the second value mentioned above) and the total number of pixels of the image blocks. When performing the image processing method provided by the present disclosure on the first image and the second image, there is no need to repeat the convex optimization process, but the parameters α and β obtained by formulas (4) and (5) can be directly used to calculate the first similarity between the currently processed first image block and multiple candidate second image blocks of the first image block. During the calculation process, the parameters that can be reused can be stored to avoid repeated calculations. For example, for the calculation of the first similarity between the same first image block and its respective candidate second image blocks, the parameters The calculation results are stored and reused to reduce the amount of calculation in the calculation process of the first similarity.

[0062] The first image and the second image may be fused based on the first similarity to obtain a target image.

[0063] In some embodiments, using the first similarity calculated using the aforementioned method, a target second image block that is most similar to the first image block can be selected from multiple candidate second image blocks of the first image block, and the target image can be obtained by fusing the first image block with the target second image block. This fusion method can be used to fuse information from high-exposure images with information from low-exposure images, thereby improving the dynamic range and image quality of the target image. Because the fusion process does not directly use image blocks at the same location, but instead fuses the most similar second image blocks after similarity comparison, it effectively avoids ghosting and other factors that affect image quality in the fused image.

[0064] Figures 2A-2D An example of a first image block and a target second image block obtained according to an embodiment of the present disclosure is shown. Figure 2A An example of a first image is shown. Figure 2A The solid-line box in indicates an example of the position of the first image block, and the dotted-line box indicates the position of the target second image block determined based on the method of the present disclosure. Figure 2B shows the image content of the first image block, Figure 2C Shown for Figure 2B The image content of the first image block determines the most similar target second image block, Figure 2D The second figure shows the Figure 2BThe image content of the second image block at the same position as the first image block in the image. It can be seen that due to the movement of the object in the image, the similarity between the second image block at the same position and the first image block is lower than the similarity between the target second image block and the first image block. Therefore, the use of Figure 2C The target second image block in Figure 2B The image quality of the target image obtained by fusing the first image block in Figure 2D The second image block in Figure 2B The image quality of the image obtained by fusing the first image block in .

[0065] In other embodiments, since the time interval between the capture of the first image and the second image is very short, the displaced object may only appear in a very small area in the image, and therefore smaller image blocks may be fused to obtain a better fusion effect.

[0066] In order to fuse image blocks of smaller size, a portion of the first image block can be determined as a first sub-image block. In some examples, the side length of the first image block can be twice the side length of the first sub-image block. The size of the first sub-image block can be one-quarter the size of the first image block. It can be understood that those skilled in the art can also determine the size of the first sub-image block to other sizes according to actual conditions. For example, the side length of the first image block can be set to Z times the side length of the first sub-image block, where Z can be an integer greater than 1. In some examples, the size of the first sub-image block can be determined based on the relationship between the search step for the second image block and the size of the first image block. For example, when the search step for the second image block is 1 / Z of the image block side length of the first image block, the size of the first sub-image block can be 1 / Z of the size of the first image block. 2 .

[0067] A plurality of second sub-image blocks corresponding to the first sub-image block in a plurality of candidate second image blocks can be determined. As mentioned above, the image can be traversed with half the side length of the first image block as a step length. Therefore, for a first sub-image block having a size of one-quarter of the first image block, four different second image blocks can be matched to the first sub-image block. The first sub-image block can be respectively calculated for a second similarity with the second sub-image blocks at the upper left, lower left, lower right, and upper right of the four different second image blocks. For each second sub-image block, the second similarity between the second sub-image block and the first sub-image block can be determined based on the transformation parameters of the candidate second image block to which the second sub-image block belongs (for example, α and β based on formulas (4) and (5)). For example, α and β determined by the pixel values ​​of the corresponding candidate second image block can be used to process the first sub-image block and the second sub-image block based on the similarity defined in formula (2) to obtain the second similarity between the second sub-image block and the first sub-image block. Based on the second similarity, a target second sub-image block that is most similar to the first sub-image block can be selected from the multiple second sub-image blocks, and the first sub-image block and the target second sub-image block can be fused to obtain the target image. Using this method, when determining the parameters for transforming the high-exposure image, larger image blocks contain more image information, thereby improving the accuracy of the similarity calculated using these parameters. Furthermore, using smaller image blocks for fusion allows for the identification of moving objects in the image at a finer granularity, further improving the image quality of the resulting fused target image.

[0068] When fusing a first image block with a target second image block or a first sub-image block with a target second sub-image block, a first weight for the first image block (or first sub-image block) and a second weight for the target second image block (or target second sub-image block) at each pixel position can be determined, and based on the corresponding first weight and second weight, the first value of the first pixel and the second value of the second pixel at each pixel position in the first image block (or first sub-image block) and the target second image block (or target second sub-image block) are fused to obtain a value at the corresponding pixel position in the target image. In some examples, the value at the corresponding pixel position in the target image can be determined by multiplying the first value of the first pixel by the first weight and the product of the second value of the second pixel by the second weight at each pixel.

[0069] In other examples, the first value of the first pixel and the second value of the second pixel may be fused based on the corresponding first weight, second weight, and brightness ratio of the second image to the first image.

[0070] The image blocks can be fused based on formula (6):

[0071] BFuse =k·(log(ratio)+B SE_LOG )+(1-k)·B LE_LOG (6)

[0072] Here, ratio may represent the brightness ratio between the second image and the first image. In some examples, the value of ratio may be determined by the ratio of the average pixel value of the second image to the average pixel value of the first image. SE_LOG It can represent the first value of the pixel in the first image block, B LE_LOG It can represent the second value of the pixel at the corresponding position in the target second image block. When the first sub-image block and the target second sub-image block are fused using formula (6), B SE_LOG It can represent the first value of the pixel in the first sub-image block, B LE_LOG It can represent the second value of the pixel at the corresponding position in the second sub-image block of the target. k represents the weight coefficient used for fusion. Fuse represents the value of the pixel at the corresponding position in the target image. In some examples, the first weight is proportional to the second value of the second pixel at the position. The weight coefficient k can be determined using formula (7):

[0073] k=[(B LE_LOG -min(I LE_log )) / (alpharate*greylevel-min(I LE_log ))] γ (7)

[0074] Among them B LE_LOG represents the second value of the pixel of the fused target second image block or target second sub-image block, min(I LE_log ) represents the minimum value of the second values ​​of all pixels in the second image, greylevel can represent the grayscale of the image, alpharate and γ can represent the debugging parameters, so that the calculated pixel value B of the target image Fuse Matches the pixel width of the image being processed. Among them, the size of alpharate*greylevel determines the grayscale level at which short exposure information starts to be used, and γ determines the amount of short exposure information used.

[0075] By traversing all first image blocks in the first image using the method provided in the present disclosure and fusing each first image block with the matching target second image block, the fused image blocks can be spliced ​​to obtain a complete ghost-free high dynamic range image.

[0076] Figure 3A-Figure 3E The following shows the exemplary effect of the image fusion method according to the present disclosure. Figure 3Aand Figure 3B shows an example of a low-exposure image and a high-exposure image used for image fusion, Figure 3C shows some details of the fused image obtained by the fusion method of the related art, Figure 3D shows some details of the fused image obtained by the ghost removal method according to the related art, Figure 3E Partial details of the fused image obtained by the image processing method according to an embodiment of the present disclosure are shown.

[0077] It can be seen that there are ghosts in the fused image obtained by the fusion method of the related art (such as Figure 3C The fused image obtained by the ghost removal method of the related art can eliminate the influence of ghosts, but the image quality is average and some detail information in the background is lost (see Figure 3D In the middle detail image, the image details in the background are severely lost, and the fused image does not reflect the detailed information reflected in the high-exposure image). Figure 3E The fused image obtained by the method according to the embodiment of the present disclosure shown in FIG has good image quality, can restore a large amount of detail information in the image background, and can eliminate the ghost phenomenon caused by moving objects.

[0078] According to an embodiment of the present disclosure, an image processing apparatus is also provided. Figure 4 , an exemplary block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown in FIG. The image processing apparatus 400 may include an acquisition unit 410 configured to acquire a first image and a second image captured for the same scene, wherein a first exposure value of the first image is less than a second exposure value of the second image; a first image block determination unit 420 configured to determine a first image block in the first image; a candidate image block determination unit 430 configured to search the second image for a candidate second image block corresponding to the first image block; and a fusion unit 440 configured to fuse the first image and the second image based on a similarity between the first image block and the corresponding candidate second image block to obtain a target image.

[0079] Here, the operations of the above-mentioned units of the image processing apparatus are similar to the operations of steps S102 to S108 described above, and are not described in detail here.

[0080] According to another aspect of the present disclosure, an electronic circuit is further provided, comprising a circuit configured to execute the steps of the above method.

[0081] According to another aspect of the present disclosure, an electronic device is provided, including: a processor; and a memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the above method.

[0082] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing a program is further provided. The program includes instructions, which, when executed by a processor of an electronic device, cause the electronic device to perform the above method.

[0083] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the above method when executed by a processor.

[0084] See also Figure 5 , electronic device 500 will now be described, which is an example of a hardware device (electronic device) that can be applied to various aspects of the present disclosure. Electronic device 500 can be any machine configured to perform processing and / or computing, and can be, but is not limited to, a workstation, server, desktop computer, laptop computer, tablet computer, personal digital assistant, robot, smartphone, vehicle-mounted computer, or any combination thereof. The above-described image processing method 100 can be implemented in whole or in part by electronic device 500 or a similar device or system.

[0085] The electronic device 500 may include elements connected to or in communication with the bus 502 (possibly via one or more interfaces). For example, the electronic device 500 may include a bus 502, one or more processors 504, one or more input devices 506, and one or more output devices 508. The one or more processors 504 may be any type of processor and may include, but are not limited to, one or more general-purpose processors and / or one or more special-purpose processors (e.g., special processing chips). The input device 506 may be any type of device capable of inputting information to the electronic device 500 and may include, but are not limited to, a mouse, a keyboard, a touch screen, a microphone, and / or a remote control. The output device 508 may be any type of device capable of presenting information and may include, but are not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The electronic device 500 may also include a non-transitory storage device 510. The non-transitory storage device may be any storage device that is non-transitory and can store data, including but not limited to a disk drive, an optical storage device, a solid-state memory, a floppy disk, a flexible disk, a hard disk, a magnetic tape, or any other magnetic medium, an optical disk or any other optical medium, a ROM (read-only memory), a RAM (random access memory), a cache memory, and / or any other memory chip or cartridge, and / or any other medium from which a computer can read data, instructions, and / or code. The non-transitory storage device 510 may be detachable from the interface. The non-transitory storage device 510 may contain data / programs (including instructions) / code for implementing the above-described methods and steps. The electronic device 500 may also include a communication device 512. The communication device 512 may be any type of device or system that enables communication with external devices and / or with a network, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication device, and / or a chipset, such as a Bluetooth™ device, an 802.11 device, a Wi-Fi device, a Wi-Max device, a cellular communication device, and / or the like.

[0086] The electronic device 500 may also include a working memory 514, which may be any type of working memory that can store programs (including instructions) and / or data useful for the operation of the processor 504, and may include but is not limited to random access memory and / or read-only memory devices.

[0087] Software elements (programs) may be located in the working memory 514, including but not limited to an operating system 516, one or more application programs 518, drivers, and / or other data and code. Instructions for executing the above-described methods and steps may be included in one or more application programs 518, and the image processing method 100 may be implemented by the processor 504 reading and executing the instructions of the one or more application programs 518. More specifically, in the image processing method 100, steps S102-S108 may be implemented, for example, by the processor 504 executing an application program 518 having instructions for performing steps S102-S108. Furthermore, other steps in the image processing method 100 may be implemented, for example, by the processor 504 executing an application program 518 having instructions for performing the corresponding steps. The executable code or source code of the instructions of the software element (program) may be stored in a non-transitory computer-readable storage medium (e.g., the storage device 510) and, upon execution, may be stored in the working memory 514 (possibly compiled and / or installed). The executable code or source code of the instructions of the software element (program) may also be downloaded from a remote location.

[0088] It should also be understood that various modifications may be made depending on specific requirements. For example, custom hardware may be used, and / or specific elements may be implemented using hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. For example, some or all of the disclosed methods and apparatus may be implemented by programming hardware (e.g., programmable logic circuits including field programmable gate arrays (FPGAs) and / or programmable logic arrays (PLAs)) using logic and algorithms according to the present disclosure in assembly language or a hardware programming language (such as VERILOG, VHDL, C++).

[0089] It should also be understood that the aforementioned method can be implemented using a server-client model. For example, the client can receive data input by a user and send the data to the server. The client can also receive data input by the user, perform a portion of the processing in the aforementioned method, and send the processed data to the server. The server can receive the data from the client, execute the aforementioned method or another portion of the aforementioned method, and return the execution results to the client. The client can receive the execution results of the method from the server and present them to the user, for example, via an output device.

[0090] It should also be understood that the components of electronic device 500 can be distributed across a network. For example, some processing can be performed by one processor while other processing can be performed by another processor remote from the processor. Other components of computing system 500 can also be similarly distributed. In this way, electronic device 500 can be interpreted as a distributed computing system that performs processing at multiple locations.

[0091] Some exemplary aspects of the disclosure are described below.

[0092] Aspect 1. An image processing method, comprising:

[0093] Acquire a first image and a second image captured for the same scene, wherein a first exposure value of the first image is smaller than a second exposure value of the second image;

[0094] determining a first image block in the first image;

[0095] searching the second image for a candidate second image block corresponding to the first image block;

[0096] Based on the similarity between the first image block and the corresponding candidate second image block, the first image and the second image are fused to obtain a target image.

[0097] Aspect 2: The image processing method according to aspect 1, wherein the similarity is determined by the following steps:

[0098] A first similarity between the candidate second image block and the first image block is determined based on the transformation of the candidate second image block.

[0099] Aspect 3. The image processing method according to Aspect 1, wherein the step size of the search is smaller than the side length of the first image block.

[0100] Aspect 4. The image processing method according to aspect 1, wherein fusing the first image and the second image based on the similarity between the first image block and the corresponding second image block comprises:

[0101] Selecting a target second image block that is most similar to the first image block from a plurality of candidate second image blocks;

[0102] The first image block and the target second image block are fused to obtain the target image.

[0103] Aspect 5. The image processing method according to Aspect 2, wherein the parameters of the transformation are determined based on the first values ​​of each first pixel in the first image block and the second values ​​of the second pixels at corresponding positions in the candidate second image block.

[0104] Aspect 6. The image processing method according to Aspect 5, wherein, based on the similarity between the first image block and the corresponding candidate second image block, fusing the first image and the second image to obtain the target image comprises:

[0105] determining a portion of the first image block as a first sub-image block;

[0106] determining a plurality of second sub-image blocks corresponding to the first sub-image block from a plurality of candidate second image blocks;

[0107] For each second sub-image block, determining a second similarity between the second sub-image block and the first sub-image block based on the transformed parameters of the candidate second image block to which the second sub-image block belongs;

[0108] determining, based on the second similarity, to select, from the plurality of second sub-image blocks, a target second sub-image block that is most similar to the first sub-image block;

[0109] The first sub-image block and the target second sub-image block are fused to obtain the target image.

[0110] Aspect 7. The method according to Aspect 6, wherein the side length of the first image block is an integer multiple of the step length.

[0111] Aspect 8. The image processing method according to Aspect 7, wherein the side length of the first image block is Z times the step length, and the size of the first sub-image block is 1 / Z times the size of the first image block. 2 .

[0112] Aspect 9. The image processing method according to aspect 6, wherein the first value is a brightness value determined based on pixel values ​​of the first image, and the second value is a brightness value determined based on pixel values ​​of the second image.

[0113] Aspect 10. The image processing method according to aspect 2, wherein the transformation is a linear transformation and is determined based on the following formula:

[0114] B LE_LOG ′=α*B LE_LOG +β

[0115] Among them, B LE_LOG ′ represents the value of the second pixel in the candidate second image block after transformation, B LE_LOG represents the second value of the second pixel in the candidate second image block, and α and β are parameters of the linear transformation.

[0116] Aspect 11. The image processing method according to Aspect 10, wherein the first similarity is determined based on a difference between a second value of a second pixel in the second image block after linear transformation and a first value of a first pixel in the first image block.

[0117] Aspect 12. The image processing method according to aspect 11, wherein the first similarity is determined based on the following formula:

[0118]

[0119] Aspect 13. The image processing method according to Aspect 12, wherein the parameters of the linear transformation are obtained by performing convex optimization on the first similarity calculation formula and are determined based on the following formula:

[0120]

[0121]

[0122] Among them, B SE_LOG is the first value of the first pixel in the first image block, B LE_LOG is the second value of the second pixel at the corresponding position in the candidate second image block, N is the number of pixels in the first image block, and Ω is the set of all pixel positions in the image block.

[0123] Aspect 14. The image processing method according to Aspect 7, wherein fusing the first sub-image block and the target second sub-image block to obtain the target image comprises:

[0124] determining a first weight for the first sub-image block and a second weight for the target second sub-image block at each pixel position;

[0125] Based on the corresponding first weight and second weight, the first value of the first pixel and the second value of the second pixel at each pixel position in the first sub-image block and the target second sub-image block are fused respectively to obtain the value at the corresponding pixel position in the target image.

[0126] Aspect 15. The image processing method according to Aspect 14, wherein the first weight is proportional to the second value of the second pixel at the position.

[0127] Aspect 16. The image processing method according to Aspect 14, wherein fusing the first value of the first pixel and the second value of the second pixel at each position in the first sub-image block and the target second sub-image block based on the corresponding first weight and second weight comprises:

[0128] The first value of the first pixel and the second value of the second pixel are fused based on the corresponding first weight, the second weight, and a brightness ratio between the second image and the first image.

[0129] Aspect 17. An image processing apparatus, comprising:

[0130] An acquisition unit is configured to acquire a first image and a second image captured for the same scene, wherein a first exposure amount of the first image is smaller than a second exposure amount of the second image;

[0131] a first image block determining unit, configured to determine a first image block in the first image;

[0132] a candidate image block determining unit, configured to search the second image for a candidate second image block corresponding to the first image block;

[0133] The fusion unit is configured to fuse the first image and the second image based on the similarity between the first image block and the corresponding candidate second image block to obtain a target image.

[0134] Aspect 18. An electronic circuit comprising:

[0135] Circuitry configured to perform the steps of the method according to any of clauses 1-16.

[0136] Aspect 19. An electronic device comprising:

[0137] processor; and

[0138] A memory storing a program, the program comprising instructions which, when executed by the processor, cause the processor to perform the method according to any one of aspects 1-16.

[0139] Aspect 20. A non-transitory computer-readable storage medium storing a program, the program comprising instructions that, when executed by a processor of an electronic device, cause the electronic device to perform the method according to any one of aspects 1-16.

[0140] Aspect 21. A computer program product, comprising a computer program, wherein the computer program implements the method according to any one of aspects 1-16 when executed by a processor.

[0141] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but is only limited by the claims after authorization and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. In addition, the steps may be performed in an order different from that described in this disclosure. Further, the various elements in the embodiments or examples may be combined in various ways. It is important that as technology evolves, many of the elements described herein may be replaced by equivalent elements that appear after this disclosure.

Claims

1. An image processing method, comprising: Acquire a first image and a second image captured for the same scene, wherein the first image and the second image have the same image area and the same image size, and wherein a first exposure value of the first image is smaller than a second exposure value of the second image; determining a first image block in the first image; Searching the second image for a candidate second image block corresponding to the first image block, comprising searching the second image within a range centered on the first image block to determine a plurality of candidate second image blocks, wherein a step length of the search is smaller than a side length of the first image block; Based on the similarity between the first image block and the corresponding candidate second image block, the first image and the second image are fused to obtain a target image, The similarity is determined by the following steps: determining a first similarity between the candidate second image block and the first image block based on the transformation of the candidate second image block, The transformation parameters are determined based on the first value of each first pixel in the first image block and the second value of the second pixel at the corresponding position in the candidate second image block.

2. The image processing method according to claim 1, wherein: Based on the similarity between the first image block and the corresponding second image block, fusing the first image and the second image includes: selecting a target second image block that is most similar to the first image block from a plurality of candidate second image blocks; The first image block and the target second image block are fused to obtain the target image.

3. The image processing method according to claim 1, wherein: Based on the similarity between the first image block and the corresponding candidate second image block, fusing the first image and the second image to obtain a target image includes: determining a portion of the first image block as a first sub-image block; determining a plurality of second sub-image blocks corresponding to the first sub-image block from a plurality of candidate second image blocks; For each second sub-image block, determining a second similarity between the second sub-image block and the first sub-image block based on the transformed parameters of the candidate second image block to which the second sub-image block belongs; determining, based on the second similarity, to select, from the plurality of second sub-image blocks, a target second sub-image block that is most similar to the first sub-image block; The first sub-image block and the target second sub-image block are fused to obtain the target image.

4. The method according to claim 3, wherein: The side length of the first image block is an integer multiple of the step length.

5. The image processing method according to claim 4, wherein the side length of the first image block is Z times the step length, and the size of the first sub-image block is 1 / Z times the size of the first image block. 2 . 6 . The image processing method according to claim 3 , wherein the first value is a brightness value determined based on pixel values ​​of the first image, and the second value is a brightness value determined based on pixel values ​​of the second image.

7. The image processing method according to claim 1, wherein: The transformation is a linear transformation and is determined based on the following formula: in, represents the value of the second pixel in the candidate second image block after the transformation, represents a second value of a second pixel in the candidate second image block, 、 are the parameters of the linear transformation.

8. The image processing method according to claim 7, wherein: The first similarity is determined based on a difference between a second value of a second pixel in the second image block after linear transformation and a first value of a first pixel in the first image block.

9. The image processing method according to claim 8, wherein: The first similarity is determined based on the following formula: in, is the first value of the first pixel in the first image block.

10. The image processing method according to claim 9, wherein the parameters of the linear transformation are obtained by performing convex optimization on the first similarity calculation formula and are determined based on the following formula: in, is the second value of the second pixel at the corresponding position in the candidate second image block, is the number of pixels in the first image block, and is the set of all pixel locations in the image block.

11. The image processing method according to claim 4, wherein: Fusing the first sub-image block and the target second sub-image block to obtain the target image includes: determining a first weight for the first sub-image block and a second weight for the target second sub-image block at each pixel position; Based on the corresponding first weight and second weight, the first value of the first pixel and the second value of the second pixel at each pixel position in the first sub-image block and the target second sub-image block are fused respectively to obtain the value at the corresponding pixel position in the target image.

12. The image processing method according to claim 11, wherein: The first weight is proportional to a second value of a second pixel at the position.

13. The image processing method according to claim 11, wherein: The step of fusing the first value of the first pixel and the second value of the second pixel at each position in the first sub-image block and the target second sub-image block based on the corresponding first weight and second weight includes: The first value of the first pixel and the second value of the second pixel are fused based on the corresponding first weight, the second weight, and a brightness ratio between the second image and the first image.

14. An image processing apparatus, comprising: an acquisition unit configured to acquire a first image and a second image captured with respect to the same scene, wherein the first image and the second image have the same image area and the same image size, and wherein a first exposure amount of the first image is smaller than a second exposure amount of the second image; a first image block determining unit, configured to determine a first image block in the first image; a candidate image block determining unit, configured to search the second image for a candidate second image block corresponding to the first image block, comprising searching the second image within a range centered on the first image block to determine a plurality of candidate second image blocks, wherein a step length of the search is smaller than a side length of the first image block; a fusion unit configured to fuse the first image and the second image to obtain a target image based on a similarity between the first image block and the corresponding candidate second image block, The similarity is determined by the following steps: determining a first similarity between the candidate second image block and the first image block based on the transformation of the candidate second image block, The transformation parameters are determined based on the first value of each first pixel in the first image block and the second value of the second pixel at the corresponding position in the candidate second image block.

15. An electronic circuit comprising: Circuitry configured to perform the steps of the method according to any one of claims 1-13.

16. An electronic device comprising: processor; as well as A memory storing a program, the program comprising instructions which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 13. 17 . A non-transitory computer-readable storage medium storing a program, the program comprising instructions, which, when executed by a processor of an electronic device, cause the electronic device to perform the method according to claim 1 .

18. A computer program product comprising a computer program, wherein The computer program implements the method according to any one of claims 1 to 13 when executed by a processor.

Images