Image processing methods and apparatuses, storage media and electronic devices

Abstract

This disclosure relates to the field of image processing technology, specifically to an image processing method and apparatus, a computer-readable storage medium, and an electronic device. The method includes: acquiring at least two candidate images with different exposure information captured on the same subject, and dividing the candidate images into a reference image and at least one image to be optimized; acquiring multiple sets of initial frequency coefficients between the image to be optimized and the reference image; determining a target offset between the image to be optimized and the reference image based on the multiple sets of initial frequency coefficients; and using the target offset to optimize the image to obtain a target image. The technical solutions of the embodiments of this disclosure improve the accuracy of image processing.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to an image processing method and apparatus, a computer-readable storage medium, and an electronic device. Background Technology

[0002] As users demand higher image quality, image optimization techniques are being used more and more widely, such as HDR (High Dynamic Range) technology and image denoising technology.

[0003] However, the image optimization techniques in related technologies lack sufficient processing precision, resulting in images whose quality fails to meet user needs.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide an image processing method, an image processing apparatus, a computer-readable medium, and an electronic device, thereby improving the accuracy of image processing to at least a certain extent.

[0006] According to a first aspect of this disclosure, an image processing method is provided, comprising: acquiring at least two candidate images with different exposure information captured on the same subject, and dividing the candidate images into a reference image and at least one image to be optimized; acquiring multiple sets of initial frequency coefficients between the image to be optimized and the reference image; determining a target offset between the image to be optimized and the reference image based on the multiple sets of initial frequency coefficients; and optimizing the image to be optimized using the target offset to obtain a target image.

[0007] According to a second aspect of this disclosure, an image processing apparatus is provided, comprising: a segmentation module, configured to acquire at least two candidate images with different exposure information captured on the same subject, and segment the candidate images into a reference image and at least one image to be optimized; an acquisition module, configured to acquire multiple sets of initial frequency coefficients between the image to be optimized and the reference image; a determination module, configured to determine a target offset between the image to be optimized and the reference image based on the multiple sets of initial frequency coefficients; and an optimization module, configured to optimize the image to be optimized using the target offset to obtain a target image.

[0008] According to a third aspect of this disclosure, a computer-readable medium is provided that stores a computer program thereon, which, when executed by a processor, implements the method described above.

[0009] According to a fourth aspect of this disclosure, an electronic device is provided, characterized in that it includes: one or more processors; and a memory for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the method described above.

[0010] One embodiment of this disclosure provides an image processing method that acquires at least two candidate images with different exposure information captured from the same subject, and divides the candidate images into a reference image and at least one image to be optimized; acquires multiple sets of initial frequency coefficients between the image to be optimized and the reference image; determines the target offset between the image to be optimized and the reference image based on the multiple sets of initial frequency coefficients; and optimizes the image to be optimized using the target offset to obtain the target image. Compared with the prior art, on the one hand, image processing using images with different exposure information can make the obtained target image richer in information. On the other hand, using initial frequency coefficients to determine the target offset between the image to be optimized and the reference image saves computational resources and can process different frequency domains, thereby improving the quality of the obtained target image.

[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0013] Figure 1 A schematic diagram of an exemplary system architecture to which embodiments of the present disclosure may be applied is shown;

[0014] Figure 2 A flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure is shown schematically.

[0015] Figure 3 This schematically illustrates a flowchart of determining a target offset in an exemplary embodiment of the present disclosure;

[0016] Figure 4 This schematically illustrates a flowchart of updating an intermediate offset in an exemplary embodiment of the present disclosure;

[0017] Figure 5 This schematically illustrates a flowchart of another method for updating intermediate offsets in an exemplary embodiment of this disclosure;

[0018] Figure 6 This schematically illustrates a data flow diagram of an image processing method in an exemplary embodiment of the present disclosure;

[0019] Figure 7 This schematically illustrates a data flow diagram of another image processing method in an exemplary embodiment of the present disclosure;

[0020] Figure 8 This schematically illustrates the data flow diagram of another image processing method in an exemplary embodiment of the present disclosure;

[0021] Figure 9 This schematically illustrates a data flow diagram of yet another image processing method in an exemplary embodiment of the present disclosure;

[0022] Figure 10 This schematically illustrates a flowchart of an image optimization process according to an exemplary embodiment of the present disclosure;

[0023] Figure 11 This schematically illustrates a flowchart of the fusion of a reference image and a target image in an exemplary embodiment of the present disclosure;

[0024] Figure 12 This schematic diagram illustrates the composition of an image processing apparatus in an exemplary embodiment of the present disclosure.

[0025] Figure 13 A schematic diagram of an electronic device to which embodiments of the present disclosure may be applied is shown. Detailed Implementation

[0026] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0027] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0028] Figure 1A schematic diagram of the system architecture is shown. System architecture 100 may include a terminal 110 and a server 120. The terminal 110 may be a smartphone, tablet, desktop computer, laptop, or other terminal device. The server 120 generally refers to the backend system providing image processing-related services in this exemplary embodiment, and may be a single server or a cluster of multiple servers. The terminal 110 and server 120 can be connected via wired or wireless communication links for data interaction.

[0029] In one implementation, the image processing method described above can be executed by the terminal 110. For example, after a user takes a picture using the terminal 110 or selects at least two candidate images with different exposure information from the photo album of the terminal 110, the terminal 110 performs image optimization processing on the image and outputs the target image.

[0030] In one implementation, the image processing method described above can be executed by server 120. For example, after a user takes a picture using terminal 110 or selects at least two candidate images with different exposure information from the album of terminal 110, terminal 110 uploads the image to server 120, and server 120 performs image optimization processing on the image and returns the target image to terminal 110.

[0031] As can be seen from the above, the execution subject of the image processing method in this exemplary embodiment can be the aforementioned terminal 110 or server 120, and this disclosure does not limit it in this regard.

[0032] The image processing method disclosed herein can be applied to HDR technology, that is, to generate high dynamic range images from low dynamic range images, and can also be applied to image noise reduction, as well as to other technical fields. The following describes its application in conjunction with... Figure 2 The image processing method in this exemplary embodiment will be described. Figure 2 An exemplary flow of this image processing method is shown, which may include:

[0033] Step S210: Obtain at least two candidate images with different exposure information captured on the same subject, and divide the candidate images into a reference image and at least one image to be optimized;

[0034] Step S220: Obtain multiple sets of initial frequency coefficients between the image to be optimized and the reference image;

[0035] Step S230: Determine the target offset between the image to be optimized and the reference image based on the multiple sets of initial frequency coefficients;

[0036] Step S240: Optimize the image to be optimized using the target offset to obtain the target image.

[0037] Based on the above method, on the one hand, image processing using images with different exposure information can enrich the information of the obtained target image. On the other hand, using initial frequency coefficients to determine the target offset between the image to be optimized and the reference image can save computational resources while processing in different frequency domains, thereby improving the quality of the obtained target image.

[0038] The following is about Figure 2 Each step in the process will be explained in detail.

[0039] refer to Figure 2 In step S210, at least two candidate images with different exposure information are acquired from the same subject, and the candidate images are divided into a reference image and at least one image to be optimized.

[0040] In one exemplary embodiment of this disclosure, at least two candidate images with different exposure information of the same subject can be acquired by taking a photograph. These candidate images can be RAW domain images, which contain rich information and facilitate better image processing. The candidate images can also be RGB domain images or YUV domain images; this exemplary embodiment does not impose any specific limitations.

[0041] The exposure information mentioned above may include exposure time and exposure gain data.

[0042] In this example implementation, after obtaining candidate images with different exposure information, the candidate images can be divided into a reference image and at least one image to be optimized.

[0043] Specifically, when there are two candidate images, the exposure times of the two candidate images can be compared. The candidate image with the longer exposure time is then used as the image to be optimized, and the image with the shorter exposure time is used as the reference image.

[0044] When the number of candidate images is greater than two, the median exposure time of the candidate images can be obtained. The candidate image with the smallest absolute difference between its exposure time and the median is then used as the reference image. For example, if the number of candidate images is odd (e.g., 3, 5), the candidate image corresponding to the median exposure time can be used as the reference image, and the other images are used as images to be optimized. Conversely, if the number of candidate images is even (e.g., 4, 6), the median can be obtained. Since the median is the average exposure time of two images, the candidate image with the smallest absolute difference between its exposure time and the median can be used as the reference image, and the other images are used as images to be optimized.

[0045] In step S220, multiple sets of initial frequency coefficients between the image to be optimized and the reference image are obtained.

[0046] In one exemplary embodiment of this disclosure, after dividing the candidate image into an image to be optimized and a reference image, multiple sets of initial frequency coefficients between each image to be optimized and the reference image can be obtained. Specifically, wavelet decomposition can be used to obtain multiple sets of initial frequency coefficients between the image to be optimized and the reference image, and Haar wavelet decomposition can be used to obtain the initial frequency coefficients.

[0047]

[0048] in,

[0049] Where X is the decomposed image.

[0050] In this example implementation, Haar wavelet decomposition can yield four sets of initial frequency coefficients, including low-frequency coefficients, two sets of mid-frequency coefficients, and one set of high-frequency coefficients.

[0051] It should be noted that wavelet decomposition methods can also include Morlet wavelet decomposition, Mexican hat wavelet decomposition, Gaussian wavelet decomposition, etc., and are not specifically limited in this example implementation.

[0052] It should be noted that Haar wavelet decomposition can yield four sets of initial frequency coefficients. The number of initial frequency coefficients obtained by other wavelet decomposition methods is not limited, and they can be further refined into high-frequency coefficients, mid-frequency coefficients, and low-frequency coefficients.

[0053] In step S230, the target offset between the image to be optimized and the reference image is determined based on the multiple sets of initial frequency coefficients.

[0054] In one exemplary embodiment of this disclosure, reference is made to Figure 3 As shown, the above steps may include steps S310 to S320.

[0055] In step S310, the initial offset is obtained by applying a matching algorithm constraint to at least one set of the initial frequency coefficients.

[0056] In this example implementation, after obtaining multiple sets of initial frequency coefficients, a matching algorithm constraint can be applied to at least one set of frequency table coefficients to obtain the initial offset.

[0057] Preferably, a set of low-frequency coefficients can be selected for matching algorithm constraints to obtain the initial offset. Specifically, the images corresponding to the reference images in the low-frequency coefficients are traversed with a first preset step size. At the current coordinate (i, j), a first image block with a radius of R1 is taken. Then, with a step size of 1, image windows with a radius of R2 are traversed near the coordinate (i, j) of the image corresponding to the image to be optimized in the low-frequency coefficients. At the current coordinate (m, n) in the image window, a second image block with a radius of R1 is taken. Then, matching algorithm constraints are applied to the first image block and the second image block. The matching algorithm can include absolute difference matching, square difference matching, correlation coefficient matching, and normalized matching algorithms, etc., which are not specifically limited in this example implementation.

[0058] Output the current coordinates (m, n) of the second image patch that has the highest similarity to the first image patch, and calculate the first offset, specifically:

[0059] offsetY = m

[0060] offsetX = n

[0061] MV1_X = offsetX - i

[0062] MV1_Y = offsetY - j

[0063] After obtaining the first offset, we can use the image windows with radii of R2+MV1_X and R2+MV1_Y, which are located near the coordinates (i, j) of the image to be optimized in the low-frequency coefficients, to traverse the image windows of the image windows to be optimized, with the first offset not being 1. We can take the third image block with radius R1 at the current coordinate (m, n) in the image window. Then, we can apply the matching algorithm constraint to the first image block and the third image block, output the current coordinates (m, n) of the third image block with the highest similarity to the first image block, and calculate the second offset. The calculation formula is the same as the calculation method of the first offset. We can repeat the above process, except that when the offset converges, we output the converged offset to obtain the initial offset.

[0064] It should be noted that the frequency coefficients mentioned above are all matrices, which can be understood as frequency coefficients composed of pixel matrices. That is, the frequency coefficients here can be represented as images, and the input of the matching constraint algorithm mentioned above can be two images.

[0065] In another example implementation, an initial offset can be obtained by using a matching algorithm to constrain each set of initial frequency table coefficients. In this case, the initial offset may include multiple sets of data, each set of data corresponding to a set of initial frequency coefficients. The calculation process for the initial offset corresponding to each set of initial frequency coefficients has been described in detail above, so it will not be repeated here.

[0066] In step S320, the initial offset is constrained, and the target offset is determined based on the constraint result.

[0067] In one example embodiment of this disclosure, after obtaining the initial frequency coefficients, the initial offset can be updated using the low-frequency coefficients from multiple sets of initial frequency coefficients to obtain the target offset.

[0068] Specifically, refer to Figure 4 As shown, the update process may include steps S410 to S440.

[0069] In step S410, the low-frequency coefficients are decomposed by wavelet to obtain the first layer of intermediate frequency coefficients.

[0070] In this real-time example, after processing the multiple initial frequency coefficients mentioned above, wavelet decomposition can be performed on the low-frequency coefficients to obtain the first-level intermediate frequency coefficients. The low-frequency coefficients contain more information, which makes the update of the initial offset more accurate.

[0071] The specific details of wavelet decomposition have already been explained in detail above, so they will not be repeated here.

[0072] In step S420, wavelet decomposition is performed on the intermediate low-frequency coefficients in the (N-1)th layer intermediate frequency coefficients to obtain the Nth layer intermediate frequency coefficients.

[0073] In this example implementation, the intermediate low-frequency coefficients in the (N-1)th layer intermediate frequency coefficients can be decomposed using wavelet decomposition to obtain the Nth layer intermediate frequency coefficients. Specifically, the intermediate frequency coefficients of each layer include high-frequency coefficients, mid-frequency coefficients, and low-frequency coefficients. During the update process, decomposing the intermediate low-frequency coefficients of the corresponding layer using wavelet decomposition to obtain the intermediate frequency coefficients of the next layer can retain more information.

[0074] The specific details of wavelet decomposition have already been explained in detail above, so they will not be repeated here.

[0075] In this example implementation, the intermediate frequency coefficients of the (N-1)th layer and the intermediate frequency coefficients of the Nth layer are any two adjacent intermediate frequency coefficients; N is a positive integer greater than or equal to 2.

[0076] In step S430, the intermediate offset corresponding to the intermediate frequency coefficient of the (N-1)th layer is updated based on the intermediate frequency coefficient of the Nth layer.

[0077] In one exemplary embodiment of this disclosure, the aforementioned intermediate frequency coefficients can be updated M times, referring to... Figure 5 As shown, the above steps may include steps S510 to S520.

[0078] In step S510, in response to the fact that N is less than M, the intermediate offset of the Nth layer is determined based on the intermediate offset of the Nth layer and the intermediate frequency coefficient of the (N-1)th layer.

[0079] In this example implementation, when N is less than or equal to M, that is, when N is not at its maximum value and the Nth layer is not the bottom layer, the middle offset of the N-1th layer can be determined based on the middle offset of the Nth layer and the middle frequency coefficient of the N-1th layer.

[0080] Specifically, in one example implementation, the intermediate offset of the Nth layer can be obtained by using a matching algorithm to constrain the intermediate frequency coefficient of the (N-1)th layer.

[0081] At this point, the intermediate offsets that can be obtained include multiple sets, which may include multiple sets of intermediate offsets corresponding to each set of frequency coefficients.

[0082] In this example implementation, the intermediate offset of the Nth layer can be calculated by applying a matching algorithm constraint to the intermediate frequency coefficients of the Nth layer. In this example implementation, the matching algorithm constraint can be applied to multiple sets of frequency coefficients in the intermediate frequency coefficients of the Nth layer, or only the low-frequency coefficients in the intermediate frequency coefficients of the Nth layer can be constrained to obtain the intermediate offset of the Nth layer, which is then applied to the other frequency coefficients in the Nth layer. The specific process of the matching constraint algorithm has been described in detail above, and therefore will not be repeated here.

[0083] In another example implementation, the intermediate offset of the Nth layer and the intermediate frequency coefficient of the Nth layer can be reconstructed by wavelet to obtain the intermediate image of the Nth layer. Then, the intermediate frequency coefficient of the (N-1)th layer can be reconstructed by wavelet to obtain the intermediate image of the (N-1)th layer. After that, the intermediate offset of the (N-1)th layer can be obtained by using the matching algorithm constraint between the intermediate image of the Nth layer and the intermediate image of the (N-1)th layer.

[0084] The specific operation of the matching algorithm constraint can be referred to the process of applying the matching algorithm constraint to low-frequency coefficients to obtain the initial offset, which will not be elaborated here.

[0085] In step S520, in response to N equaling M, a matching algorithm is performed on at least one set of intermediate frequency coefficients of the Nth layer to obtain the intermediate offset of the Nth layer, and the intermediate offset of the Nth layer is upsampled to obtain the intermediate offset of the (N-1)th layer.

[0086] In this example implementation, when N equals M, it means that the Nth layer is the bottom layer. At this time, the intermediate frequency coefficients of the Nth layer can be constrained by a matching algorithm to obtain the intermediate offset of the Nth layer. Then, the intermediate offset of the Nth layer is upsampled to obtain the intermediate offset of the (N-1)th layer.

[0087] In step S440, the initial offset is updated using the intermediate offset corresponding to the intermediate frequency coefficient of the first layer to obtain the target offset.

[0088] In this example implementation, the updates proceed sequentially upwards, eventually updating the intermediate offset corresponding to the intermediate frequency coefficient of the first layer. Then, the initial offset can be updated using the intermediate offset corresponding to the first layer. Specifically, the initial offset can be updated by using the intermediate offset corresponding to the first layer and the initial frequency coefficient as a matching algorithm constraint, and the updated initial offset is upsampled to obtain the target offset.

[0089] In another example embodiment of this disclosure, the target offset can be obtained by directly upsampling the initial offset.

[0090] The following detailed explanation uses a single image to be optimized as an example. For specific details, please refer to... Figure 6 As shown, wavelet decomposition is first performed on the image to be optimized and the reference image to obtain the initial frequency coefficients. Specifically, this may include a set of high-frequency coefficients L-HH1 and S-HH1, and two sets of mid-frequency coefficients, including L-HL1 and S-HL1, and L-LH1 and S-LH1, as well as a set of low-frequency coefficients L-LL1 and S-LL1. Among them, L-HH1, L-HL1, L-LH1, and L-LL1 are the frequency coefficients corresponding to the image to be optimized, and S-HH1, S-HL1, S-LH1, and S-LL1 are the frequency coefficients corresponding to the reference image.

[0091] The low-frequency coefficients can be further decomposed into wavelet coefficients to obtain the first-level intermediate frequency coefficients. Specifically, the L-LL1 and S-LL1 can be decomposed into wavelet coefficients to obtain the first-level intermediate frequency coefficients. Specifically, the first-level intermediate frequency coefficients can include a set of high-frequency coefficients L-HH2 and S-HH2, two sets of intermediate frequency coefficients including L-HL2 and S-HL2, L-LH2 and S-LH2, and a set of low-frequency coefficients L-LL2 and S-LL2.

[0092] Then, wavelet decomposition is performed on the low-frequency coefficients corresponding to the first-layer intermediate frequency coefficients to obtain the second-layer intermediate frequency coefficients. Specifically, it may include a set of high-frequency coefficients L-HH3 and S-HH3, and two sets of intermediate frequency coefficients, including L-HL3 and S-HL3, and L-LH3 and S-LH3, as well as a set of low-frequency coefficients L-LL3 and S-LL3.

[0093] In this example implementation, in one example implementation of this disclosure, the intermediate offset corresponding to the second layer low-frequency coefficient can be obtained by performing a matching algorithm constraint on each of the above-mentioned frequency coefficients. Specifically, it can include MV3-LL, MV3-LH, MV3-HL, and MV3-HH. Then, the intermediate offset is upsampled to obtain the candidate offset corresponding to the first layer intermediate frequency coefficient. Specifically, it can include MV2-LL, MV2-LH, MV2-HL, and MV2-HH.

[0094] In one example implementation, the intermediate offset of the first layer can be applied to the intermediate frequency coefficients of the first layer to update the intermediate frequency coefficients of the first layer. Specifically, the updated intermediate frequency coefficients of the first layer may include a set of high-frequency coefficients L-HH2+MV2-HH and S-HH2+MV2-HH, and two sets of mid-frequency coefficients, including L-HL2+MV2-HL and S-HL2+MV2-HL, and L-LH2+MV2-LH and S-LH2+MV2-LH, and a set of low-frequency coefficients L-LL2+MV2-LL and S-LL2+MV2-LL.

[0095] The following example illustrates how to apply the intermediate offset to the intermediate frequency coefficients. Specifically, the above L-HH2 is copied to obtain the first L-HH2 and the second L-HH2. The first L-HH2 is traversed with a preset step size. A fourth image block with a radius of R3 is taken at the coordinates (i+MV2-HH, j+MV2-HH). Then, the second L-HH2 is traversed with a preset step size. A fifth image block with a radius of R3 is taken at the coordinates (i, j). The pixel values ​​in the fifth image block are then updated using the average of the fourth and fifth image blocks. After traversing each pixel, the second image to be optimized is used as the above L-HH2+MV2-HH.

[0096] Then, a matching algorithm constraint can be applied to each of the above frequency coefficients to obtain the intermediate offsets corresponding to the intermediate frequency coefficients of the first layer, namely MV20-LL, MV20-LH, MV20-HL, and MV20-HH.

[0097] After updating the intermediate offsets corresponding to the first-layer frequency coefficients, the intermediate offsets corresponding to the first-layer intermediate frequency coefficients can be upsampled to obtain initial candidate offsets, which may include MV1-LL, MV1-LH, MV1-HL, and MV1-HH. The obtained candidate offsets are then used to update the initial frequency coefficients. The updated initial frequency coefficients may include a set of high-frequency coefficients L-HH1+MV1-HH and S-HH1+MV1-HH, and two sets of mid-frequency coefficients, including L-HL1+MV1-HL and S-HL1+MV1-HL, and L-LH1+MV1-LH and S-LH1+MV1-LH, and a set of low-frequency coefficients L-LL1+MV1-LL and S-LL1+MV1-LL.

[0098] It should be noted that the process of updating the above frequency coefficients using the intermediate offset can be referred to the explanation of applying MV2-HH to L-HH2 to obtain L-HH2+MV2-HH, which will not be repeated here.

[0099] Then, a matching algorithm constraint can be applied to each of the above frequency coefficients to obtain the updated initial offset, which can specifically include MV10-LL, MV10-LH, MV10-HL, and MV10-HH. Then, the updated initial offset can be upsampled to obtain the target offset, which can specifically include MV0-LL, MV0-LH, MV0-HL, and MV0-HH.

[0100] In another example implementation, refer to Figure 7 As shown, after obtaining the intermediate offset corresponding to the intermediate frequency coefficients of the second layer, the intermediate frequency coefficients of the second layer can be updated using the intermediate frequency coefficients of the second layer. The updated intermediate frequency coefficients can include a set of high-frequency coefficients L-HH3+MV3-HH and S-HH3+MV3-HH, and two sets of mid-frequency coefficients, including L-HL3+MV3-HL and S-HL3+MV3-HL, and L-LH3+MV3-LH and S-LH3+MV3-LH, and a set of low-frequency coefficients L-LL3+MV3-LL and S-LL3+MV3-LL. Then, wavelet reconstruction is performed on the frequency coefficients corresponding to the updated intermediate image to be optimized in the second layer to obtain the intermediate image of the second layer. Then, the low-frequency coefficients corresponding to the reference image in the first layer, namely S-LL2, are used to perform a matching constraint algorithm to obtain the intermediate offset corresponding to the first layer.

[0101] After obtaining the intermediate offset MV2 corresponding to the intermediate frequency coefficients of the first layer, the frequency coefficients corresponding to the first layer image to be optimized can be updated using the intermediate offset MV2. The updated frequency coefficients corresponding to the first layer image to be optimized include high frequency coefficients L-HH2+MV2, mid frequency coefficients L-HL2+MV2 and L-LH2+MV2, and a set of low frequency coefficients L-LL2+MV2. Then, wavelet reconstruction is performed on the frequency coefficients corresponding to the updated first layer image to be optimized to obtain the first layer intermediate image corresponding to the first layer frequency coefficients. Then, based on the low frequency coefficients corresponding to the reference image in the initial frequency coefficients, a matching algorithm constraint is performed to obtain the updated initial offset.

[0102] In this example implementation, the frequency coefficients corresponding to the image to be optimized in the initial frequency coefficients can be updated using the updated initial offset. The updated frequency coefficients corresponding to the image to be optimized include the intermediate frequency coefficients L-HL1+MV1 and L-LH1+MV1 and a set of low frequency coefficients L-LL1+MV1. Then, wavelet reconstruction is performed on it to obtain the initial intermediate image. Then, the target offset is obtained by using the matching algorithm between the initial intermediate image and the reference image.

[0103] In one example implementation, the two schemes described above can be nested. For instance, in odd-numbered layers, the offset is directly updated using frequency coefficients, resulting in an intermediate offset that includes the offsets corresponding to high-frequency, mid-frequency, and low-frequency coefficients. In even-numbered layers, the offset is calculated using the image obtained after wavelet reconstruction of the frequency coefficients, resulting in an overall intermediate offset. This overall intermediate offset can be copied into four copies, representing the offsets corresponding to high-frequency, mid-frequency, and low-frequency coefficients respectively, to connect the two schemes.

[0104] It should be noted that each of the above two methods can be used for each layer, and no specific limitation is made in this example implementation.

[0105] In another example implementation, the target offset can be obtained directly by upsampling the initial offset, that is, the initial offset does not need to be updated.

[0106] In one exemplary embodiment of this disclosure, when performing wavelet decomposition on the image to be optimized and the reference image to obtain multiple sets of initial frequency coefficients, the reference image and the image to be optimized can first be downsampled to obtain the sub-image to be optimized and the reference sub-image; wavelet decomposition is then performed on the reference sub-image and the image to be optimized to obtain the initial frequency coefficients.

[0107] Specifically, refer to Figure 8 As shown, taking the image to be processed as an example, the above scheme is explained in detail. First, the exposure ratio is calculated based on the exposure time and exposure gain data. Then, the image to be optimized is downsampled to obtain the sub-image to be optimized. Finally, the product of the sub-image to be optimized and the exposure ratio is used as the reference sub-image. It should be noted that the reference sub-image can also be obtained by directly downsampling the reference image. The downsampling factor can be 2x, 4x, etc., or it can be customized according to user needs. In this example implementation, no specific limitation is made.

[0108] In this example embodiment, after obtaining the sub-image to be optimized and the reference sub-image, wavelet decomposition can be performed to obtain the initial frequency coefficients. Specifically, it can include a set of high-frequency coefficients L-HH1 and S-HH1, and two sets of mid-frequency coefficients, including L-HL1 and S-HL1, and L-LH1 and S-LH1, and a set of low-frequency coefficients L-LL1 and S-LL1, wherein L-HH1, L-HL1, L-LH1, and L-LL1 are the frequency coefficients corresponding to the sub-image to be optimized, and S-HH1, S-HL1, S-LH1, and S-LL1 are the frequency coefficients corresponding to the reference image.

[0109] The low-frequency coefficients can be further decomposed into wavelet coefficients to obtain the first-level intermediate frequency coefficients. Specifically, the L-LL1 and S-LL1 can be decomposed into wavelet coefficients to obtain the first-level intermediate frequency coefficients. Specifically, the first-level intermediate frequency coefficients can include a set of high-frequency coefficients L-HH2 and S-HH2, two sets of intermediate frequency coefficients including L-HL2 and S-HL2, L-LH2 and S-LH2, and a set of low-frequency coefficients L-LL2 and S-LL2.

[0110] Then, wavelet decomposition is performed on the low-frequency coefficients corresponding to the first-layer intermediate frequency coefficients to obtain the second-layer intermediate frequency coefficients. Specifically, it may include a set of high-frequency coefficients L-HH3 and S-HH3, and two sets of intermediate frequency coefficients, including L-HL3 and S-HL3, and L-LH3 and S-LH3, as well as a set of low-frequency coefficients L-LL3 and S-LL3.

[0111] In this example implementation, in one example implementation of this disclosure, the intermediate offset corresponding to the second layer low-frequency coefficient can be obtained by performing a matching algorithm constraint on each of the above-mentioned frequency coefficients. Specifically, it can include MV3-LL, MV3-LH, MV3-HL, and MV3-HH. Then, the intermediate offset is upsampled to obtain the candidate offset corresponding to the first layer intermediate frequency coefficient. Specifically, it can include MV2-LL, MV2-LH, MV2-HL, and MV2-HH.

[0112] In one example implementation, the intermediate offset of the first layer can be applied to the intermediate frequency coefficients of the first layer to update the intermediate frequency coefficients of the first layer. Specifically, the updated intermediate frequency coefficients of the first layer may include a set of high-frequency coefficients L-HH2+MV2-HH and S-HH2+MV2-HH, and two sets of mid-frequency coefficients, including L-HL2+MV2-HL and S-HL2+MV2-HL, and L-LH2+MV2-LH and S-LH2+MV2-LH, and a set of low-frequency coefficients L-LL2+MV2-LL and S-LL2+MV2-LL.

[0113] In this example implementation, the following describes in detail the application of intermediate offset to intermediate frequency coefficients by applying MV2-HH to L-HH2 to obtain L-HH2+MV2-HH. Specifically, the above L-HH2 is copied to obtain a first L-HH2 and a second L-HH2. The first L-HH2 is traversed with a preset step size. A fourth image block with a radius of R3 is taken at coordinates (i+MV2-HH, j+MV2-HH). Then, the second L-HH2 is traversed with a preset step size. A fifth image block with a radius of R3 is taken at coordinates (i, j). Then, the pixel values ​​in the fifth image block are updated using the mean of the fourth and fifth image blocks. After the iteration is completed, the second image to be optimized is used as the above L-HH2+MV2-HH.

[0114] Then, a matching algorithm constraint can be applied to each of the above frequency coefficients to obtain the intermediate offsets corresponding to the intermediate frequency coefficients of the first layer, namely MV20-LL, MV20-LH, MV20-HL, and MV20-HH.

[0115] After updating the intermediate offsets corresponding to the first-layer frequency coefficients, the intermediate offsets corresponding to the first-layer intermediate frequency coefficients can be upsampled to obtain initial candidate offsets, which may include MV1-LL, MV1-LH, MV1-HL, and MV1-HH. The obtained candidate offsets are then used to update the initial frequency coefficients. The updated initial frequency coefficients may include a set of high-frequency coefficients L-HH1+MV1-HH and S-HH1+MV1-HH, and two sets of mid-frequency coefficients, including L-HL1+MV1-HL and S-HL1+MV1-HL, and L-LH1+MV1-LH and S-LH1+MV1-LH, and a set of low-frequency coefficients L-LL1+MV1-LL and S-LL1+MV1-LL.

[0116] It should be noted that the process of updating the above frequency coefficients using the intermediate offset can be referred to the explanation of applying MV2-HH to L-HH2 to obtain L-HH2+MV2-HH, which will not be repeated here.

[0117] Then, a matching algorithm constraint can be applied to each of the above frequency coefficients to obtain the updated initial offset, which can specifically include MV10-LL, MV10-LH, MV10-HL, and MV10-HH. Then, the updated initial offset can be upsampled to obtain the target offset, which can specifically include MV0-LL, MV0-LH, MV0-HL, and MV0-HH.

[0118] After obtaining the target offset, the target offset can be substituted into the reference sub-image and the sub-image to be optimized to update the reference sub-image and the sub-image to be optimized. Then, a matching algorithm constraint is applied to the reference sub-image and the sub-image to be optimized to update the target offset. After this, the target offset can be upsampled to match the target downsampling to complete a second update of the target offset, resulting in the updated target offsets MV-LL, MV-LH, MV-HL, and MVHH.

[0119] In another example implementation, refer to Figure 9 As shown, after obtaining the intermediate offset corresponding to the intermediate frequency coefficients of the second layer, the intermediate frequency coefficients of the second layer can be updated using the intermediate frequency coefficients of the second layer. The updated intermediate frequency coefficients can include a set of high-frequency coefficients L-HH3+MV3-HH and S-HH3+MV3-HH, and two sets of mid-frequency coefficients, including L-HL3+MV3-HL and S-HL3+MV3-HL, and L-LH3+MV3-LH and S-LH3+MV3-LH, and a set of low-frequency coefficients L-LL3+MV3-LL and S-LL3+MV3-LL. Then, wavelet reconstruction is performed on the frequency coefficients corresponding to the updated intermediate image to be optimized in the second layer to obtain the intermediate image of the second layer. Then, the low-frequency coefficients corresponding to the reference image in the first layer, namely S-LL2, are used to perform a matching constraint algorithm to obtain the intermediate offset corresponding to the first layer.

[0120] After obtaining the intermediate offset MV2 corresponding to the intermediate frequency coefficients of the first layer, the frequency coefficients corresponding to the first layer image to be optimized can be updated using the intermediate offset MV2. The updated frequency coefficients corresponding to the first layer image to be optimized include high frequency coefficients L-HH2+MV2, mid frequency coefficients L-HL2+MV2 and L-LH2+MV2, and a set of low frequency coefficients L-LL2+MV2. Then, wavelet reconstruction is performed on the updated frequency coefficients corresponding to the first layer image to be optimized to obtain the intermediate image corresponding to the first layer frequency coefficients. Then, based on the intermediate image and the low frequency coefficients corresponding to the reference image in the initial frequency coefficients, a matching algorithm constraint is performed to obtain the updated initial offset.

[0121] In this example implementation, the frequency coefficients corresponding to the image to be optimized in the initial frequency coefficients can be updated using the updated initial offset. The updated frequency coefficients corresponding to the image to be optimized include the intermediate frequency coefficients L-HL1+MV1 and L-LH1+MV1 and a set of low frequency coefficients L-LL1+MV1. Then, wavelet reconstruction is performed on it to obtain the initial intermediate image. Then, the target offset is obtained by using the matching algorithm between the initial intermediate image and the reference image.

[0122] After obtaining the target offset, the target offset can be directly upsampled to update the target offset.

[0123] After obtaining the target offset mentioned above, step S240 can be executed, specifically:

[0124] In step S240, the target image is obtained by optimizing the image to be optimized using the target offset.

[0125] In one exemplary embodiment of this disclosure, reference is made to Figure 10 As shown, optimizing the image to be optimized using the target offset to obtain the target image may include steps S1010 to S1030.

[0126] In step S1010, the image to be optimized is aligned using the target offset to obtain at least one image to be fused.

[0127] In step S1020, the weights of each of the images to be fused are determined based on the frequency information in the target offset;

[0128] In step S1030, at least one of the images to be fused is fused to obtain the target image based on the weights of each image to be fused.

[0129] In this example implementation, the target offset obtained above can be used to perform alignment operations on the image to be optimized to obtain a fused image. For example, if the target offset obtained above includes MV0, and MV0 includes MV0.x and MV0.y, then the target offset can be applied to the image to be optimized. Specifically, the image to be optimized is copied to obtain a first image to be optimized and a second image to be optimized. The first image to be optimized is traversed with a second preset step size. A fourth image block with a radius of R3 is taken at coordinates (i+MV0.x, j+MV0.y). Then, the second image to be optimized is traversed with a second preset step size. A fifth image block with a radius of R3 is taken at coordinates (i, j). Then, the pixel values ​​in the fifth image block are updated using the mean of the fourth and fifth image blocks. After the iteration is completed, the second image to be optimized is used as the image to be fused.

[0130] It should be noted that the second preset step size and R3 can be customized according to user needs, and are not specifically limited in this example implementation.

[0131] In this example implementation, since only one image with fusion is obtained, the image to be fused can be directly used as the target image.

[0132] In another example implementation, if the target offsets include MV-LL, MV-LH, MV-HL, and MVHH, they can be applied to the above-mentioned optimization items respectively. For example, if the image to be optimized is RAWL and the reference image is RAWS, then MV-LL, MV-LH, MV-HL, and MVHH can be applied to the above-mentioned image to be optimized, that is, the low-frequency image to be fused, RAW_LL, the first intermediate-frequency image to be fused, RAW_LH, the second intermediate-frequency image to be fused, and the high-frequency image to be fused, RAW_HH, can be obtained from RAWL.

[0133] The following explanation uses the application of MV-LL to RAWL to obtain the low-frequency image to be fused, RAW_LL, as an example. Specifically, firstly, the above RAWL can be repeated to obtain the first image to be optimized, RAWL and the second image to be optimized, RAWL. Then, the first image to be optimized is traversed with a second preset step size, and a fourth image block with a radius of R3 is taken at the coordinates (i+MV-LL.x, j+MV-LL.y). Then, the second image to be optimized is traversed with a second preset step size, and a fifth image block with a radius of R3 is taken at the coordinates (i, j). Then, the pixel values ​​in the fifth image block are updated using the mean of the fourth and fifth image blocks. After the iteration is completed, the second image to be optimized is used as the low-frequency image to be fused, RAW_LL.

[0134] The process of applying MV-LH, MV-HL, and MVHH to the above-mentioned images to be optimized to obtain the first intermediate frequency image to be fused, RAW_LH, the second intermediate frequency image to be fused, and the high frequency image to be fused, RAW_HH, can be referred to as applying MV-LL to RAWL to obtain the low frequency image to be fused, RAW_LL, and will not be repeated here.

[0135] The weight information corresponding to the low-frequency image to be fused (RAW_LL), the first intermediate-frequency image to be fused (RAW_LH), the second intermediate-frequency image to be fused (RAW_HL), and the high-frequency image to be fused (RAW_HH) can be determined based on the aforementioned frequency information. The weights can be determined based on prior information. For example, the weights corresponding to the low-frequency image to be fused (RAW_LL), the first intermediate-frequency image to be fused (RAW_LH), the second intermediate-frequency image to be fused (RAW_HL), and the high-frequency image to be fused (RAW_HH) are W1 = 0.5; W2 = 0.2; W3 = 0.2; and W4 = 0.1, respectively. Wherein, W1 + W2 + W3 + W4 = 1.

[0136] It should be noted that the weights corresponding to the fused images mentioned above can also be customized according to user needs, and are not specifically limited in this example implementation.

[0137] In this example implementation, after obtaining the above weight information, the images to be fused are fused according to the above weight information to obtain the target image, so as to complete the denoising of the image to be optimized. Specifically, RAW_DENOISE = W1*RAW_LL + W2*RAW_LH + W3*RAW_HL + W4*RAW_HH, where RAW_DENOISE represents the above target image.

[0138] Refer to the original Figure 11 As shown, after obtaining the above target information, the image processing method of this disclosure may further include steps S1110 to S1130. The following steps are described in detail.

[0139] In step S1110, a first pixel threshold and a second pixel threshold are set;

[0140] In step S1120, the weight of each pixel in the target image is determined based on the pixel value of each pixel in the target image, the first pixel threshold, and the second pixel threshold;

[0141] In step S1130, the target image is fused with the reference image using the weights of each pixel to update the target image.

[0142] In one example embodiment of this disclosure, two pixel thresholds can be set first, specifically including a first pixel threshold and a second pixel threshold. The second pixel threshold can be greater than the first pixel threshold. For example, the first pixel threshold and the second pixel threshold are TH1 and TH2 (TH2>TH1), respectively.

[0143] After determining the aforementioned pixel thresholds, the pixel value of each pixel in the target image can be determined. Then, the weight of the pixel is determined based on the pixel value, the first pixel threshold, and the second pixel threshold. Specifically, if the pixel value is greater than the second pixel threshold, the weight of the pixel is set to 1; if the pixel value is less than the first pixel threshold, the weight of the pixel is set to 0; if the pixel value is greater than or equal to the first pixel threshold and less than or equal to the second pixel threshold, the difference between the first pixel threshold and the second pixel threshold is calculated, and the ratio of the pixel value to the difference is used as the weight of the pixel.

[0144] After obtaining the weight information for each pixel, the reference image and the target image can be fused together based on these weights to obtain the updated target image. Specifically:

[0145] OUTPUT=Mask*RAWS+(1-Mask)*RAWL2S

[0146] Where RAWL2S represents the target image before update, Mask represents the weight matrix corresponding to the reference image, (1-Mask) represents the weight matrix corresponding to the target image, and RAWS represents the reference image.

[0147] By fusing the reference images, the dynamic range of the target image can be widened, and the amount of information in the target image can be increased, thereby improving the accuracy of image processing.

[0148] In summary, this exemplary embodiment employs image processing with different exposure information, resulting in a richer information in the target image. Furthermore, using initial frequency coefficients to determine the target offset between the image to be optimized and the reference image saves computational resources while enabling processing in different frequency domains, thus improving the quality of the target image. The target offset can be obtained by first calculating the initial offset and then updating it using multiple wavelet decompositions, resulting in a more accurate target offset and thus higher processing precision. Moreover, using the frequency coefficients after wavelet decomposition to calculate the target offset reduces the computational burden on the original image. The target offset is then used to denoise the image to be optimized, improving the accuracy of the target image. Further, it is fused with the reference image, further enhancing the dynamic characteristics of the target image and improving the overall image processing precision.

[0149] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0150] Further reference Figure 12 As shown, this example embodiment also provides an image processing apparatus 1200, including a segmentation module 1210, an acquisition module 1220, a determination module 1230, and an optimization module 1240. Wherein:

[0151] The segmentation module 1210 can be used to acquire at least two candidate images with different exposure information for the same subject, and to segment the candidate images into a reference image and at least one image to be optimized.

[0152] In one example implementation, the segmentation module 1210 can be configured to, in response to the number of candidate images being two, use the candidate image with longer exposure as the image to be optimized and the image with shorter exposure as the reference image; and in response to the number of candidate images being more than two, obtain the median of the exposure times of the multiple candidate images and use the candidate image with the smallest difference between the exposure time and the median as the reference image.

[0153] The acquisition module 1220 can be used to acquire multiple sets of initial frequency coefficients between the image to be optimized and the reference image.

[0154] In one example implementation, the acquisition module 1220 may be configured to perform wavelet decomposition on the image to be optimized and the reference image to obtain multiple sets of the initial frequency coefficients.

[0155] The determination module 1230 can be used to determine the target offset between the image to be optimized and the reference image based on the multiple sets of initial frequency coefficients.

[0156] In one example implementation, the determining module 1230 can be configured to perform a matching algorithm constraint on at least one set of initial frequency coefficients to obtain an initial offset; constrain the initial offset; and determine the target offset based on the constraint result. Specifically, the target offset can be obtained by updating the initial offset using low-frequency coefficients from multiple sets of initial frequency coefficients.

[0157] In one example implementation, the target offset is obtained by updating the initial offset using low-frequency coefficients from multiple sets of initial frequency coefficients, including: performing wavelet decomposition on the low-frequency coefficients to obtain first-layer intermediate frequency coefficients; performing wavelet decomposition on the intermediate low-frequency coefficients from the (N-1)th layer intermediate frequency coefficients to obtain Nth-layer intermediate frequency coefficients; updating the intermediate offset corresponding to the (N-1)th layer intermediate frequency coefficients based on the Nth-layer intermediate frequency coefficients; and updating the initial offset using the intermediate offset corresponding to the first-layer intermediate frequency coefficients to obtain the target offset; wherein the (N-1)th layer intermediate frequency coefficients and the Nth layer intermediate frequency coefficients are any two adjacent layers of intermediate frequency coefficients; and N is a positive integer greater than or equal to 2.

[0158] In this example embodiment, the determining module 1230 can be configured to update the intermediate frequency coefficients M times. Updating the intermediate offset corresponding to the intermediate frequency coefficients of the (N-1)th layer based on the intermediate frequency coefficients of the Nth layer can include responding to the fact that N is less than M, determining the intermediate offset of the (N-1)th layer according to the intermediate offset of the Nth layer and the intermediate frequency coefficients of the (N-1)th layer; responding to the fact that N is equal to M, performing a matching algorithm constraint on at least one set of intermediate frequency coefficients of the Nth layer to obtain the intermediate offset of the Nth layer, and upsampling the intermediate offset of the Nth layer to obtain the intermediate offset of the (N-1)th layer.

[0159] In this example implementation, determining the intermediate offset of the (N-1)th layer based on the intermediate offset of the Nth layer and the intermediate frequency coefficients of the (N-1)th layer includes: performing wavelet reconstruction on the intermediate offset of the Nth layer and the intermediate frequency coefficients of the Nth layer image to be optimized to obtain an intermediate image corresponding to the Nth layer image to be optimized; and determining the intermediate offset of the (N-1)th layer based on the low-frequency coefficients corresponding to the reference image in the intermediate frequency coefficients of the Nth layer intermediate image and the intermediate frequency coefficients of the (N-1)th layer intermediate image.

[0160] In one example implementation, determining the intermediate offset of the (N-1)th layer by matching the intermediate image of the Nth layer with the intermediate frequency coefficients of the (N-1)th layer may include performing wavelet reconstruction on the intermediate frequency coefficients corresponding to the reference image of the (N-1)th layer to obtain the intermediate image corresponding to the reference image of the (N-1)th layer; and determining the intermediate offset of the (N-1)th layer by matching the intermediate image corresponding to the image to be optimized of the Nth layer with the low-frequency coefficients corresponding to the reference image in the intermediate frequency coefficients of the (N-1)th layer.

[0161] In one example implementation, the determining module 1230 can be configured to update the initial frequency coefficients using the intermediate offsets corresponding to the first-layer intermediate frequency coefficients; calculate the updated initial offsets using the updated initial frequency coefficients; and perform an upsampling operation on the updated initial offsets to obtain the target offsets.

[0162] In this example implementation, updating the initial offset using the intermediate offset corresponding to the intermediate frequency coefficients of the first layer to obtain the target offset may include updating the frequency coefficients corresponding to the first layer image to be optimized using the intermediate offset corresponding to the intermediate frequency coefficients of the first layer; performing wavelet reconstruction on the frequency coefficients corresponding to the first layer image to be optimized to obtain a first layer intermediate image; determining the updated initial offset using the first layer intermediate image and the low-frequency coefficients corresponding to the reference image in the initial frequency coefficients; performing wavelet reconstruction on the initial frequency coefficients corresponding to the image to be optimized using the updated initial offset to obtain an initial intermediate image; and constraining the initial intermediate image and the reference image using a matching algorithm to obtain the target offset.

[0163] In one example implementation, performing wavelet decomposition on the image to be optimized and the reference image to obtain multiple sets of initial frequency coefficients may include performing target downsampling on the reference image and the image to be optimized to obtain a sub-image to be optimized and a reference sub-image; and performing wavelet decomposition on the reference sub-image and the sub-image to be optimized to obtain the initial frequency coefficients.

[0164] The optimization module 1240 can be used to optimize the image to be optimized using the target offset to obtain the target image.

[0165] In one example implementation, the optimization module 1240 can be configured to optimize the image to be optimized using the target offset to obtain a target image, including: performing an alignment operation on the image to be optimized using the target offset to obtain at least one image to be fused; determining the weight of each of the images to be fused based on the frequency information in the target offset; and fusing the at least one image to be fused based on the weight of each image to be fused to obtain the target image.

[0166] In one example implementation, the image processing apparatus 1200 described above can also be used to set a first pixel threshold and a second pixel threshold; determine the weight of each pixel in the target image based on the pixel value of each pixel in the target image, the first pixel threshold, and the second pixel threshold; and use the weight of each pixel to fuse the target image with the reference image to update the target image.

[0167] The specific details of each module in the above-mentioned device have been described in detail in the method section of the implementation. For any undisclosed details, please refer to the implementation content of the method section, and therefore will not be repeated here.

[0168] Exemplary embodiments of this disclosure also provide an electronic device for performing the above-described image processing method. This electronic device may be the terminal 110 or the server 120 described above. Generally, the electronic device may include a processor and a memory, the memory for storing executable instructions of the processor, and the processor configured to perform the above-described image processing method by executing the executable instructions.

[0169] The following is based on Figure 13 Taking the mobile terminal 1300 as an example, the construction of this electronic device will be described by way of example. Those skilled in the art will understand that, apart from components specifically designed for mobile purposes, Figure 13 The structure can also be applied to fixed types of equipment.

[0170] like Figure 13 As shown, the mobile terminal 1300 may specifically include: a processor 1301, a memory 1302, a bus 1303, a mobile communication module 1304, an antenna 1, a wireless communication module 1305, an antenna 13, a display screen 1306, a camera module 1307, an audio module 1308, a power module 1309, and a sensor module 13210.

[0171] Processor 1301 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, an encoder, a decoder, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). The image processing method in this exemplary embodiment can be executed by an AP, GPU, or DSP. When the method involves neural network-related processing, it can be executed by an NPU.

[0172] An encoder can encode (i.e., compress) images or videos. For example, it can encode a target image into a specific format to reduce data size for easier storage or transmission. A decoder can decode (i.e., decompress) the encoded data of an image or video to restore the image or video data. For example, it can read the encoded data of a target image, decode it, and restore the target image data, then perform image processing on the data. The mobile terminal 1300 can support one or more encoders and decoders. Thus, the mobile terminal 1300 can process images or videos in various encoding formats, such as JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), BMP (Bitmap), and MPEG (Moving Picture Experts Group) 1, MPEG2, H.263, H.264, HEVC (High Efficiency Video Coding).

[0173] The processor 1301 can be connected to the memory 1302 or other components via the bus 1303.

[0174] The memory 1302 can be used to store computer executable program code, which includes instructions. The processor 1301 executes various functional applications and data processing of the mobile terminal 1300 by running the instructions stored in the memory 1302. The memory 1302 can also store application data, such as images, videos, and other files.

[0175] The communication function of mobile terminal 1300 can be implemented through mobile communication module 1304, antenna 1, wireless communication module 1305, antenna 2, modem processor, and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Mobile communication module 1304 can provide 2G, 3G, 4G, and 5G mobile communication solutions for mobile terminal 1300. Wireless communication module 1305 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication for mobile terminal 200.

[0176] The display screen 1306 is used to implement display functions, such as displaying the user interface, images, and videos. The camera module 1307 is used to implement shooting functions, such as capturing images and videos. The audio module 1308 is used to implement audio functions, such as playing audio and capturing voice. The power module 1309 is used to implement power management functions, such as charging the battery, supplying power to the device, and monitoring battery status. The sensor module 1310 may include a depth sensor 13101, a pressure sensor 13102, a gyroscope sensor 13103, a barometric pressure sensor 13104, etc., to implement corresponding sensing and detection functions.

[0177] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0178] Exemplary embodiments of this disclosure also provide a computer-readable storage medium having a program product stored thereon capable of implementing the methods described above in this specification. In some possible embodiments, various aspects of this disclosure may also be implemented as a program product including program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0179] It should be noted that the computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0180] In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0181] Furthermore, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0182] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0183] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized by, The method comprises the following steps: obtaining at least two candidate images with different exposure information of the same shooting object, and dividing the candidate images into a reference image and at least one image to be optimized; obtaining a plurality of sets of initial frequency coefficients between the image to be optimized and the reference image; determining a target offset between the image to be optimized and the reference image based on the plurality of sets of initial frequency coefficients; optimizing the image to be optimized by using the target offset to obtain a target image; dividing the candidate images into a reference image and at least one image to be optimized, comprising: in response to the number of candidate images being two, taking the candidate image with longer exposure as the image to be optimized, and taking the candidate image with shorter exposure as the reference image; in response to the number of candidate images being greater than two, obtaining the median of the exposure time of a plurality of candidate images, taking the candidate image with the smallest difference between the exposure time and the median as the reference image, and taking the other candidate images as the image to be optimized; wherein, based on the plurality of sets of initial frequency coefficients, the target offset between the image to be optimized and the reference image is determined, comprising: performing matching algorithm constraint on at least one set of initial frequency coefficients to obtain an initial offset; and updating the initial offset by using the low-frequency coefficients in the plurality of sets of initial frequency coefficients to obtain the target offset; wherein, the target offset is obtained by updating the initial offset by using the low-frequency coefficients in the plurality of sets of initial frequency coefficients, comprising: wavelet decomposing the low-frequency coefficients to obtain first layer intermediate frequency coefficients; wavelet decomposing the intermediate low-frequency coefficients in the N-1 layer intermediate frequency coefficients to obtain N layer intermediate frequency coefficients; updating the intermediate offset corresponding to the N-1 layer intermediate frequency coefficients based on the N layer intermediate frequency coefficients; updating the initial offset by using the intermediate offset corresponding to the first layer intermediate frequency coefficients to obtain the target offset; wherein, the N-1 layer intermediate frequency coefficients and the N layer intermediate frequency coefficients are any two adjacent layers of intermediate frequency coefficients; N is a positive integer greater than or equal to 2.

2. The method of claim 1, wherein, obtaining a plurality of sets of initial frequency coefficients between the image to be optimized and the reference image, comprising: wavelet decomposing the image to be optimized and the reference image to obtain a plurality of sets of initial frequency coefficients.

3. The method of claim 1, wherein, updating the intermediate frequency coefficients M times, and updating the intermediate offset corresponding to the N-1 layer intermediate frequency coefficients based on the N layer intermediate frequency coefficients, comprising: in response to N being less than M, determining the N-1 layer intermediate offset according to the N layer intermediate offset and the N-1 layer intermediate frequency coefficients; in response to N being equal to M, performing matching algorithm constraint on at least one set of N layer intermediate frequency coefficients to obtain N layer intermediate offset, and upsampling the N layer intermediate offset to obtain N-1 layer intermediate offset.

4. The method of claim 3, wherein, determining the N-1 layer intermediate offset according to the N layer intermediate offset and the N-1 layer intermediate frequency coefficients, comprising: upsampling the N layer intermediate offset to obtain N-1 layer candidate offset; The N-1 layer intermediate frequency coefficient is updated by using the N-1 layer candidate offset, and the N-1 layer intermediate offset is determined by performing a matching algorithm constraint on the updated N-1 layer intermediate frequency coefficient.

5. The method of claim 3, wherein, The N-1 layer intermediate offset is determined according to the N layer intermediate offset and the N-1 layer intermediate frequency coefficient, including: The N layer intermediate image corresponding to the N layer to-be-optimized image is obtained by wavelet reconstruction on the N layer intermediate offset and the intermediate frequency coefficient of the N layer to-be-optimized image. The N-1 layer intermediate offset is determined according to the low frequency coefficient of the N layer intermediate image and the reference image in the N-1 layer intermediate frequency coefficient.

6. The method of claim 5, wherein, The N-1 layer intermediate offset is determined by performing a matching algorithm constraint on the N layer intermediate image and the N-1 layer intermediate frequency coefficient, including: The N-1 layer reference image corresponding to the intermediate frequency coefficient of the reference image of the N-1 layer is obtained by wavelet reconstruction. The N-1 layer intermediate offset is determined by performing a matching algorithm constraint on the low frequency coefficient of the N layer intermediate image corresponding to the N layer to-be-optimized image and the reference image in the N-1 layer intermediate frequency coefficient.

7. The method of claim 1, wherein, The N-1 layer intermediate offset is determined by performing a matching algorithm constraint on the N layer intermediate image and the N-1 layer intermediate frequency coefficient, including: The N-1 layer reference image corresponding to the intermediate frequency coefficient of the reference image of the N-1 layer is obtained by wavelet reconstruction. The N-1 layer intermediate offset is determined by performing a matching algorithm constraint on the low frequency coefficient of the N layer intermediate image corresponding to the N layer to-be-optimized image and the reference image in the N-1 layer intermediate frequency coefficient. The target offset is obtained by updating the initial offset by using the intermediate offset corresponding to the first layer intermediate frequency coefficient, including:

8. The method of claim 1, wherein, The initial frequency coefficient is updated by using the intermediate offset corresponding to the first layer intermediate frequency coefficient. The updated initial offset is calculated by using the updated initial frequency coefficient. The target offset is obtained by upsampling the updated initial offset. The target offset is obtained by updating the initial offset by using the intermediate offset corresponding to the first layer intermediate frequency coefficient, including: The frequency coefficient corresponding to the first layer to-be-optimized image is updated by using the intermediate offset corresponding to the first layer intermediate frequency coefficient. The first layer intermediate image is obtained by wavelet reconstruction on the frequency coefficient corresponding to the first layer to-be-optimized image.

9. The method of claim 2, wherein, The updated initial offset is determined by using the low frequency coefficient of the reference image in the initial frequency coefficient and the first layer intermediate image. The initial intermediate image is obtained by wavelet reconstruction on the initial frequency coefficient of the to-be-optimized image by using the updated initial offset. The target offset is obtained by performing a matching algorithm constraint on the initial intermediate image and the reference image.

10. The method of claim 9, wherein, The initial frequency coefficient is obtained by wavelet decomposition on the to-be-optimized image and the reference image, including: The target sub-image and the reference sub-image are obtained by target downsampling on the reference image and the to-be-optimized image. The initial frequency coefficient is obtained by wavelet decomposition on the reference sub-image and the to-be-optimized sub-image.

11. The method of claim 1, wherein, The method further includes: The target offset is substituted into the to-be-optimized sub-image and the reference sub-image, and a matching algorithm constraint is performed on the to-be-optimized sub-image and the reference sub-image to update the target offset; and / or The target offset is subjected to upsampling processing adapted to the target downsampling processing to update the target offset. The target image is obtained by optimizing the to-be-optimized image by using the target offset, including: aligning the to-be-optimized image according to the target offset to obtain at least one to-be-fused image; determining a weight of each to-be-fused image according to frequency information in the target offset; fusing at least one to-be-fused image based on the weight of each to-be-fused image to obtain the target image.

12. The method according to any one of claims 1 to 11, characterized in that, The method further comprises: setting a first pixel threshold and a second pixel threshold; determining a weight of each pixel point in the target image based on a pixel value of each pixel point in the target image, the first pixel threshold and the second pixel threshold; fusing the target image and the reference image based on the weight of each pixel point to update the target image.

13. An image processing apparatus characterized by comprising: comprises: a division module configured to obtain at least two candidate images having different exposure information collected for a same shooting object, and divide the candidate images into a reference image and at least one to-be-optimized image; an acquisition module configured to obtain a plurality of groups of initial frequency coefficients between the to-be-optimized image and the reference image; a determination module configured to determine a target offset of the to-be-optimized image and the reference image based on the plurality of groups of initial frequency coefficients; an optimization module configured to perform optimization processing on the to-be-optimized image according to the target offset to obtain a target image; the division module is configured to, in response to the number of candidate images being two, take a candidate image with longer exposure as the to-be-optimized image, and take a candidate image with shorter exposure as the reference image; in response to the number of candidate images being greater than two, acquire a median of exposure times of the plurality of candidate images, take a candidate image with a minimum difference between the exposure time and the median as the reference image, and take other candidate images as the to-be-optimized images; wherein the determination of the target offset of the to-be-optimized image and the reference image based on the plurality of groups of initial frequency coefficients comprises: performing matching algorithm constraint on at least one group of initial frequency coefficients to obtain an initial offset; and updating the initial offset according to low-frequency coefficients in the plurality of groups of initial frequency coefficients to obtain the target offset; wherein the updating of the initial offset according to the low-frequency coefficients in the plurality of groups of initial frequency coefficients to obtain the target offset comprises: performing wavelet decomposition on the low-frequency coefficients to obtain first-layer intermediate frequency coefficients; performing wavelet decomposition on intermediate low-frequency coefficients in N-1-layer intermediate frequency coefficients to obtain N-layer intermediate frequency coefficients; updating an intermediate offset corresponding to the N-1-layer intermediate frequency coefficients based on the N-layer intermediate frequency coefficients; and updating the initial offset according to an intermediate offset corresponding to the first-layer intermediate frequency coefficients to obtain the target offset; wherein the N-1-layer intermediate frequency coefficients and the N-layer intermediate frequency coefficients are any two adjacent layers of intermediate frequency coefficients; and N is a positive integer greater than or equal to 2.

14. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by a processor to implement the image processing method in any one of claims 1 to 12.

15. An electronic device, comprising: comprises: one or more processors; and a memory for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the image processing method of any one of claims 1 to 12.

Images