Image processing method, image processing device, electronic device and readable storage medium
By using different exposure parameters in electronic devices to acquire image sequences, perform filtering and fusion processing, and using the target weight function to achieve high dynamic range imaging, the problem of expensive imaging devices is solved, simplifying the processing process and enhancing product competitiveness.
Patent Information
- Application Number
- CN202110565156.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-05-24
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-05-24
AI Technical Summary
Existing high dynamic range imaging devices are expensive and difficult to become the mainstream method to realize dynamic range imaging.
Image sequences are acquired through different exposure parameters, filtering and fusion processing are performed, and images are pre-processed and fused using the target weight function to achieve high dynamic range imaging without relying on expensive imaging devices.
High dynamic range imaging is realized, the image processing process is simplified, the equipment costs are reduced, and the product competitiveness is enhanced.
Smart Images

Figure CN113284063B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of communication equipment, and specifically relates to an image processing method, an image processing device, an electronic device and a readable storage medium. Background Art
[0002] As users' requirements for the use of smart products increase, in order to improve the competitiveness of products, it is necessary to improve the imaging technology of digital cameras in electronic devices.
[0003] At present, high dynamic range imaging can be achieved through high-bit output imaging devices. Although the imaging effect is good, the imaging devices that can achieve high dynamic range imaging are expensive and it is difficult to become the mainstream method for achieving dynamic range imaging. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide an image processing method, an image processing device, an electronic device and a readable storage medium, which can solve the problem that the current imaging devices for high dynamic range imaging are expensive and difficult to become the mainstream method for realizing dynamic range imaging.
[0005] In order to solve the above technical problems, this application is implemented as follows:
[0006] In a first aspect, an embodiment of the present application provides an image processing method, comprising:
[0007] Acquire a first image sequence with different exposure parameters, and perform filtering processing on the first image sequence to obtain a second image sequence, wherein the second image sequence is in a mapping relationship with the first image sequence, and the brightness difference value of a pixel point of each image in the second image sequence is greater than or equal to the brightness difference value of a pixel point of a corresponding image in the first image sequence;
[0008] A target weight function is determined, and a fusion process is performed on the second image sequence according to the target weight function to obtain a target image.
[0009] In a second aspect, an embodiment of the present application provides an image processing device, the device comprising:
[0010] A filtering module, wherein the filtering module obtains a first image sequence with different exposure parameters, and performs filtering processing on the first image sequence to obtain a second image sequence, wherein the second image sequence is in a mapping relationship with the first image sequence, and a brightness difference value of a pixel point of each image in the second image sequence is greater than or equal to a brightness difference value of a pixel point of a corresponding image in the first image sequence;
[0011] The fusion module determines a target weight function and performs fusion processing on the second image sequence according to the target weight function to obtain a target image.
[0012] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the image processing method in the first aspect.
[0013] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the image processing method of the first aspect are implemented.
[0014] In the embodiment of the present application, it is defined that the first image sequence is obtained by different exposure parameters, that is, when the user uses the electronic device to shoot, the electronic device will obtain multiple images by different exposure parameters, and the multiple images are the first image sequence. Filtering the first image sequence is the preprocessing of the image, and the multiple images after processing are the second image sequence. The second image sequence and the first image sequence are in a mapping relationship, wherein the preprocessed image is reflected in that the brightness difference of the pixel points of each image in the second image sequence is greater than or equal to the brightness difference of the pixel points of the corresponding image in the first image sequence, so the filtering process is to pre-adjust the brightness of the image, increase the difference of at least some pixels in each image in the first image sequence, for example, the brightness difference of some pixels in the first image sequence is small, and the brightness difference of the part of the pixels can be increased. If the difference between the two ends of some pixels in the first image sequence is large, it is not necessary to adjust the brightness difference of the part of the pixels. After processing the images in the first image sequence, the non-edge details of each image in the second image sequence will become blurred, while the edges of the images are retained, which can provide convenience for subsequent image fusion operations.
[0015] After the image is preprocessed, the second image sequence is fused according to the target weight, so that multiple images are fused into one target image. By fusing multiple images through the target weight, the obtained target image can eliminate part of the halo and overcome the defect of local detail loss, thereby avoiding the imbalance of the overall brightness ratio of the image, so as to provide an image with a high dynamic range. Since the image is filtered in the above process, an intermediate image can be provided for the subsequent fusion step, so that multiple images can be fused in the spatial domain of pixels without the need for complex operations of multi-resolution decomposition or two-layer decomposition, thereby simplifying the image processing time. In the embodiments of the present application, the electronic device can achieve high dynamic range imaging by preprocessing and fusion of the image, which is not simply dependent on the imaging device. The electronic device does not need to be equipped with expensive imaging devices to achieve high dynamic range imaging, which is conducive to improving the competitiveness of the product. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1One of the flowcharts of the image processing method according to one embodiment of the present application is shown;
[0017] Figure 2 A second flowchart of an image processing method according to an embodiment of the present application is shown;
[0018] Figure 3 A third flowchart of an image processing method according to an embodiment of the present application is shown;
[0019] Figure 4 A fourth flowchart of an image processing method according to an embodiment of the present application is shown;
[0020] Figure 5 A fifth flowchart of an image processing method according to an embodiment of the present application is shown;
[0021] Figure 6 A sixth flowchart of an image processing method according to an embodiment of the present application is shown;
[0022] Figure 7 A structural block diagram of an image processing device according to an embodiment of the present application is shown;
[0023] Figure 8 One of the structural block diagrams of an electronic device according to an embodiment of the present application is shown;
[0024] Fig. 9 A second structural block diagram of an electronic device according to an embodiment of the present application is shown;
[0025] Fig.10 A schematic diagram showing a pixel brightness difference value according to an embodiment of the present application;
[0026] Fig.11 A schematic diagram showing the functional relationship between pixel brightness difference and kernel function value in one embodiment of the present application is shown. DETAILED DESCRIPTION
[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0028] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here. In addition, the "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally represents that the objects associated with each other are in an "or" relationship.
[0029] The image processing method, image processing device, electronic device, readable storage medium and chip provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.
[0030] In some embodiments of the present application, Figure 1 A flowchart of an image processing method according to an embodiment of the present application is shown. The image processing method includes:
[0031] Step S102, obtaining a first image sequence with different exposure parameters, filtering the first image sequence to obtain a second image sequence, the second image sequence is in a mapping relationship with the first image sequence, and the brightness difference of the pixel points of each image in the second image sequence is greater than or equal to the brightness difference of the pixel points of the corresponding image in the first image sequence.
[0032] In the embodiment of the present application, it is defined that the first image sequence is obtained by different exposure parameters, that is, when the user uses the electronic device to shoot, the electronic device will obtain multiple images by different exposure parameters, and the multiple images are the first image sequence. Filtering the first image sequence is the preprocessing of the image, and the multiple images after processing are the second image sequence. The second image sequence and the first image sequence are in a mapping relationship, wherein the preprocessed image is reflected in that the brightness difference of the pixel points of each image in the second image sequence is greater than or equal to the brightness difference of the pixel points of the corresponding image in the first image sequence, so the filtering process is to pre-adjust the brightness of the image, increase the difference of at least some pixels in each image in the first image sequence, for example, the brightness difference of some pixels in the first image sequence is small, and the brightness difference of the part of the pixels can be increased. If the difference between the two ends of some pixels in the first image sequence is large, it is not necessary to adjust the brightness difference of the part of the pixels. After processing the images in the first image sequence, the non-edge details of each image in the second image sequence will become blurred, while the edges of the images are retained, which can provide convenience for subsequent image fusion operations.
[0033] Step S104, determining a target weight function, and performing fusion processing on the second image sequence according to the target weight function to obtain a target image.
[0034] After the image is preprocessed, the second image sequence is fused according to the target weight, so that multiple images are fused into one target image. By fusing multiple images through the target weight, the obtained target image can eliminate part of the halo and overcome the defect of local detail loss, thereby avoiding the imbalance of the overall brightness ratio of the image, so as to provide an image with a high dynamic range. Since the image is filtered in the above process, an intermediate image can be provided for the subsequent fusion step, so that multiple images can be fused in the spatial domain of pixels without the need for complex operations of multi-resolution decomposition or two-layer decomposition, thereby simplifying the image processing time. In the embodiments of the present application, the electronic device can achieve high dynamic range imaging by preprocessing and fusion of the image, which is not simply dependent on the imaging device. The electronic device does not need to be equipped with expensive imaging devices to achieve high dynamic range imaging, which is conducive to improving the competitiveness of the product.
[0035] like Figure 2 As shown, in a possible embodiment, filtering the first image sequence to obtain the second image sequence specifically includes:
[0036] Step S202, determining a brightness difference sequence of pixels in each image of the first image sequence;
[0037] Step S204, selecting a target brightness difference value in the brightness difference value sequence according to a preset brightness difference threshold;
[0038] Step S206, determining a brightness kernel function value according to the target brightness difference;
[0039] Step S208: filtering each image in the first image sequence according to the brightness kernel function value to obtain a second image sequence.
[0040] In this embodiment, the first image sequence includes multiple subsequences, such as a brightness difference sequence, a color sequence, etc. In this embodiment, the brightness difference sequence in the first image sequence is acquired and preprocessed.
[0041] According to the preset brightness difference threshold, a target brightness difference in the brightness difference sequence is selected. Specifically, the brightness difference between two pixels in the brightness difference sequence is obtained, and the brightness difference is compared with the brightness difference threshold. The brightness difference that meets the set conditions is the target brightness difference.
[0042] The brightness kernel function value is calculated based on the target brightness difference. Exemplarily, the brightness difference threshold in this embodiment includes a first threshold and a second threshold, and the first threshold is greater than the second threshold. When the brightness difference between two pixels is less than the second threshold, it is considered that the two pixels that obtain the brightness difference belong to the same category, and their two kernel function values are 1. When the brightness difference between two pixels is greater than the first threshold, it is considered that the two pixels that obtain the brightness difference are not in the same category, and their kernel function value is 0. When the two differences between two pixels are between the first threshold and the second threshold, it is considered that the two pixels that obtain the two differences approximately belong to the same category, and their kernel function value is a value between 0 and 1, such as Fig.11 As shown, in this embodiment, the kernel function between 0 and 1 is in a linear function relationship with the brightness difference threshold, and the first threshold and the second threshold can be values set according to usage requirements. According to the brightness difference threshold, a target brightness difference is selected, for example, a brightness difference less than the second threshold is selected as the target two differences, and then the brightness kernel function value is calculated based on the two differences, and each image of the first image sequence is filtered according to the brightness kernel function value. Through the above operation, the non-edge details of each image in the first image sequence become blurred, and the edges of the original image are retained, so that in the subsequent image fusion process, the fusion can be directly based on the hole domain of the pixel, and the framework with high time complexity such as multi-resolution decomposition is not used, which saves fusion time, reduces fusion difficulty, and improves fusion effect.
[0043] like Figure 3 As shown, in a possible embodiment, determining a brightness difference sequence of pixels in each image of the first image sequence specifically includes:
[0044] Step S302, calculating the brightness difference between each pixel in the image and the pixels in its neighborhood to obtain a brightness difference sequence.
[0045] In this embodiment, the brightness difference of each pixel in its area is calculated. Specifically, a pixel is taken as the central pixel, and the pixels adjacent to the central pixel belong to the neighborhood pixel of the central pixel. For example, Fig.10 As shown, the central pixel point is P, and its neighborhood Ω has 8 pixels, that is, a 3×3 neighborhood. Only the 3×3 neighborhood is taken as an example to illustrate the kernel function value calculation method of this embodiment. First, the central pixel point P is taken as a reference, P represents the spatial coordinates of the pixel point, and the brightness distance between it and the neighboring pixel points is calculated. Taking the neighborhood point q as an example, the calculation formula is:
[0046] d k,p =|L k,p -L k,q |,q∈Ω;
[0047] Among them, d k,pis the brightness distance between the pixel p in the kth brightness difference sequence and the pixels in its neighborhood Ω, L k,p is the brightness of pixel P, L k,q is the brightness of pixel q, at brightness distance d k,p On the basis of Fig.11 The mapping relationship can calculate the kernel function ω of pixel point p k,p , the specific description is as follows: when the brightness distance d k,p Greater than the threshold σ H When the kernel function ω k,p The value is 0; when the brightness distance d k,p Less than the threshold σ L When the kernel function ω k,p The value is 1; when the brightness distance d k,p Between the threshold σ H and σ L When the kernel function ω k,p The value of is the brightness distance d k,p A linear function of H and σ L Keep it continuous.
[0048] Calculate the brightness difference of each pixel in its area to obtain a brightness difference sequence. According to the preset brightness difference threshold, select the target brightness difference in the brightness difference sequence, and determine the brightness kernel function value according to the target brightness difference. Specifically, when the brightness distance between the neighborhood pixel and the central pixel is small, the threshold σ is used. L As the boundary, the neighborhood pixel and the central pixel are considered to belong to the same category, and its function value is set to 1; when the brightness distance between the neighborhood pixel and the central pixel is large, the threshold σ H As the boundary, it is considered that the neighborhood pixel and the central pixel do not belong to the same category, and its function value is set to 0; when the brightness distance between the neighborhood pixel and the central pixel is moderate, the threshold σ L and σ H As the boundary, it is considered that the neighborhood pixels and the central pixel can approximately belong to the same category, and its function value is set to a value between 0 and 1.
[0049] When calculating the brightness difference between each central point and the adjacent pixel points, repeated calculations may occur. This can be further accelerated to save the steps of repeated calculations.
[0050] When the value of the kernel function is 0, it indicates that the pixel does not participate in the filtering operation of the central pixel p; when the value of the kernel function is 1, it indicates that the pixel fully participates in the filtering operation of the central pixel p; when the value of the kernel function is between 0 and 1, it indicates that the pixel p partially participates in the filtering operation of the central pixel.
[0051] Each image in the first image sequence is filtered according to the brightness kernel function value. Each pixel point p in the multi-exposure image sequence has a kernel function ω that matches its neighborhood content. k,p , and perform the same filtering operation on the RGB (full color image) three channels, which refer to: red channel, green channel and blue channel.
[0052] The calculation is performed through the following disclosure:
[0053]
[0054] Among them, I k,c,p Refers to the pixel value of pixel p on channel c (c can be one of the three channels mentioned above) of the k-th multi-exposure image, represents the summation operation in the neighborhood of pixel p, F k,c,p is the filtering result, i.e., the pixel value of pixel point p on channel c of the k-th filtered image. Through the filtering operation, the filtered second image sequence is obtained for use in subsequent steps. The purpose of this step is to provide an intermediate image for the calculation of the weight function, so that the calculated weight map can be directly used for image fusion in the spatial domain, thereby reducing the time complexity of the calculation process, and at the same time effectively suppressing existing defects such as halo.
[0055] like Figure 4 As shown, in a possible embodiment, determining the target weight function specifically includes:
[0056] Step S402, determining a global weight value and a local weight value of each image in the first image sequence;
[0057] Step S404, calculating the target weight function according to the global weight value and the local weight value.
[0058] In this embodiment, it is proposed that a target weight function is calculated by a global weight value and a local weight value. When multiple images are fused by a global weight value, attention is paid to the overall brightness distribution of the image, and there is a possibility of loss of local details. When multiple images are fused by a local weight value, attention is paid to the presentation of details, but there is a possibility of contrast abnormalities. In this embodiment, the above problems are comprehensively considered, and a target weight function is obtained by combining the global weight value and the local weight value. This function not only takes into account the overall brightness distribution of the image, but also pays attention to the local details of the image, thereby avoiding imbalance in the overall brightness distribution of the image and facilitating the realization of high dynamic range imaging.
[0059] The global weight focuses on the overall brightness distribution of the fused image, and there is a possibility of local detail loss. The local weight focuses on the local detail presentation of the fused image, and there is a possibility of contrast abnormality. Therefore, the combined global weight {Wg k} and local weight {Wlk}, and get the final weight function:
[0060] W k =(Wg k ) α ×Wl k ;
[0061] Where α represents the combined ratio of global weight and local weight. The ratio of global weight and local weight can be appropriately adjusted according to the actual needs of the application.
[0062] In order to solve the defects of existing exposure fusion algorithms such as edge halo, local detail loss, high time complexity, and imbalanced brightness distribution, this embodiment starts from the method of direct fusion in the pixel domain, does not adopt high time-complexity frameworks such as multi-resolution decomposition, but pre-processes the image through an edge-preserving filtering method, and then calculates the weight function to eliminate possible defects such as halo and local detail loss, and combines global weights and local weights to avoid imbalanced overall brightness distribution of the image.
[0063] The weight combination calculation method proposed in this embodiment can realize scene-adaptive combination parameter selection based on the content of the target scene by using ideas such as machine learning to achieve a more intelligent visual effect.
[0064] like Figure 5 As shown, in a possible embodiment, determining the global weight value of each image in the first image sequence specifically includes:
[0065] Step S502: Calculate the global weight value of each image material according to the brightness parameter in the first image sequence.
[0066] In this embodiment, a method for calculating the global weight value is defined. The global weight calculation is to assign a global weight to each image in the multi-exposure image sequence, that is, the target image is regarded as a whole, and its weight value is a constant, which can be described as:
[0067] W k =f g (I k );
[0068] Typically, the global weight function can be used in the following form:
[0069]
[0070] Where μ(·) represents the normalized brightness mean of the image, is a control parameter. The physical meaning of this global weight function is that when the mean brightness of an image is close to 0.5, its global weight is larger, and vice versa. In this way, when the overall brightness of the image is closer to 0.5, it means that it can provide more useful information, so its global weight is set to a larger value.
[0071] Global weight calculation is a calculation method in which the global weight function f g (·) Not limited to the example, when the input multi-exposure image sequence {I k}, output {Wg k When} is a constant vector, it can be used as a global weight function. For example, when combined with histogram regularization, the fused image is made to meet certain histogram conditions, and the corresponding global weight function f can be obtained. g The specific form of (·).
[0072] like Figure 6 As shown, in a possible embodiment, determining the local weight value of each image in the first image sequence specifically includes:
[0073] Step S602, calculating a weight value of each pixel point of an image in the second image sequence according to the pixel value in the second image sequence;
[0074] Step S604: Calculate according to the weight value of each pixel to obtain a local weight value of each image in the first image sequence.
[0075] In this embodiment, a method of calculating a local weight value is defined, and the local weight value is calculated to filter the image sequence {F k} is the basis for calculation, for each multi-exposure image {I k}Give a local weight that reflects the local details. The weight value is a two-dimensional function, which can be described as:
[0076] Wl k =f l (F k );
[0077] Typically, the local weight function can be of the following form:
[0078]
[0079] in, Indicates that the local weights are calculated for the three channels of the filtered image and their product is calculated. k,c represents the pixel value of channel c of the kth filtered image, and ρ is the control parameter.
[0080] The local weight calculation is a calculation method in which the local weight function f g(·) Not limited to the example, the two-dimensional function calculated pixel by pixel can be used as the local weight function {Wl k The local weight calculation method of this embodiment is that the weight function is calculated in the filtered image sequence {F k}, but not on the multi-exposure image sequence {I k}, which has the characteristic of eliminating artifacts such as halos while retaining local details.
[0081] It should be noted that the image processing method provided in the embodiment of the present application can be executed by an image processing device or a control module in the image processing device for executing the method for loading image processing. In the embodiment of the present application, the image processing device provided in the embodiment of the present application is described by taking the image processing device executing the method for loading image processing as an example.
[0082] like Figure 7 As shown, in some embodiments of the present application, the image processing device 900 includes:
[0083] A filtering module 910, wherein the filtering module 910 obtains a first image sequence with different exposure parameters, and performs filtering processing on the first image sequence to obtain a second image sequence, wherein the second image sequence is in a mapping relationship with the first image sequence, and a brightness difference value of a pixel point of each image in the second image sequence is greater than or equal to a brightness difference value of a pixel point of an image in the corresponding first image sequence;
[0084] The fusion module 920 determines a target weight function, and performs fusion processing on the second image sequence according to the target weight function to obtain a target image.
[0085] In the embodiment of the present application, it is defined that the first image sequence is obtained by different exposure parameters, that is, when the user uses the electronic device to shoot, the electronic device will obtain multiple images by different exposure parameters, and the multiple images are the first image sequence. Filtering the first image sequence is the preprocessing of the image, and the multiple images after processing are the second image sequence. The second image sequence and the first image sequence are in a mapping relationship, wherein the preprocessed image is reflected in that the brightness difference of the pixel points of each image in the second image sequence is greater than or equal to the brightness difference of the pixel points of the corresponding image in the first image sequence, so the filtering process is to pre-adjust the brightness of the image, increase the difference of at least some pixels in each image in the first image sequence, for example, the brightness difference of some pixels in the first image sequence is small, and the brightness difference of the part of the pixels can be increased. If the difference between the two ends of some pixels in the first image sequence is large, it is not necessary to adjust the brightness difference of the part of the pixels. After processing the images in the first image sequence, the non-edge details of each image in the second image sequence will become blurred, while the edges of the images are retained, which can provide convenience for subsequent image fusion operations.
[0086] After the image is preprocessed, the second image sequence is fused according to the target weight, so that multiple images are fused into one target image. By fusing multiple images through the target weight, the obtained target image can eliminate part of the halo and overcome the defect of local detail loss, thereby avoiding the imbalance of the overall brightness ratio of the image, so as to provide an image with a high dynamic range. Since the image is filtered in the above process, an intermediate image can be provided for the subsequent fusion step, so that multiple images can be fused in the spatial domain of pixels without the need for complex operations of multi-resolution decomposition or two-layer decomposition, thereby simplifying the image processing time. In the embodiments of the present application, the electronic device can achieve high dynamic range imaging by preprocessing and fusion of the image, which is not simply dependent on the imaging device. The electronic device does not need to be equipped with expensive imaging devices to achieve high dynamic range imaging, which is conducive to improving the competitiveness of the product.
[0087] In a possible embodiment, the filtering module 910 includes: a first determination module, which is used to determine a brightness difference value sequence of pixel points in each image of the first image sequence; a selection module, which selects a target brightness difference value in the brightness difference value sequence according to a preset brightness difference threshold; a second determination module, which determines a brightness kernel function value according to the target brightness difference value; and a filtering submodule, which filters each image of the first image sequence according to the brightness kernel function value to obtain a second image sequence.
[0088] In this embodiment, the first image sequence includes multiple subsequences, such as a brightness difference sequence, a color sequence, etc. In this embodiment, the brightness difference sequence in the first image sequence is acquired and preprocessed.
[0089] According to the preset brightness difference threshold, a target brightness difference in the brightness difference sequence is selected. Specifically, the brightness difference between two pixels in the brightness difference sequence is obtained, and the brightness difference is compared with the brightness difference threshold. The brightness difference that meets the set conditions is the target brightness difference.
[0090] The brightness kernel function value is calculated based on the target brightness difference. Exemplarily, the brightness difference threshold in this embodiment includes a first threshold and a second threshold, and the first threshold is greater than the second threshold. When the brightness difference between two pixels is less than the second threshold, it is considered that the two pixels that obtain the brightness difference belong to the same category, and their two kernel function values are 1. When the brightness difference between two pixels is greater than the first threshold, it is considered that the two pixels that obtain the brightness difference are not in the same category, and their kernel function value is 0. When the two differences between two pixels are between the first threshold and the second threshold, it is considered that the two pixels that obtain the two differences approximately belong to the same category, and their kernel function value is a value between 0 and 1, such as Fig.11 As shown, in this embodiment, the kernel function between 0 and 1 is in a linear function relationship with the brightness difference threshold, and the first threshold and the second threshold can be values set according to usage requirements. According to the brightness difference threshold, a target brightness difference is selected, for example, a brightness difference less than the second threshold is selected as the target two differences, and then the brightness kernel function value is calculated based on the two differences, and each image of the first image sequence is filtered according to the brightness kernel function value. Through the above operation, the non-edge details of each image in the first image sequence become blurred, and the edges of the original image are retained, so that in the subsequent image fusion process, the fusion can be directly based on the hole domain of the pixel, and the framework with high time complexity such as multi-resolution decomposition is not used, which saves fusion time, reduces fusion difficulty, and improves fusion effect.
[0091] In a possible embodiment, the filtering module 910 further includes: a first calculation module, which is used to calculate the brightness difference between each pixel in the image and the pixels in its neighborhood to obtain a brightness difference sequence.
[0092] In this embodiment, the brightness difference of each pixel in its area is calculated. Specifically, a pixel is taken as the central pixel, and the pixels adjacent to the central pixel belong to the neighborhood pixel of the central pixel. For example, Fig.10As shown, the central pixel point is P, and its neighborhood Ω has 8 pixels, that is, a 3×3 neighborhood. Only the 3×3 neighborhood is taken as an example to illustrate the kernel function value calculation method of this embodiment. First, the central pixel point P is taken as a reference, P represents the spatial coordinates of the pixel point, and the brightness distance between it and the neighboring pixel points is calculated. Taking the neighborhood point q as an example, the calculation formula is:
[0093] d k,p =|L k,p -L k,q |,q∈Ω;
[0094] Among them, d k,p is the brightness distance between the pixel p in the kth brightness difference sequence and the pixels in its neighborhood Ω, L k,p is the brightness of pixel P, L k,q is the brightness of pixel q, at brightness distance d k,p On the basis of Fig.11 The mapping relationship can calculate the kernel function ω of pixel point p k,p , the specific description is as follows: when the brightness distance d k,p Greater than the threshold σ H When the kernel function ω k,p The value is 0; when the brightness distance d k,p Less than the threshold σ L When the kernel function ω k,p The value is 1; when the brightness distance d k,p Between the threshold σ H and σ L When the kernel function ω k,p The value of is the brightness distance d k,p A linear function of H and σ L Keep it continuous.
[0095] Calculate the brightness difference of each pixel in its area to obtain a brightness difference sequence. According to the preset brightness difference threshold, select the target brightness difference in the brightness difference sequence, and determine the brightness kernel function value according to the target brightness difference. Specifically, when the brightness distance between the neighborhood pixel and the central pixel is small, the threshold σ is used. L As the boundary, the neighborhood pixel and the central pixel are considered to belong to the same category, and its function value is set to 1; when the brightness distance between the neighborhood pixel and the central pixel is large, the threshold σ H As the boundary, it is considered that the neighborhood pixel and the central pixel do not belong to the same category, and its function value is set to 0; when the brightness distance between the neighborhood pixel and the central pixel is moderate, the threshold σ L and σ H As the boundary, it is considered that the neighborhood pixels and the central pixel can approximately belong to the same category, and its function value is set to a value between 0 and 1.
[0096] When calculating the brightness difference between each central point and the adjacent pixel points, repeated calculations may occur. This can be further accelerated to save the steps of repeated calculations.
[0097] When the value of the kernel function is 0, it indicates that the pixel point does not participate in the filtering operation of the central pixel point p, and the filtering operation is step S13; when the value of the kernel function is 1, it indicates that the pixel point fully participates in the filtering operation of the central pixel point p; when the value of the kernel function is between 0 and 1, it indicates that the pixel point p partially participates in the filtering operation of the central pixel point.
[0098] Each image in the first image sequence is filtered according to the brightness kernel function value. Each pixel point p in the multi-exposure image sequence has a kernel function ω that matches its neighborhood content. k,p , and perform the same filtering operation on the RGB (full color image) three channels, which refer to: red channel, green channel and blue channel.
[0099] The calculation is performed through the following disclosure:
[0100]
[0101] Among them, I k,c,p Refers to the pixel value of pixel p on channel c (c can be one of the three channels mentioned above) of the k-th multi-exposure image, represents the summation operation in the neighborhood of pixel p, F k,c,p is the filtering result, i.e., the pixel value of pixel point p on channel c of the k-th filtered image. Through the filtering operation, the filtered second image sequence is obtained for use in subsequent steps. The purpose of this step is to provide an intermediate image for the calculation of the weight function, so that the calculated weight map can be directly used for image fusion in the spatial domain, thereby reducing the time complexity of the calculation process, while effectively suppressing existing defects such as halo.
[0102] In a possible embodiment, the fusion module 920 includes: a third determination module, which determines the global weight value and the local weight value of each image in the first image sequence; and a second calculation module, which calculates the target weight function according to the global weight value and the local weight value.
[0103] In this embodiment, it is proposed that a target weight function is calculated by a global weight value and a local weight value. When multiple images are fused by a global weight value, attention is paid to the overall brightness distribution of the image, and there is a possibility of loss of local details. When multiple images are fused by a local weight value, attention is paid to the presentation of details, but there is a possibility of contrast abnormalities. In this embodiment, the above problems are comprehensively considered, and a target weight function is obtained by combining the global weight value and the local weight value. This function not only takes into account the overall brightness distribution of the image, but also pays attention to the local details of the image, thereby avoiding imbalance in the overall brightness distribution of the image and facilitating the realization of high dynamic range imaging.
[0104] The global weight focuses on the overall brightness distribution of the fused image, and there is a possibility of local detail loss. The local weight focuses on the local detail presentation of the fused image, and there is a possibility of contrast abnormality. Therefore, the combined global weight {Wg k} and local weight {Wl k}, and get the final weight function:
[0105] W k =(Wg k ) α ×Wl k ;
[0106] Where α represents the combined ratio of global weight and local weight. The ratio of global weight and local weight can be appropriately adjusted according to the actual needs of the application.
[0107] In order to solve the defects of existing exposure fusion algorithms such as edge halo, local detail loss, high time complexity, and imbalanced brightness distribution, this embodiment starts from the method of direct fusion in the pixel domain, does not adopt high time-complexity frameworks such as multi-resolution decomposition, but pre-processes the image through an edge-preserving filtering method, and then calculates the weight function to eliminate possible defects such as halo and local detail loss, and combines global weights and local weights to avoid imbalanced overall brightness distribution of the image.
[0108] The weight combination calculation method proposed in this embodiment can realize scene-adaptive combination parameter selection based on the content of the target scene by using ideas such as machine learning to achieve a more intelligent visual effect.
[0109] In a possible embodiment, the fusion module 920 further includes: a third calculation module, which calculates a global weight value of each image material according to a brightness parameter in the first image sequence.
[0110] In this embodiment, a method for calculating the global weight value is defined. The global weight calculation is to assign a global weight to each image in the multi-exposure image sequence, that is, the target image is regarded as a whole, and its weight value is a constant, which can be described as:
[0111] W k =f g (I k );
[0112] Typically, the global weight function can be used in the following form:
[0113]
[0114] Where μ(·) represents the normalized brightness mean of the image, is a control parameter. The physical meaning of this global weight function is that when the mean brightness of an image is close to 0.5, its global weight is larger, and vice versa. In this way, when the overall brightness of the image is closer to 0.5, it means that it can provide more useful information, so its global weight is set to a larger value.
[0115] Global weight calculation is a calculation method in which the global weight function f g (·) Not limited to the example, when the input multi-exposure image sequence {I k}, output {Wg k When} is a constant vector, it can be used as a global weight function. For example, when combined with histogram regularization, the fused image is made to meet certain histogram conditions, and the corresponding global weight function f can be obtained. g The specific form of (·).
[0116] In a possible embodiment, the fusion module 920 also includes: a fourth calculation module, which calculates the weight value of each pixel point of the image in the second image sequence according to the pixel value in the second image sequence; and a fifth calculation module, which performs calculation according to the weight value of each pixel point to obtain the local weight value of each image in the first image sequence.
[0117] In this embodiment, a method of calculating a local weight value is defined, and the local weight value is calculated to filter the image sequence {F k} is the basis for calculation, for each multi-exposure image {I k}Give a local weight that reflects the local details. The weight value is a two-dimensional function, which can be described as:
[0118] Wl k =f l (F k );
[0119] Typically, the local weight function can be of the following form:
[0120]
[0121] in, Indicates that the local weights are calculated for the three channels of the filtered image and their product is calculated. k,c represents the pixel value of channel c of the kth filtered image, and ρ is the control parameter.
[0122] The local weight calculation is a calculation method in which the local weight function f g (·) Not limited to the example, the two-dimensional function calculated pixel by pixel can be used as the local weight function {Wl k The local weight calculation method of this embodiment is that the weight function is calculated in the filtered image sequence {F k}, but not on the multi-exposure image sequence {I k}, which has the characteristic of eliminating artifacts such as halos while retaining local details.
[0123] The image processing device in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.
[0124] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.
[0125] like Figure 8 As shown, the embodiment of the present application also provides an electronic device 100, including a processor 110, a memory 109, and a program or instruction stored in the memory 109 and executable on the processor 110. When the program or instruction is executed by the processor 110, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. It should be noted that the electronic device in the embodiment of the present application includes the above-mentioned mobile electronic device and non-mobile electronic device.
[0126] Fig. 9A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.
[0127] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.
[0128] Those skilled in the art will appreciate that the electronic device 100 may further include a power source 111 (such as a battery) for supplying power to various components. The power source 111 may be logically connected to the processor 110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.
[0129] Among them, the processor 110 is used to obtain a first image sequence with different exposure parameters, filter the first image sequence to obtain a second image sequence, the second image sequence is mapped to the first image sequence, and the brightness difference of the pixel points of each image in the second image sequence is greater than or equal to the brightness difference of the pixel points of the corresponding image in the first image sequence; determine the target weight function; and fuse the second image sequence according to the target weight function to obtain the target image.
[0130] Optionally, the processor 110 is used to determine a brightness difference sequence of pixel points in each image of the first image sequence; select a target brightness difference value in the brightness difference sequence according to a preset brightness difference threshold; determine a brightness kernel function value according to the target brightness difference value; and filter each image of the first image sequence according to the brightness kernel function value to obtain a second image sequence.
[0131] Optionally, the processor 110 is configured to calculate a brightness difference between each pixel in the image and pixels in its neighborhood to obtain a brightness difference sequence.
[0132] Optionally, the processor 110 is used to determine a global weight value and a local weight value of each image in the first image sequence; and calculate a target weight function according to the global weight value and the local weight value.
[0133] Optionally, the processor 110 is configured to calculate a global weight value of each image material according to a brightness parameter in the first image sequence.
[0134] Optionally, the processor 110 is used to calculate the weight value of each pixel point of the image in the second image sequence according to the pixel value in the second image sequence; and perform calculation according to the weight value of each pixel point to obtain the local weight value of each image in the first image sequence.
[0135] By filtering the image, an intermediate image can be provided for the subsequent fusion step, so that multiple images can be fused in the spatial domain of pixels without having to perform complex operations of multi-resolution decomposition or two-layer decomposition, thereby simplifying the image processing time. In the embodiments of the present application, the electronic device can achieve high dynamic range imaging by preprocessing and fusion of the image, which does not simply rely on imaging devices. High dynamic range imaging can be achieved without the need to equip the electronic device with expensive imaging devices.
[0136] It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include but are not limited to a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here. The memory 109 can be used to store software programs and various data, including but not limited to applications and operating systems. The processor 110 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, the user interface, and the application program, etc., and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 110.
[0137] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0138] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0139] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0140] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
[0141] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0142] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0143] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. An image processing method, characterized in that: include: Acquire a first image sequence with different exposure parameters, perform filtering processing on the first image sequence to obtain a second image sequence, the second image sequence is in a mapping relationship with the first image sequence, the brightness difference value of the pixel points of each image in the second image sequence is greater than or equal to the brightness difference value of the corresponding pixel points of the image in the first image sequence, the brightness difference value of the pixel points of the image in the first image sequence is the brightness difference value of each pixel point and the pixel points in its neighborhood, and the filtering processing is used to increase the brightness difference value of at least part of the pixel points in each image in the first image sequence; Determine the target weight function; The second image sequence is fused according to the target weight function to obtain a target image.
2. The image processing method according to claim 1, characterized in that: The filtering the first image sequence to obtain the second image sequence specifically includes: Determine a brightness difference value sequence of pixel points in each image of the first image sequence; According to a preset brightness difference threshold, selecting a target brightness difference value in the brightness difference value sequence; Determine a brightness kernel function value according to the target brightness difference value; Each image in the first image sequence is filtered according to the brightness kernel function value to obtain the second image sequence.
3. The image processing method according to claim 2, characterized in that: The step of determining a sequence of brightness difference values of pixels in each image of the first image sequence specifically includes: The brightness difference between each pixel in the image and the pixels in its neighborhood is calculated to obtain the brightness difference sequence.
4. The image processing method according to any one of claims 1 to 3, characterized in that: The determining of the target weight function specifically includes: Determining a global weight value and a local weight value for each image in the first image sequence; The target weight function is calculated according to the global weight value and the local weight value.
5. The image processing method according to claim 4, characterized in that: The determining of the global weight value of each image in the first image sequence specifically includes: The global weight value of each image material is calculated according to the brightness parameter in the first image sequence.
6. The image processing method according to claim 4, characterized in that: The determining of the local weight value of each image in the first image sequence specifically includes: Calculating a weight value of each pixel point of an image in the second image sequence according to a pixel value in the second image sequence; Calculation is performed according to the weight value of each pixel to obtain a local weight value of each image in the first image sequence.
7. An image processing device, characterized in that: include: a filtering module, wherein the filtering module acquires a first image sequence with different exposure parameters, and performs filtering processing on the first image sequence to obtain a second image sequence, wherein the second image sequence is in a mapping relationship with the first image sequence, the brightness difference value of the pixel points of each image in the second image sequence is greater than or equal to the brightness difference value of the corresponding pixel points of the image in the first image sequence, the brightness difference value of the pixel points of the image in the first image sequence is the brightness difference value of each pixel point and the pixel points in its neighborhood, and the filtering processing is used to increase the brightness difference value of at least some pixel points in each image in the first image sequence; A fusion module is used to determine a target weight function and to perform fusion processing on the second image sequence according to the target weight function to obtain a target image.
8. The image processing device according to claim 7, characterized in that: The filtering module comprises: A first determining module, the first determining module is used to determine a brightness difference value sequence of pixel points in each image of the first image sequence; A selection module, wherein the selection module selects a target brightness difference value in the brightness difference value sequence according to a preset brightness difference threshold; a second determination module, wherein the second determination module determines a brightness kernel function value according to the target brightness difference value; A filtering submodule is configured to filter each image in the first image sequence according to the brightness kernel function value to obtain the second image sequence.
9. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the image processing method as claimed in any one of claims 1 to 6.
10. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 6 are implemented.
