Multi-scale Registration and Fusion Method, Device, Terminal Device and Storage Medium of Images

Abstract

The present invention discloses a multi-scale registration and fusion method, apparatus, terminal device and computer-readable storage medium for images. The method performs image registration processing on the original image data to be fused and obtains the grayscale image data corresponding to the original image data; processes the grayscale image data to determine the multi-focus fusion weight coefficients of the grayscale image data; performs weight calculation based on pixel intensity and weight calculation based on global gradient on the grayscale image data to determine the multi-exposure fusion weight coefficients of the grayscale image data; and performs multi-focus and multi-exposure image fusion processing on the original image data according to the multi-focus fusion weight coefficients and the multi-exposure fusion weight coefficients to obtain the fused image data.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a multi-scale registration and fusion method, device, terminal device, and computer-readable storage medium for images. Background Art

[0002] With the rapid development of technology, in order to obtain richer, more accurate, and more reliable useful information, it is often necessary to perform multi-faceted and multi-level data fusion processing on information from multiple sensors using various mathematical statistics and other methods. Among them, image fusion technology is a type of visual data fusion technology that combines disciplines such as computer technology, signal processing technology, image processing technology, and sensor technology. The process of image fusion is to fuse several images of the same target according to a specific purpose to obtain an image with a more accurate and comprehensive description of the information of this target.

[0003] Currently, in the actual imaging process of images, due to different focal points, the unfocused target and the focused target cannot be clearly imaged simultaneously. Due to the uneven light source intensity, under-exposed and over-exposed parts appear in the imaging, and these situations will cause the scenery in an image not to be clearly presented. However, in the mainstream algorithms in the field of image fusion, usually only one of the multi-exposure or multi-focus images is fused, without considering the image fusion method when both situations exist simultaneously. In addition, the mainstream algorithms cannot simultaneously ensure high precision and high speed in the fusion process, and cannot meet the applications of actual intelligent devices.

[0004] In summary, how to improve the algorithm accuracy while reducing the algorithm complexity, so as to realize the image fusion in both multi-focus and multi-exposure situations to obtain high-quality and high-resolution real-time fusion images has become an urgent technical problem to be solved in the field of image processing technology. Summary of the Invention

[0005] The main purpose of the present invention is to provide a multi-scale registration and fusion method, device, terminal device, and computer-readable storage medium for images. The aim is to improve the algorithm accuracy while reducing the algorithm complexity, and to realize the image fusion in both multi-focus and multi-exposure situations to obtain high-quality and high-resolution real-time fusion images.

[0006] To achieve the above object, the present invention provides a multi-scale registration and fusion method for images, and the multi-scale registration and fusion method for images includes:

[0007] Performing image registration processing on the original image data to be fused, and obtaining the grayscale image data corresponding to the original image data;

[0008] Process the grayscale image data to obtain the multi-focus fusion weight coefficients of the grayscale image data;

[0009] Perform weight calculation based on pixel intensity and weight calculation based on global gradient on the grayscale image data to obtain the multi-exposure fusion weight coefficients of the grayscale image data;

[0010] Perform multi-focus multi-exposure image fusion processing on the original image data according to the multi-focus fusion weight coefficients and the multi-exposure fusion weight coefficients to obtain the fused image data.

[0011] Optionally, the step of processing the grayscale image data to obtain the multi-focus fusion weight coefficients of the grayscale image data includes:

[0012] Determine the initial decision map of the grayscale image data;

[0013] Process the initial decision map through a controlled guided filter to obtain the multi-focus fusion weight coefficients of the grayscale image data.

[0014] Optionally, the step of determining the initial decision map of the grayscale image data includes:

[0015] Determine the first focus map of the grayscale image data;

[0016] Process the first focus map according to the per-pixel maximum rule to obtain the initial decision map of the grayscale image data.

[0017] Optionally, the step of determining the first focus map of the grayscale image data includes:

[0018] Determine the second focus map of the grayscale image data;

[0019] Refine the second focus map through a controlled guided filter to obtain the first focus map of the grayscale image data.

[0020] Optionally, the step of determining the second focus map of the grayscale image data includes:

[0021] Perform mean filtering on the grayscale image data to obtain mean-filtered image data;

[0022] Extract the high-frequency information in the mean-filtered image data to obtain the second focus map of the grayscale image data.

[0023] Optionally, the step of performing weight calculation based on pixel intensity and weight calculation based on global gradient on the grayscale image data to obtain the multi-exposure fusion weight coefficients of the grayscale image data includes:

[0024] Perform weight calculation based on pixel intensity for the grayscale image data to determine the first weight coefficient;

[0025] Perform weight calculation based on global gradient for the grayscale image data to determine the second weight coefficient;

[0026] Determine the multi - exposure fusion weight coefficient of the grayscale image data according to the first weight coefficient and the second weight coefficient.

[0027] Optionally, the step of performing multi - focus multi - exposure image fusion processing on the original image data according to the multi - focus fusion weight coefficient and the multi - exposure fusion weight coefficient includes:

[0028] Perform Gaussian pyramid decomposition processing on the original image data to obtain multi - layer pyramid images;

[0029] Perform multi - focus multi - exposure image fusion processing on the multi - layer pyramid images according to the multi - focus fusion weight coefficient and the multi - exposure fusion weight coefficient.

[0030] In addition, to achieve the above object, the present invention further provides a multi - scale registration and fusion device for images, and the multi - scale registration and fusion device for images includes:

[0031] An image registration module, which performs image registration processing on the original image data to be fused and obtains the grayscale image data corresponding to the original image data;

[0032] A multi - focus weight module, which processes the grayscale image data to obtain the multi - focus fusion weight coefficient of the grayscale image data;

[0033] A multi - exposure weight module, which performs weight calculation based on pixel intensity and weight calculation based on global gradient on the grayscale image data to obtain the multi - exposure fusion weight coefficient of the grayscale image data;

[0034] An image fusion module, which performs multi - focus multi - exposure image fusion processing on the original image data according to the multi - focus fusion weight coefficient and the multi - exposure fusion weight coefficient to obtain fused image data.

[0035] In addition, to achieve the above object, the present invention further provides a terminal device, and the terminal device includes: a memory, a processor, and a multi - scale registration and fusion program for images stored in the memory and executable on the processor. When the multi - scale registration and fusion program for images of the terminal device is executed by the processor, the steps of the above - mentioned multi - scale registration and fusion method for images are implemented.

[0036] In addition, to achieve the above object, the present invention further provides a computer-readable storage medium, on which a multi-scale registration and fusion program of images is stored. When the multi-scale registration and fusion program of images is executed by a processor, the steps of the multi-scale registration and fusion method of images as described above are implemented.

[0037] A multi-scale registration and fusion method, device, terminal device and computer-readable storage medium for images proposed in an embodiment of the present invention. The method includes obtaining original image data, performing image registration processing on the original image data, and obtaining grayscale image data corresponding to the original image data; processing the grayscale image data to obtain multi-focus fusion weight coefficients of the grayscale image data; performing weight calculation based on pixel intensity and weight calculation based on global gradient on the grayscale image data to obtain multi-exposure fusion weight coefficients of the grayscale image data; and performing multi-focus and multi-exposure image fusion processing on the original image data according to the multi-focus fusion weight coefficients and the multi-exposure fusion weight coefficients to obtain fused image data.

[0038] In an embodiment of the present invention, by performing image registration preprocessing on the obtained original image data, grayscale image data corresponding to the original image data is obtained, and a series of data processing is performed on the grayscale image data, so as to determine the multi-focus fusion weight coefficients of the grayscale image data, and perform weight calculation based on pixel intensity and weight calculation based on global gradient on the grayscale image data, so as to determine the multi-exposure fusion weight coefficients of the grayscale image data. Finally, according to the multi-focus fusion weight coefficients and the multi-exposure fusion weight coefficients, multi-focus and multi-exposure image fusion processing of the original image data is completed to obtain the final fused image data. In this way, the present invention realizes the improvement of algorithm accuracy while reducing the algorithm complexity by adopting a method combining multi-scale decomposition in the transform domain and pixel weighting in the spatial domain, and further realizes image fusion in both multi-focus and multi-exposure cases to obtain a high-quality and high-resolution real-time fused image. Description of the Drawings

[0039] Figure 1 It is a schematic diagram of the device structure of the hardware operating environment of the terminal device involved in the solution of the embodiment of the present invention;

[0040] Figure 2 It is a schematic diagram of the step flow of the first embodiment of the multi-scale registration and fusion method of images of the present invention;

[0041] Figure 3 It is a schematic diagram of the implementation of the FEF algorithm involved in an embodiment of the multi-scale registration and fusion method of images of the present invention;

[0042] Figure 4Schematic diagram of the image registration algorithm involved in an embodiment of the multi-scale registration and fusion method of the images of the present invention;

[0043] Figure 5 Schematic diagram of the overall process of multi-focus fusion involved in an embodiment of the multi-scale registration and fusion method of the images of the present invention;

[0044] Figure 6 Schematic diagram of the overall process of multi-exposure fusion involved in an embodiment of the multi-scale registration and fusion method of the images of the present invention;

[0045] Figure 7 Schematic diagram of the simulation processing results of the FEF algorithm involved in an embodiment of the multi-scale registration and fusion method of the images of the present invention.

[0046] Figure 8 Schematic diagram of the functional modules of an embodiment of the multi-scale registration and fusion device for the images of the present invention.

[0047] The realization, functional characteristics and advantages of the objectives of the present invention will be further described with reference to the embodiments and the accompanying drawings. Detailed implementation manners

[0048] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0049] Refer to Figure 1 , Figure 1 Schematic diagram of the device structure of the hardware operating environment of the terminal device involved in the solution of the embodiment of the present invention.

[0050] The terminal device in the embodiment of the present invention can be a terminal device applied to the field of image fusion processing technology and integrated with the FEF (multi-Focus and multi-Exposure fusion) algorithm. Among them, the FEF algorithm is an algorithm proposed by combining the algorithm ideas of filter analysis, multi-focus multi-exposure fusion, multi-scale registration, etc. Specifically, the terminal device can be a smart phone, a PC (Personal Computer), a tablet computer, a portable computer, etc.

[0051] Such as Figure 1As shown in the figure, the terminal device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (DiSplay) and an input unit such as a keyboard (Keyboard). Optionally, the user interface 1003 may further include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0052] Those skilled in the art can understand that Figure 1 the terminal device structure shown in does not constitute a limitation on the terminal device, and it may include more or fewer components than shown in the figure, or combine some components, or have different component arrangements.

[0053] As Figure 1 shown, the memory 1005, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a multi-scale registration and fusion program for images.

[0054] In Figure 1 the terminal shown, the network interface 1004 is mainly used to connect to the background server and communicate with the background server for data; the user interface 1003 is mainly used to connect to the client and communicate with the client for data; and the processor 1001 may be used to call the multi-scale registration and fusion program for images stored in the memory 1005 and perform the following operations:

[0055] Perform image registration processing on the original image data to be fused and obtain the grayscale image data corresponding to the original image data;

[0056] Process the grayscale image data to obtain the multi-focus fusion weight coefficients of the grayscale image data;

[0057] Perform weight calculation based on pixel intensity and weight calculation based on global gradient on the grayscale image data to determine the multi-exposure fusion weight coefficients of the grayscale image data;

[0058] Perform multi-focus and multi-exposure image fusion processing on the original image data according to the multi-focus fusion weight coefficients and the multi-exposure fusion weight coefficients to obtain the fused image data.

[0059] Optionally, the processor 1001 can also be used to call the multi-scale registration and fusion program of the images stored in the memory 1005, and perform the following operations:

[0060] Determine the initial decision map of the grayscale image data;

[0061] Process the initial decision map through a controlled guided filter to obtain the multi-focus fusion weight coefficients of the grayscale image data.

[0062] Optionally, the processor 1001 can also be used to call the multi-scale registration and fusion program of the images stored in the memory 1005, and perform the following operations:

[0063] Determine the first focus map of the grayscale image data;

[0064] Process the first focus map according to the per-pixel maximum rule to determine the initial decision map of the grayscale image data.

[0065] Optionally, the processor 1001 can also be used to call the multi-scale registration and fusion program of the images stored in the memory 1005, and perform the following operations:

[0066] Determine the second focus map of the grayscale image data;

[0067] Refine the second focus map through a controlled guided filter to determine the first focus map of the grayscale image data.

[0068] Optionally, the processor 1001 can also be used to call the multi-scale registration and fusion program of the images stored in the memory 1005, and perform the following operations:

[0069] Perform mean filtering on the grayscale image data to obtain mean-filtered image data;

[0070] Extract the high-frequency information in the mean-filtered image data to obtain the rough focus map of the grayscale image data.

[0071] Optionally, an absolute encoder is provided on the walking wheels of the ship unloader. The processor 1001 can also be used to call the multi-scale registration and fusion program of the images stored in the memory 1005, and perform the following operations:

[0072] Perform weight calculation based on pixel intensity on the grayscale image data to determine the first weight coefficient;

[0073] Perform weight calculation based on global gradient on the grayscale image data to determine the second weight coefficient;

[0074] Determine the multi-exposure fusion weight coefficients of the grayscale image data according to the first weight coefficient and the second weight coefficient.

[0075] Optionally, the processor 1001 can also be used to call the multi-scale registration and fusion program of the image stored in the memory 1005 and perform the following operations:

[0076] Perform Gaussian pyramid decomposition processing on the original image data to obtain multi-layer pyramid images;

[0077] Perform multi-focus and multi-exposure image fusion processing on the multi-layer pyramid images according to the multi-focus fusion weight coefficients and the multi-exposure fusion weight coefficients.

[0078] Based on the above terminal device, various embodiments of the multi-scale registration and fusion method of the image of the present invention are proposed.

[0079] Currently, in the actual imaging process of an image, due to different focal points, the un-focused target and the focused target cannot be clearly imaged simultaneously. Due to the uneven light source intensity, underexposed and overexposed parts appear in the imaging. These situations will cause the various scenes in an image not to be clearly presented. However, in the mainstream algorithms in the field of image fusion, usually only one of the multi-exposure or multi-focus image fusion is realized, and the fusion method for the case where both situations exist simultaneously is not considered. In addition, the mainstream algorithms cannot ensure both high precision and high speed in the fusion processing, and cannot meet the applications of actual intelligent devices.

[0080] In view of the above phenomenon, the present invention proposes a multi-scale registration and fusion method for images. The multi-scale registration and fusion method for images of the present invention performs image registration preprocessing on the acquired original image data, obtains the grayscale image data corresponding to the original image data, performs a series of data processing on the grayscale image data, thereby determining the multi-focus fusion weight coefficients of the grayscale image data, and performing weight calculation based on pixel intensity and weight calculation based on global gradient on the grayscale image data, thereby determining the multi-exposure fusion weight coefficients of the grayscale image data. Finally, according to the multi-focus fusion weight coefficients and the multi-exposure fusion weight coefficients, the multi-focus and multi-exposure image fusion processing of the original image data is completed to obtain the final fusion image data. Therefore, the present invention realizes the improvement of the algorithm precision while reducing the algorithm complexity by adopting the method of combining multi-scale decomposition in the transform domain and pixel weighting in the spatial domain, and further realizes the image fusion in both multi-focus and multi-exposure situations to obtain a high-quality and high-resolution real-time fusion image.

[0081] Please refer to Figure 2 , Figure 2This is a schematic flowchart of the first embodiment of the multi-scale registration and fusion method for the images of the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the multi-scale registration and fusion method for the images of the present invention can of course be executed in a different order from the steps shown or described herein.

[0082] In the first embodiment of the multi-scale registration and fusion method for the images of the present invention, the multi-scale registration and fusion method for the images of the present invention includes:

[0083] Step S10: Perform image registration processing on the original image data to be fused, and obtain the grayscale image data corresponding to the original image data;

[0084] In this embodiment, the FEF algorithm is called by the terminal device integrated with the FEF algorithm to perform preprocessing of image registration on the original image data to be fused, and obtain the grayscale image data corresponding to the original image data.

[0085] It should be noted that the multi-focus and multi-exposure fusion algorithm FEF consists of three major parts, namely, the preprocessing part of image registration, the multi-focus fusion part, and the multi-exposure fusion part: among them, the image registration part is implemented by the SSD (Sum of Squared Difference) algorithm; the multi-focus fusion part uses multi-scale Gaussian pyramid hierarchical processing based on a guided filter, and its weight is determined by a decision map obtained by a series of image decomposition filtering processes; the multi-exposure fusion part obtains weight coefficients through a series of processes such as mean histogram normalization and Gaussian filtering, and then performs hierarchical processing through Gaussian pyramid decomposition. The above two fusion processes are both processed multi-scale in the Gaussian pyramid. Therefore, the Gaussian pyramid decomposition is first performed on the original image, and the multi-focus and multi-exposure fusion processing is performed on each level of the decomposition, and finally an overall reconstruction is performed to obtain a fused image.

[0086] Exemplarily, image registration is mainly a necessary intermediate step in the process of multi-frame image fusion. In a dual-lens device, due to the physical distance between the lenses, in the imaging process, the two lenses cannot obtain exactly the same image of a target, and there will be pixel displacement, resulting in misalignment in the fusion and causing poor fusion effects such as ghosting. At this time, it is very necessary to perform pixel registration during the fusion of two frames of images. In a single-lens device, in actual situations, usually due to external interference, the pixel alignment of multiple frames of images taken is not good, which will affect the image fusion effect. If a tripod or rapid photography technology is used to make the pixels of consecutive multiple frames of images perfectly aligned, then there is no need to perform registration operations during the fusion process. Therefore, image matching is mainly to complete the registration and alignment work between the input images to ensure the smooth progress of subsequent image fusion.

[0087] Exemplarily, in this embodiment, the SSD algorithm based on grayscale information is adopted. The terminal device integrated with the FEF algorithm calls the FEF algorithm to obtain two pieces of original image data RGB (Red, Green, Blue, red, green, blue) 1 and RGB2. After the YCrCb color space transformation, the luminance information of the two pieces of original image data is extracted to obtain the grayscale image data Y1 and Y2 corresponding to the two pieces of original image data. The sum of the squares of the differences of each pixel of the two grayscale images is calculated, that is, the L2 distance between the sub-image and the template image. The formula is as follows:

[0088]

[0089] Among them, T(x’, y’) represents the pixel point of the template image, which is a constant. I(x + x’, y + y’) represents the pixel point of the original image data and can also be approximated as a constant. R(x, y) represents the sum of the squares of the differences of each pixel of the two images.

[0090] When finding the optimal matching block, the SSD function measures the approximate information between image blocks through the R, G, and B elements corresponding to the pixel points between image blocks. It is a method for measuring the approximation of image blocks used more frequently. The SSD function can be introduced to search for the optimal matching block. As Figure 4 shown, taking the image of one of the channels as an example, starting from the middle position of the two frames of original images to be registered, the size of the matching block is set. Here, the matching blocks croppedA and croppedB with a size of 151x151 pixels are selected. The SSD optimization is performed on these two matching blocks to find the horizontal and vertical displacement amounts corresponding to the minimum SSD value between the two blocks. Set the corresponding displacement range and continuously loop this step. Here, the displacement range is set to 20x20 pixels. When the displacement amount of the two frames of images exceeds this range, the algorithm will no longer be applicable, and croppedA and croppedB will be cycled and moved simultaneously. When the optimal horizontal and vertical displacement amounts are found, the entire image to be matched is shifted by this displacement amount value, and the two images can be successfully registered and simply fused.

[0091] It should be noted that in this embodiment, the terminal device integrated with the FEF algorithm can call the FEF algorithm to obtain two or more pieces of original image data and complete the multi-focus multi-exposure fusion processing of two or more pieces of original image data. It should be understood that based on different design requirements of actual applications, in different feasible implementation manners, the number of image data to be processed can be any number, and the multi-scale registration and fusion method of the images of the present invention does not limit the specific number of the image data to be processed.

[0092] Step S20: Process the grayscale image data to obtain the multi-focus fusion weight coefficient of the grayscale image data;

[0093] In this embodiment, the terminal device integrated with the FEF algorithm calls the FEF algorithm to process the grayscale image data corresponding to the original image data, so as to determine the multi-focus fusion weight coefficient of the grayscale image data.

[0094] It should be noted that in a general optical imaging system, the system function is a low-pass filter. The system function of the blurred area, i.e., the defocused area, has a narrower bandwidth than that of the clear area, i.e., the focused area. In an ideal case, the focused area has more high-frequency information than the defocused area. Therefore, the detection of the focused area is an important step in multi-focus image fusion.

[0095] Exemplarily, the terminal device integrated with the FEF algorithm calls the FEF algorithm to perform focused area detection on the grayscale image data Y1 and Y2, so as to determine the multi-focus fusion weight coefficient Wmap of the grayscale image data Y1 and Y2.

[0096] Step S30, perform weight calculation based on pixel intensity and weight calculation based on global gradient on the grayscale image data to obtain the multi-exposure fusion weight coefficient of the grayscale image data;

[0097] In this embodiment, the terminal device integrated with the FEF algorithm calls the FEF algorithm to perform weight calculation on the grayscale image data corresponding to the original image data, where the weight calculation includes weight calculation based on pixel intensity and weight calculation based on global gradient, so as to determine the multi-exposure fusion weight coefficient of the grayscale image data.

[0098] Exemplarily, as Figure 6 shown, the terminal device integrated with the FEF algorithm calls the FEF algorithm to perform two weight function calculations on the grayscale image data Y1 and Y2. The first weight function is based on pixel intensity and measures the importance of the pixel value relative to the overall brightness and adjacent exposure images. The second weight function is based on the global gradient and reflects the importance of a pixel value within a range where the global gradient is relatively large compared to other exposures. By performing weight calculation on the grayscale image data through the two weight functions, the multi-exposure fusion weight coefficient Wexp of the grayscale image data is determined.

[0099] Step S40, perform multi-focus multi-exposure image fusion processing on the original image data according to the multi-focus fusion weight coefficient and the multi-exposure fusion weight coefficient to obtain fused image data.

[0100] In this embodiment, the terminal device integrated with the FEF algorithm calls the FEF algorithm to perform multi-focus multi-exposure image fusion processing on the original image data according to the above multi-focus fusion weight coefficient Wmap and multi-exposure fusion weight coefficient Wexp, and obtains the final fused image data.

[0101] Further, in a feasible embodiment, in step S40, the step of "performing multi-focus and multi-exposure image fusion processing on the original image data according to the multi-focus fusion weight coefficient and the multi-exposure fusion weight coefficient" includes:

[0102] Step A, performing Gaussian pyramid decomposition processing on the original image data to obtain multiple layers of pyramid images;

[0103] Step B, performing multi-focus and multi-exposure image fusion processing on the multiple layers of pyramid images according to the multi-focus fusion weight coefficient and the multi-exposure fusion weight coefficient.

[0104] In this embodiment, the terminal device integrated with the FEF algorithm calls the FEF algorithm to perform Gaussian pyramid decomposition on the original image data to obtain multiple layers of pyramid images, and performs multi-focus and multi-exposure image fusion processing on each decomposed layer of pyramid images according to the multi-focus fusion weight coefficient and the multi-exposure fusion weight coefficient.

[0105] Exemplarily, as Figure 3 shown, the terminal device integrated with the FEF algorithm calls the FEF algorithm to perform Gaussian pyramid decomposition on the obtained original image data RGB1 and RGB2 to obtain each layer of the image after Gaussian pyramid decomposition, namely Py1, Py2... Pyn. Then, using the multi-focus fusion weight coefficient Wmap, that is, the final decision map FDM in the following formula, the original image data RGB1 and RGB2 are fused together through the pixel-by-pixel weighted average rule, and the final fused image IF is obtained:

[0106] I F (x,y) = FDM(x,y)I1(x,y) + (1 - FDM(x,y))I2(x,y)

[0107] where, I F (x,y) represents the pixel point in the fused image, FDM(x,y) represents the pixel point in the final decision map, I1(x,y) represents the pixel point in the original image data, and I2(x,y) represents the pixel point in another piece of original image data.

[0108] The multi-exposure fusion weight coefficient Wexp adopts multi-scale Gaussian pyramid decomposition, processes multi-exposure fusion in each pyramid, and synthesizes the final result.

[0109] In this embodiment, the multi-scale registration and fusion method of the image of the present invention calls the FEF algorithm through a terminal device integrated with the FEF algorithm to perform image registration preprocessing on the original image data to be fused, and obtains the grayscale image data corresponding to the original image data; the terminal device integrated with the FEF algorithm calls the FEF algorithm to perform data processing on the grayscale image data corresponding to the original image data, so as to determine the multi-focus fusion weight coefficient of the grayscale image data; the terminal device integrated with the FEF algorithm calls the FEF algorithm to perform weight calculation on the grayscale image data corresponding to the original image data, wherein the weight calculation includes weight calculation based on pixel intensity and weight calculation based on global gradient, so as to determine the multi-exposure fusion weight coefficient of the grayscale image data; the terminal device integrated with the FEF algorithm calls the FEF algorithm to perform multi-focus and multi-exposure image fusion processing on the original image data according to the above multi-focus fusion weight coefficient Wmap and multi-exposure fusion weight coefficient Wexp, and obtains the final fused image data; the terminal device integrated with the FEF algorithm calls the FEF algorithm to perform Gaussian pyramid decomposition on the original image data to obtain multi-layer pyramid images, and performs multi-focus and multi-exposure image fusion processing on each decomposed pyramid image according to the multi-focus fusion weight coefficient and multi-exposure fusion weight coefficient.

[0110] Thus, in the embodiment of the present invention, image registration preprocessing is performed on the original image data to be fused, and the grayscale image data corresponding to the original image data is obtained. Data processing is performed on the grayscale image data to determine the multi-focus fusion weight coefficient of the grayscale image data, and weight calculation based on pixel intensity and weight calculation based on global gradient are performed on the grayscale image data to determine the multi-exposure fusion weight coefficient of the grayscale image data. Finally, multi-focus and multi-exposure image fusion processing of the original image data is completed according to the multi-focus fusion weight coefficient and multi-exposure fusion weight coefficient, and the final fused image data is obtained. Further, the present invention combines multi-scale decomposition in the transform domain and pixel weighting in the spatial domain, realizes improving the algorithm accuracy while reducing the algorithm complexity, and further realizes image fusion in both multi-focus and multi-exposure cases to obtain a high-quality and high-resolution real-time fused image.

[0111] Further, based on the first embodiment of the multi-scale registration and fusion method of the image of the present invention, a second embodiment of the multi-scale registration and fusion method of the image of the present invention is proposed.

[0112] In this embodiment, for the multi-scale registration and fusion method of the image of the present invention, the above step S20 may include:

[0113] Step S201, determining an initial decision map of the grayscale image data;

[0114] Step S202: Process the initial decision graph through a guided filter to obtain the multi-focus fusion weight coefficients of the grayscale image data.

[0115] In this embodiment, a terminal device integrated with the FEF algorithm calls the FEF algorithm to determine the initial decision graph of the grayscale image data corresponding to the original image data, and controls the guided filter to perform filtering processing on the initial decision graph, thereby determining the multi-focus fusion weight coefficients of the grayscale image data.

[0116] Exemplarily, a terminal device integrated with the FEF algorithm calls the FEF algorithm to determine the initial decision graph DGYM, that is, IDM in the following formula. Since the initial decision graph adds false and irrelevant information, resulting in false detection, in order to obtain an ideal fused image, a guided filter is used to verify the spatial consistency of the initial fused image as a guidance image and generate the desired multi-focus fusion weight coefficients Wmap, that is, the final decision graph FDM in the following formula:

[0117] I IF (x,y) = IDM(x,y)I1(x,y) + (1 - IDM(x,y))I2(x,y)

[0118] FDM = GF2(I IF (x,y), IDM(x,y))

[0119] Wherein, I IF (x,y) represents the intermediate variable of the two original images acted on by the initial decision graph, and IDM(x,y) is the pixel point in the initial decision graph.

[0120] Further, in a feasible embodiment, for the multi-scale registration and fusion method of the image of the present invention, the above step S201 may include:

[0121] Step C: Determine the first focus map of the grayscale image data;

[0122] Step D: Process the first focus map according to the per-pixel maximum rule to obtain the initial decision graph of the grayscale image data.

[0123] In this embodiment, a terminal device integrated with the FEF algorithm calls the FEF algorithm to determine the accurate focus map of the grayscale image data corresponding to the original image data, that is, the first focus map, and processes the accurate focus map according to the per-pixel maximum rule, thereby determining the initial decision graph of the grayscale image data.

[0124] Exemplarily, the terminal device integrated with the FEF algorithm calls the FEF algorithm to determine the precise focus maps GYM1 and GYM2 corresponding to the original image data RGB1 and RGB2, that is, AFM1 and AFM2 in the following formula. The initial decision map DGYM, that is, IDM in the following formula, is obtained by adopting the per-pixel maximum rule of the corresponding precise focus maps AFM1 and AFM2:

[0125]

[0126] Among them, AFM1(x, y) represents the precise focus map of the original image data, and AFM2(x, y) represents the precise focus map of another piece of original image data.

[0127] Furthermore, in a feasible embodiment, for the multi-scale registration and fusion method of the image of the present invention, the above step C may include:

[0128] Step E, determining the second focus map of the grayscale image data;

[0129] Step F, refining the second focus map through a controlled guided filter to obtain the first focus map of the grayscale image data.

[0130] In this embodiment, the terminal device integrated with the FEF algorithm calls the FEF algorithm to determine the rough focus map of the grayscale image data corresponding to the original image data, that is, the second focus map, and controls the guided filter to perform filtering and refinement processing on the rough focus map, so as to determine the precise focus map of the grayscale image data.

[0131] Exemplarily, the terminal device integrated with the FEF algorithm calls the FEF algorithm to determine the rough focus maps YM1 and YM2 corresponding to the original image data RGB1 and RGB2, that is, RFM1 and RFM2 in the following formula. Taking the grayscale image data Y1 and Y2, that is, I1 and I2 in the following formula, as the guidance images, the high-frequency information of the rough focus maps is enhanced through the guided filter to obtain the precise focus maps AFM1 and AFM2, which have more high-frequency information than the rough focus maps, as shown in the equation:

[0132] AFM1(x, y) = GF1(I1(x, y), RFM1(x, y))

[0133] AFM2(x, y) = GF1(I2(x, y), RFM2(x, y))

[0134] Furthermore, in a feasible embodiment, for the multi-scale registration and fusion method of the image of the present invention, the above step E may include:

[0135] Step G, performing mean filtering on the grayscale image data to obtain mean-filtered image data;

[0136] In this embodiment, the terminal device integrated with the FEF algorithm calls the FEF algorithm to perform mean filtering on the grayscale image data corresponding to the original image data, thereby generating mean-filtered image data.

[0137] Exemplarily, simple mean filtering is used to blur the grayscale image data Y1, Y2, that is, I1 and I2 in the following formula, and generate the mean-filtered images M1 and M2, as shown in the following formula:

[0138] M1(x,y) = I1(x,y) * f m

[0139] M2(x,y) = I2(x,y) * f m

[0140] Among them, M1(x,y) represents the mean-filtered image corresponding to one piece of original image data to be fused, M2(x,y) represents the mean-filtered image corresponding to another piece of original image data to be fused, and f m represents the mean filter kernel.

[0141] Step H: Extract the high-frequency information from the mean-filtered image data to obtain a rough focus map of the grayscale image data.

[0142] In this embodiment, the terminal device integrated with the FEF algorithm calls the FEF algorithm to extract the high-frequency information from the mean-filtered image data, thereby obtaining a rough focus map of the grayscale image data.

[0143] Exemplarily, compared with the focused regions of the grayscale image data Y1, Y2, that is, I1, I2 in the following formula, the corresponding regions of the mean-filtered images M1, M2 are blurred. Calculate the absolute value of the difference between I1, I2 and the mean-filtered images M1, M2, and extract this part of the high-frequency information to generate the rough focus maps YM1, YM2, that is, RFM1 and RFM2 in the following formula, as shown in the following formula:

[0144] RFM1(x,y) = |I1(x,y) - M1(x,y)|

[0145] RFM2(x,y) = |I2(x,y) - M2(x,y)|

[0146] Exemplarily, as Figure 5As shown, the overall process of multi-focus fusion performed by the FEF algorithm called by the terminal device integrated with the FEF algorithm is as follows. First, the source RGB images RGB1 and RGB2 are determined, and the luminance images Y, that is, the corresponding grayscale images Y1 and Y2, extracted after YCrCb color space transformation are registered. Then, the registered grayscale images Y1 and Y2 are subjected to mean filtering to obtain the mean-filtered images M1 and M2. Then, rough detail maps YM1 and YM2 are obtained through a differential operator, and the rough detail maps YM1 and YM2 are filtered and refined using a guided filter to obtain precise detail maps GYM1 and GYM2. Then, an initial decision map DGYM is obtained through the pixel maximization rule, and the guided filter is used again to optimize and generate the final decision map Wmap. Finally, according to the final decision map Wmap, a fused image is obtained using the per-pixel weighted average rule.

[0147] In this embodiment, the multi-scale registration and fusion method of the images of the present invention generates mean-filtered image data by calling the FEF algorithm by the terminal device integrated with the FEF algorithm to perform mean filtering on the grayscale image data corresponding to the original image data; extracts high-frequency information in the mean-filtered image data by calling the FEF algorithm by the terminal device integrated with the FEF algorithm to determine the rough focus map of the grayscale image data; determines the rough focus map of the grayscale image data corresponding to the original image data by calling the FEF algorithm by the terminal device integrated with the FEF algorithm, and controls the guided filter to perform filtering and refinement processing on the rough focus map to determine the precise focus map of the grayscale image data; determines the precise focus map of the grayscale image data corresponding to the original image data by calling the FEF algorithm by the terminal device integrated with the FEF algorithm, and processes the precise focus map according to the per-pixel maximum rule to determine the initial decision map of the grayscale image data; determines the initial decision map of the grayscale image data corresponding to the original image data by calling the FEF algorithm by the terminal device integrated with the FEF algorithm, and controls the guided filter to perform filtering processing on the initial decision map to determine the multi-focus fusion weight coefficient of the grayscale image data.

[0148] In this way, a multi-focus fusion weight coefficient is obtained through a series of algorithmic processes on the original image data, and according to this multi-focus fusion weight coefficient, two images with different focuses are fused into an image in which all focused parts are clearly displayed, as Figure 7 shown. Multi-focus image1 and Multi-focus image1 have different focused parts, and fusedimage is an image in which the focused parts in Multi-focus image1 and Multi-focus image1 are clearly displayed.

[0149] Further, based on the first embodiment and / or the second embodiment of the multi-scale registration and fusion method of the images of the present invention described above, a third embodiment of the multi-scale registration and fusion method of the images of the present invention is proposed.

[0150] In this embodiment, for the multi-scale registration and fusion method of the images of the present invention, the above step S30 may include:

[0151] Step S301: Perform a weight calculation based on pixel intensity on the grayscale image data to determine a first weight coefficient.

[0152] In this embodiment, a terminal device integrated with the FEF algorithm calls the FEF algorithm to perform a weight calculation based on pixel intensity on the grayscale image data corresponding to the original image data, so as to determine the first weight coefficient of the grayscale image data based on pixel intensity.

[0153] Exemplarily, the traditional per-pixel multi-exposure fusion algorithm generates a fused image I fused which can be expressed as:

[0154]

[0155] where N is the number of images in a set of multiple-exposure images, In(x, y) is the pixel intensity of the nth image in the set, and Wn(x, y) represents the importance weight of the pixel in In(x, y).

[0156] The most important part of the multi-exposure fusion algorithm is how to design Wn(x, y). For pixels in well-exposed areas, it needs to be large, and vice versa. The following formula shows the simplest and most intuitive method for calculating Wn:

[0157]

[0158] Denote the mean pixel intensity of the nth image as Mn. It should be noted that the value range of Mn is from 0 to 1. When In(x, y) is close to 1 - Mn, the weight should be large, which can be expressed as:

[0159] exp(-(I n (x,y)-(1 - m n )) 2 )

[0160] Then the first weight coefficient W1map of the relative luminance encoding based on pixel intensity, that is, the first weight W 1,n (x,y) in the following formula is expressed as:

[0161]

[0162] where σ nControl the weight according to the difference in Mn, as shown in the following formula: α = 0.75. From the equation, we can see that when Mn is close to 1 (when the overall brightness of the image is high), darker pixels (pixels with lower median values in In(x, y)) will be assigned a larger weight, and vice versa. In addition, when the average brightness difference between adjacent exposure images is large, a large weight is assigned.

[0163]

[0164] Step S302: Perform weight calculation based on the global gradient on the grayscale image data to determine the second weight coefficient.

[0165] In this embodiment, the terminal device integrated with the FEF algorithm calls the FEF algorithm to perform weight calculation based on the global gradient on the grayscale image data corresponding to the original image data, so as to determine the second weight coefficient of the grayscale image data based on the global gradient.

[0166] Exemplarily, in a low-exposure image, pixels in the dark region are saturated to values close to 0, while pixel values in the bright region have large variations. Therefore, the bright region usually has a high contrast (large gradient of pixel values); in the case of a high-exposure image, the opposite relationship holds. The low-exposure image has many saturated pixels close to zero, and the cumulative histogram increases sharply at the beginning. Therefore, when a pixel is in the range where the cumulative histogram changes slowly, we can say that it is in a well-exposed region. In the case of medium-exposure and high-exposure images, the gradient of the cumulative histogram at lower pixel values is smaller than that of the low-exposure image. Therefore, when a pixel is in the range of the cumulative histogram with a smaller gradient, we need to assign a larger weight to this pixel:

[0167]

[0168] ε is a very small positive value to prevent the denominator from being zero, and Gradn(In(x, y)) represents the gradient of the cumulative histogram at the intensity In(x, y). Since the above gradient is not the local gradient around the pixel, but the gradient relative to other remote pixels in a similar range, we call it the global gradient.

[0169] Step S303: Determine the multi-exposure fusion weight coefficient of the grayscale image data according to the first weight coefficient and the second weight coefficient.

[0170] In this embodiment, the terminal device integrated with the FEF algorithm calls the FEF algorithm to determine the multi-exposure fusion weight coefficient of the grayscale image data according to the first weight coefficient based on pixel intensity and the second weight coefficient based on the global gradient.

[0171] Exemplarily, the final weight of each image is calculated by combining and normalizing two weights as follows:

[0172]

[0173] where p1, p2 > 0 are parameters that determine which one to emphasize more. However, in practice, these parameters are set to be the same (p1 = p2 = 1).

[0174] Exemplarily, as Figure 6 shown, the overall process of the FEF algorithm being called by a terminal device integrated with the FEF algorithm to perform multi-exposure fusion. First, the source RGB images RGB1 and RGB2 are determined. The luminance images Y extracted after YCrCb color space transformation, namely the corresponding grayscale images Y1 and Y2, are subjected to registration processing. Then, the registered grayscale images Y1 and Y2 are subjected to mean filtering to obtain the mean-filtered images M1 and M2. Then, the weight calculation based on pixel intensity and the weight calculation based on global gradient are performed on the mean-filtered images M1 and M2 to obtain the first weight coefficient W1map based on pixel intensity and the second weight coefficient W2map based on global gradient. W1map and W2map are combined and normalized to calculate the multi-exposure fusion weight coefficient Wexp.

[0175] In this embodiment, the multi-scale registration and fusion method of the images of the present invention determines the first weight coefficient based on pixel intensity of the grayscale image data corresponding to the original image data by calling the FEF algorithm by a terminal device integrated with the FEF algorithm to perform weight calculation based on pixel intensity on the grayscale image data corresponding to the original image data; determines the second weight coefficient based on global gradient of the grayscale image data by calling the FEF algorithm by a terminal device integrated with the FEF algorithm to perform weight calculation based on global gradient on the grayscale image data corresponding to the original image data; and determines the multi-exposure fusion weight coefficient of the grayscale image data by calling the FEF algorithm by a terminal device integrated with the FEF algorithm according to the first weight coefficient based on pixel intensity and the second weight coefficient based on global gradient.

[0176] In this way, the multi-exposure fusion weight coefficient is obtained through a series of algorithmic processes on the original image data. According to this multi-exposure fusion weight coefficient, two images with different exposure amounts are fused into a normally exposed image, as Figure 7 shown, Multi-exposure image1 is an underexposed image, Multi-exposure image2 is an overexposed image, and fused image is the normally exposed image obtained by fusing Multi-exposure image1 and Multi-exposure image2.

[0177] In addition, an embodiment of the present invention further provides a multi-scale registration and fusion device for images.

[0178] Please refer to Figure 8 , Figure 8 , which is a schematic diagram of the functional modules of an embodiment of the multi-scale registration and fusion device for images of the present invention. As Figure 8 shown, the multi-scale registration and fusion device for images of the present invention includes:

[0179] An image registration module 10, configured to perform image registration processing on the original image data to be fused, and obtain grayscale image data corresponding to the original image data;

[0180] A multi-focus weight module 20, configured to process the grayscale image data to obtain a multi-focus fusion weight coefficient of the grayscale image data;

[0181] A multi-exposure weight module 30, configured to perform weight calculation based on pixel intensity and weight calculation based on global gradient on the grayscale image data, and obtain a multi-exposure fusion weight coefficient of the grayscale image data;

[0182] An image fusion module 40, configured to perform multi-focus and multi-exposure image fusion processing on the original image data according to the multi-focus fusion weight coefficient and the multi-exposure fusion weight coefficient, and obtain fused image data.

[0183] Optionally, the multi-focus weight module 20 is further configured to determine an initial decision map of the grayscale image data; and process the initial decision map through a controlled guidance filter to obtain a multi-focus fusion weight coefficient of the grayscale image data.

[0184] Optionally, the multi-focus weight module 20 is further configured to determine a first focus map of the grayscale image data; and process the first focus map according to the per-pixel maximum rule to determine an initial decision map of the grayscale image data.

[0185] Optionally, the multi-focus weight module 20 is further configured to determine a second focus map of the grayscale image data; and refine the second focus map through a controlled guidance filter to determine a first focus map of the grayscale image data.

[0186] Optionally, the multi-focus weight module 20 is further configured to perform mean filtering on the grayscale image data to obtain mean-filtered image data; and extract high-frequency information in the mean-filtered image data to obtain a second focus map of the grayscale image data.

[0187] Optionally, the multi-exposure weight module 30 is further configured to calculate a weight based on pixel intensity for the grayscale image data to determine a first weight coefficient; calculate a weight based on the global gradient for the grayscale image data to determine a second weight coefficient; and determine a multi-exposure fusion weight coefficient for the grayscale image data according to the first weight coefficient and the second weight coefficient.

[0188] Optionally, the image fusion module 40 is further configured to perform Gaussian pyramid decomposition processing on the original image data to obtain multiple layers of pyramid images; and perform multi-focus multi-exposure image fusion processing on the multiple layers of pyramid images according to the multi-focus fusion weight coefficient and the multi-exposure fusion weight coefficient.

[0189] The present invention also provides a computer storage medium, on which a multi-scale registration and fusion program of an image is stored. When the multi-scale registration and fusion program of the image is executed by a processor, the steps of the multi-scale registration and fusion program method of the image described in any one of the above embodiments are implemented.

[0190] The specific embodiments of the computer storage medium of the present invention are basically the same as those of the above embodiments of the multi-scale registration and fusion program method of the image of the present invention, and will not be described in detail here.

[0191] The present invention also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the multi-scale registration and fusion method of the image of the present invention described in any one of the above embodiments are implemented, and will not be described in detail here.

[0192] It should be noted that in this article, the terms "include", "comprise" or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or system including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or system. Without further limitation, an element defined by the phrase "including a..." does not exclude the presence of additional identical elements in the process, method, article or system including the element.

[0193] The serial numbers of the above embodiments of the present invention are only for description and do not represent the merits of the embodiments.

[0194] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-described example methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium as described above (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions for causing a terminal device (which can be a TWS earphone, etc.) to execute the methods described in the various embodiments of the present invention.

[0195] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be similarly included in the patent protection scope of the present invention.

Claims

1. A multi-scale registration and fusion method for images, characterized in that, The multi-scale registration and fusion method of the image includes: Performing image registration processing on the original image data to be fused, and obtaining the grayscale image data corresponding to the original image data; Determining an initial decision map of the grayscale image data; Processing the initial decision map through a controlled guided filter to obtain the multi-focus fusion weight coefficients of the grayscale image data; Calculating weights based on pixel intensity for the grayscale image data to determine the first weight coefficient; Calculating weights based on the global gradient for the grayscale image data to determine the second weight coefficient; Determining the multi-exposure fusion weight coefficients of the grayscale image data according to the first weight coefficient and the second weight coefficient; Performing Gaussian pyramid decomposition processing on the original image data to obtain multi-layer pyramid images; Performing multi-focus and multi-exposure fusion processing on the multi-layer pyramid images according to the multi-focus fusion weight coefficients and the multi-exposure fusion weight coefficients, and then performing overall reconstruction to obtain the fused image data.

2. The multi-scale registration and fusion method of an image according to claim 1, characterized in that, The step of determining the initial decision map of the grayscale image data includes: Determining a first focus map of the grayscale image data; Processing the first focus map according to the per-pixel maximum rule to obtain the initial decision map of the grayscale image data.

3. The multi-scale registration and fusion method of an image according to claim 2, wherein The step of determining the first focus map of the grayscale image data includes: Determining a second focus map of the grayscale image data; Refining the second focus map through a controlled guided filter to obtain the first focus map of the grayscale image data.

4. The multi-scale registration and fusion method of an image according to claim 3, wherein, The step of determining the second focus map of the grayscale image data includes: Performing mean filtering on the grayscale image data to obtain mean-filtered image data; Extracting the high-frequency information in the mean-filtered image data to obtain the second focus map of the grayscale image data.

5. A multi-scale registration and fusion device for images, characterized in that, The multi-scale registration and fusion device of the image includes: An image registration module that performs image registration processing on the original image data to be fused and obtains the grayscale image data corresponding to the original image data; A multi-focus weight module that determines the initial decision map of the grayscale image data; processes the initial decision map through a controlled guided filter to obtain the multi-focus fusion weight coefficients of the grayscale image data; A multi-exposure weight module that calculates weights based on pixel intensity for the grayscale image data to determine the first weight coefficient; calculates weights based on the global gradient for the grayscale image data to determine the second weight coefficient; determines the multi-exposure fusion weight coefficients of the grayscale image data according to the first weight coefficient and the second weight coefficient; An image fusion module that performs Gaussian pyramid decomposition processing on the original image data to obtain multi-layer pyramid images; performs multi-focus and multi-exposure fusion processing on the multi-layer pyramid images according to the multi-focus fusion weight coefficients and the multi-exposure fusion weight coefficients, and then performs overall reconstruction to obtain the fused image data.

6. A terminal device, characterized in that, The terminal device includes: a memory, a processor, and a multi-scale registration and fusion program of images stored on the memory and executable on the processor. When the multi-scale registration and fusion program of images is executed by the processor, the steps of the multi-scale registration and fusion method of images according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium, characterized in that, A multi-scale registration and fusion program of images is stored on the computer-readable storage medium. When the multi-scale registration and fusion program of images is executed by a processor, the steps of the multi-scale registration and fusion method of images according to any one of claims 1 to 4 are implemented.

Images