Image fusion method, device, terminal equipment and storage medium

Abstract

The embodiment of the present application proposes an image fusion method, apparatus, terminal device and storage medium, which relates to the field of image processing technology, and achieves the purpose of ensuring more accurate image fusion with a method of lower complexity. The method includes: obtaining a reference image block and an image block to be fused; obtaining a deviation correction parameter for each pixel in the image block to be fused, and using the deviation correction parameter for each pixel to adjust the difference between each pixel in the image block to be fused and the pixel corresponding to the same scene position in the reference image block, to obtain the fusion weight of the image block to be fused corresponding to the adjusted difference; according to the fusion weight of the image block to be fused, the image block to be fused and the reference image block are fused.

Description

[Technical field]

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to an image fusion method, apparatus, terminal device, and storage medium. [Background technology]

[0002] In order to obtain images with richer details of the target scene, many imaging devices take multiple images of the target scene in a continuous period of time, and then fuse the multiple continuous images to suppress the noise of single-frame images and enhance the details of the image. The fused image with rich details in bright and dark areas and clear colors is obtained, which is of great significance.

[0003] If there are moving targets in the target scene and the position of the moving targets is different in each frame of the image; or if there is a registration deviation in the process of fusing multiple frames of images, ghost images or double images will appear in the fused image obtained by fusing multiple frames of images. [Summary of the invention]

[0004] The embodiments of the present application provide an image fusion method, apparatus, terminal device and storage medium, which use deviation correction parameters that affect the difference in pixel points of image blocks to adjust the fusion weights of image blocks, so that the image blocks are fused with the reference image with the adjusted fusion weights, thereby achieving a method with lower complexity to ensure more accurate image fusion.

[0005] In a first aspect, an embodiment of the present application provides an image fusion method, the method comprising:

[0006] Obtaining a reference image block and an image block to be fused;

[0007] Obtaining a deviation correction parameter for each pixel in the image block to be fused, and adjusting the difference between each pixel in the image block to be fused and a pixel corresponding to the same scene position in the reference image block using the deviation correction parameter of each pixel, to obtain a fusion weight of the image block to be fused corresponding to the adjusted difference;

[0008] The image block to be fused and the reference image block are fused according to the fusion weight of the image block to be fused.

[0009] In the above-mentioned image fusion method, a deviation correction parameter that affects the difference in pixel points of the image block to be fused is obtained, and the difference between the pixel points of the image block to be fused and the pixel points of the reference image block is adjusted by using the deviation correction parameter to restore the real difference between the pixel points of the image block to be fused and the pixel points of the reference image block. According to the real difference between the pixel points of the image block to be fused and the pixel points of the reference image block, a fusion weight that conforms to the real difference between the pixel points of the image block and the reference image block is set for the image block, so that the image fusion will not be affected by the registration deviation and the moving target, thereby ensuring that the image fusion of the pixel points of the image block to be fused and the pixel points of the reference image block is more accurate, and solving the problem of ghosting in the fused image.

[0010] In one possible implementation manner, obtaining the deviation correction parameter of each pixel in the image block to be fused includes:

[0011] Obtaining an alignment deviation correction parameter and a pixel brightness correction parameter for each pixel in the image block to be fused;

[0012] The larger one between the alignment deviation correction parameter and the pixel brightness correction parameter is taken as the deviation correction parameter.

[0013] In one possible implementation, the deviation correction parameter includes an alignment deviation correction parameter, and obtaining the deviation correction parameter of each pixel in the image block to be fused includes:

[0014] Obtaining an alignment deviation correction parameter for each pixel in the image block to be fused according to an alignment deviation occurring during the alignment process between the reference image block and the image block to be fused;

[0015] The step of adjusting the difference between each pixel in the image block to be fused and a pixel corresponding to the same scene position in the reference image block by using the deviation correction parameter of each pixel includes:

[0016] The alignment deviation correction parameter of each pixel is used to adjust the difference between each pixel in the image block to be fused and the pixel corresponding to the same scene position in the reference image block.

[0017] In one possible implementation, the image block to be fused is an image block in an image frame to be fused, image blocks near the image block to be fused and the image block to be fused form an image region, and according to the alignment deviation between the reference image block and the image block to be fused during the alignment process, an alignment deviation correction parameter for each pixel in the image block to be fused is obtained, including:

[0018] Calculating the alignment vector difference strength of two image blocks with the largest alignment vector difference in the image region where the image block to be fused is located; wherein the alignment vector difference strength represents the overall alignment accuracy of all image blocks in the image region;

[0019] The average value of the alignment vector difference strengths of all image regions corresponding to the image block to be fused is substituted into a preset alignment curve to obtain an alignment deviation correction parameter for each pixel in the image block to be fused.

[0020] In one possible implementation, the deviation correction parameter includes a pixel brightness correction parameter, and obtaining the deviation correction parameter of each pixel in the image block to be fused includes:

[0021] Obtaining the brightness information value of each pixel in the image block to be fused;

[0022] Substituting the brightness information value of each pixel into a brightness curve pre-set according to the scene sensitivity value, and obtaining the pixel brightness correction parameter of each pixel in the image block to be fused;

[0023] The step of adjusting the difference between each pixel in the image block to be fused and a pixel corresponding to the same scene position in the reference image block by using the deviation correction parameter of each pixel includes:

[0024] The pixel brightness correction parameter of each pixel is used to adjust the difference between each pixel in the image block to be fused and the pixel corresponding to the same scene position in the reference image block.

[0025] In one possible implementation, the method further includes:

[0026] Calculating the difference between the average value of pixels in the image block to be fused and the average value of pixels in the reference image block to obtain the pixel mean difference of the image block to be fused;

[0027] According to the scene brightness information value, setting the image block brightness correction parameter of the image block to be fused;

[0028] Using a preset image block brightness correction parameter, adjusting the pixel mean difference of the image block to be fused, to obtain a similarity weight of the image block to be fused, wherein the image block brightness correction parameter is inversely proportional to the scene brightness information value;

[0029] Using the image block similarity weight, adjusting the fusion weight;

[0030] According to the fusion weight of the image block to be fused, the image block to be fused and the reference image block are fused, including:

[0031] The image block to be fused and the reference image block are fused according to the adjusted fusion weight of the image block to be fused.

[0032] In one possible implementation, the image region where the image block to be fused is located includes a horizontal region and a vertical region; and calculating the alignment vector difference strength of two image blocks with the largest alignment vector difference in the image region where the image block to be fused is located includes:

[0033] Sorting the horizontal displacements of all the image blocks in the horizontal region to obtain an image block with a maximum horizontal displacement and an image block with a minimum horizontal displacement in the horizontal region;

[0034] sorting the vertical displacements of all the image blocks in the vertical region to obtain an image block with a maximum vertical displacement and an image block with a minimum vertical displacement in the vertical region;

[0035] Calculating the difference between the maximum horizontal displacement and the minimum horizontal displacement to obtain the alignment vector difference strength of the horizontal area;

[0036] Calculating the difference between the maximum vertical displacement and the minimum vertical displacement to obtain the alignment vector difference strength of the vertical area;

[0037] Substituting the average value of the alignment vector difference strength of all image regions corresponding to the image block to be fused into a preset alignment curve to obtain the alignment deviation correction parameter of each pixel in the image block to be fused, including:

[0038] Taking a square root of the sum of the square of the alignment vector difference strength of the horizontal region and the square of the alignment vector difference strength of the vertical region to obtain an average value of the alignment vector difference strength of the horizontal region and the alignment vector of the vertical region;

[0039] The alignment vector difference strength of the horizontal region and the average value of the alignment vector of the vertical region are brought into a preset alignment curve to obtain an alignment deviation correction parameter for each pixel in the image block to be fused.

[0040] In one possible implementation, the method further includes:

[0041] Obtaining a difference matrix composed of differences between each pixel in the image block to be fused and a pixel corresponding to the same scene position in the reference image block;

[0042] Using the deviation correction parameter of each pixel point, adjusting the difference between each pixel point in the image block to be fused and the pixel point corresponding to the same scene position in the reference image block, and obtaining the fusion weight of the image block to be fused corresponding to the adjusted difference value, including:

[0043] Substitute the deviation correction parameter into formula (1) to adjust the difference matrix;

[0044] Among them, d pix represents the difference matrix, sigma pix represents the deviation correction parameter, Indicates the fusion weight of the image block to be fused corresponding to the adjusted difference.

[0045] In one possible implementation, using a preset image block brightness correction parameter to adjust the pixel mean difference of the image block to be fused, and obtaining the similarity weight of the image block to be fused, includes:

[0046] Substitute the image block brightness correction parameter into formula (2) to adjust the pixel mean difference of the image block to be fused;

[0047] Among them, d blk Represents the pixel mean difference, sigma blk represents the brightness correction parameter of the image block, Represents the similarity weight of the image block to be fused.

[0048] In a second aspect, an embodiment of the present application provides an image fusion device, the device comprising:

[0049] An acquisition module, used for acquiring a reference image block and an image block to be fused;

[0050] A first adjustment module is used to obtain a deviation correction parameter for each pixel in the image block to be fused, and to adjust the difference between each pixel in the image block to be fused and a pixel corresponding to the same scene position in the reference image block using the deviation correction parameter of each pixel, so as to obtain a fusion weight of the image block to be fused corresponding to the adjusted difference;

[0051] A fusion module is used to fuse the image block to be fused and the reference image block according to the fusion weight of the image block to be fused.

[0052] In one possible implementation manner, the first adjustment module includes:

[0053] A first correction parameter acquisition submodule is used to obtain an alignment deviation correction parameter and a pixel brightness correction parameter for each pixel in the image block to be fused;

[0054] The second correction parameter obtaining submodule takes the larger one between the alignment deviation correction parameter and the pixel brightness correction parameter as the deviation correction parameter.

[0055] In one possible implementation, the deviation correction parameter includes an alignment deviation correction parameter, and the first adjustment module includes:

[0056] A third correction parameter obtaining submodule, obtaining an alignment deviation correction parameter of each pixel in the image block to be fused according to an alignment deviation occurring during the alignment process between the reference image block and the image block to be fused;

[0057] The first adjustment submodule is used to adjust the difference between each pixel in the image block to be fused and the pixel corresponding to the same scene position in the reference image block by using the alignment deviation correction parameter of each pixel.

[0058] In one possible implementation, the image block to be fused is an image block in an image frame to be fused, image blocks near the image block to be fused and the image block to be fused form an image region, and the third correction parameter obtaining submodule includes:

[0059] A calculation subunit, configured to calculate the alignment vector difference strength of two image blocks with the largest alignment vector difference in the image region where the image block to be fused is located; wherein the alignment vector difference strength represents the overall alignment accuracy of all image blocks in the image region;

[0060] The sub-unit is used to substitute the average value of the alignment vector difference strength of all image regions corresponding to the image block to be fused into a preset alignment curve to obtain the alignment deviation correction parameter of each pixel in the image block to be fused.

[0061] In one possible implementation, the deviation correction parameter includes a pixel brightness correction parameter, and the first adjustment module includes:

[0062] A brightness information value obtaining submodule is used to obtain the brightness information value of each pixel in the image block to be fused;

[0063] The input submodule is used to input the brightness information value of each pixel into a brightness curve preset according to the scene sensitivity value to obtain the pixel brightness correction parameter of each pixel in the image block to be fused;

[0064] The second adjustment submodule uses the pixel brightness correction parameter of each pixel to adjust the difference between each pixel in the image block to be fused and the pixel corresponding to the same scene position in the reference image block.

[0065] In one possible implementation, the device further includes:

[0066] A calculation module, used for calculating the difference between the average value of pixels in the image block to be fused and the average value of pixels in the reference image block, to obtain the pixel mean difference of the image block to be fused;

[0067] A setting module, used for setting the image block brightness correction parameter of the image block to be fused according to the scene brightness information value;

[0068] The second adjustment module adjusts the pixel mean difference of the image block to be fused by using a preset image block brightness correction parameter to obtain a similarity weight of the image block to be fused, wherein the image block brightness correction parameter is inversely proportional to the scene brightness information value;

[0069] A third adjustment module, used to adjust the fusion weight using the image block similarity weight;

[0070] The fusion module includes:

[0071] The fusion submodule is used to fuse the image block to be fused and the reference image block according to the adjusted fusion weight of the image block to be fused.

[0072] In one possible implementation, the image region where the image block to be fused is located includes a horizontal region and a vertical region; and the calculation subunit includes:

[0073] A first sorting subunit is used to sort the horizontal displacements of all the image blocks in the horizontal region to obtain the image block with the maximum horizontal displacement and the image block with the minimum horizontal displacement in the horizontal region;

[0074] A second sorting subunit is used to sort the vertical displacements of all the image blocks in the vertical region to obtain an image block with a maximum vertical displacement and an image block with a minimum vertical displacement in the vertical region;

[0075] a first alignment vector difference strength obtaining subunit, configured to calculate a difference between the maximum horizontal displacement and the minimum horizontal displacement to obtain an alignment vector difference strength of the horizontal region;

[0076] a second alignment vector difference strength obtaining subunit, configured to calculate a difference between the maximum vertical displacement and the minimum vertical displacement, and obtain an alignment vector difference strength of the vertical region;

[0077] The brought-in subunit comprises:

[0078] an average value obtaining subunit, configured to take a square root of the sum of the square of the alignment vector difference strength of the horizontal region and the square of the alignment vector difference strength of the vertical region, to obtain an average value of the alignment vector difference strength of the horizontal region and the alignment vector of the vertical region;

[0079] The alignment deviation correction parameter obtaining subunit is used to bring the alignment vector difference strength of the horizontal area and the average value of the alignment vector of the vertical area into a preset alignment curve to obtain the alignment deviation correction parameter of each pixel in the image block to be fused.

[0080] In one possible implementation, the device further includes:

[0081] A difference matrix obtaining module, used to obtain a difference matrix composed of the difference between each pixel in the image block to be fused and the pixel corresponding to the same scene position in the reference image block;

[0082] The first adjustment module includes:

[0083] A third adjustment submodule, used for bringing the deviation correction parameter into formula (1) to adjust the difference matrix;

[0084] Among them, d pix represents the difference matrix, sigma pix represents the deviation correction parameter, Indicates the fusion weight of the image block to be fused corresponding to the adjusted difference.

[0085] In one possible implementation, the second adjustment module includes:

[0086] A fourth adjustment submodule, used for substituting the image block brightness correction parameter into equation (2) to adjust the pixel mean difference of the image block to be fused;

[0087] Among them, d blk Represents the pixel mean difference, sigma blk represents the brightness correction parameter of the image block, Represents the similarity weight of the image block to be fused.

[0088] In a third aspect, an embodiment of the present application provides a terminal device, comprising: at least one processor; and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the method provided in the first aspect.

[0089] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the method provided in the first aspect.

[0090] It should be understood that the second to fourth aspects of the embodiments of the present application are consistent with the technical solutions of the first aspect of the embodiments of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here.

Brief Description of the Drawings

[0091] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0092] Figure 1 It is a flowchart of multi-frame image fusion;

[0093] Figure 2 is a flowchart of the steps of the image fusion method proposed in an embodiment of the present invention;

[0094] Figure 3 is a schematic diagram of an image area in one embodiment of the present invention;

[0095] Figure 4 is another schematic diagram of an image area in yet another embodiment of the present invention;

[0096] Figure 5 is a schematic diagram of a preset alignment curve in an example of the present invention;

[0097] Figure 6 is a schematic diagram of a brightness curve preset in an example of the present invention;

[0098] Figure 7 is a flowchart of an exemplary method for performing image fusion according to the present invention;

[0099] Figure 8 is a functional module diagram of an image fusion device proposed in an embodiment of the present invention;

[0100] Fig. 9 A schematic diagram of the structure of an electronic terminal device provided by an embodiment of the present invention;

[0101] Fig.10 A schematic diagram of the hardware structure of an electronic terminal device provided in an embodiment of the present invention. [Specific implementation method]

[0102] In order to better understand the technical solution of this specification, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0103] It should be clear that the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0104] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit this specification. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0105] Multi-frame image fusion may refer to fusing multiple frames of images captured for the same target scene into a fused image with rich bright and dark details and clear colors. The multi-frame images fused in the embodiment of the present invention are multiple frames of images captured by the same imaging device at continuous time.

[0106] Figure 1 It is a flowchart of multi-frame image fusion, such as Figure 1 As shown;

[0107] For multi-frame image fusion, the collected multi-frame images are first preprocessed. Image preprocessing includes: image denoising, image color correction, image bad pixel removal, and invalid frame elimination. The specific processing can be performed according to the situation of the imaging system itself.

[0108] Then perform image segmentation on multiple frames, dividing each frame into uniform and equal square blocks to prevent geometric deformation such as rotation or stretching of the image. After the image is divided into relatively small blocks, when aligning or fusion is performed in blocks, it can be assumed that there is only translation in different directions between the images. Subsequent alignment and fusion operations can be performed in units of the segmented image blocks. The size of the block is generally set according to the original size of the image.

[0109] Select one frame from multiple frames as a reference frame. Generally, the first frame captured by the imaging device is selected as the reference frame by default. You can also select the frame with the best quality as the reference frame according to the image quality of different frames. The other frames in the multiple frames except the reference frame can be used as the image frames to be fused. The image blocks obtained by segmenting the reference frame can be used as the reference image blocks, and the image blocks obtained by segmenting the image frame to be fused can be used as the image blocks to be fused.

[0110] After obtaining the reference image block and the image blocks to be fused corresponding to different image frames, the reference image block and the image block to be fused are aligned. Based on the selected reference frame, the images of the remaining frames are aligned. The registration and alignment methods can be based on ORB features, optical flow method, multi-scale alignment algorithm, etc. After the alignment operation, for each reference image block, an image block matching it can be found in each of the remaining frames. For example, the target scene cat is segmented to obtain image frame 1, image frame 2 and image frame 3. Image frame 1 is used as the reference image frame, and the reference image frame is segmented to obtain reference image block 1, reference image block 2 and reference image block 3. Image frame 2 is segmented to obtain image block 1 to be fused, image block 2 to be fused and image block 3 to be fused. Image frame 3 is segmented to obtain image block 4 to be fused, image block 5 to be fused and image block 6 to be fused. Reference image block 1 corresponds to cat ears, and image block 1 to be fused and image block 5 to be fused correspond to cat ears. Then image block 1 to be fused and image block 5 to be fused are image blocks in image frame 2 and image frame 3 that match reference image block 1, respectively.

[0111] Finally, the reference image block after alignment and registration and the image block to be fused are fused to obtain a fused image. The prior art also sets different fusion weights for different image blocks to be fused based on the difference between the image block to be fused and the reference image block, for example, based on the different features of the image block to be fused and the reference image block, so as to select and retain the information of the image block that is more similar or identical to the reference image block during the image fusion process.

[0112] However, if the target scene includes moving targets, or the alignment of the image blocks deviates, ghosting or double images will appear in the fused image obtained by fusing multiple frames of images. To solve the above problems, the following solutions are proposed in related technologies: detecting moving areas in the image fusion process, processing moving areas separately, estimating mutual noise between different images and the reference image, or using similar redundant information of multiple frames of images to perform weighted averaging at similar pixel locations in multiple frames of images. However, the above methods involve complex iterative operations, have high computational complexity, and increase the power consumption and time cost of device operation.

[0113] The embodiments of the present invention find that no matter whether ghosting appears in the fused image due to a moving target or due to alignment and registration, the fundamental reason is that the pixel points corresponding to different target points in the reference image block and the image block to be fused are regarded as the pixel points corresponding to the same target point, resulting in errors in the similarity calculation between the pixel points respectively located in the reference image block and the image block to be fused; or errors in the similarity calculation between the pixel points corresponding to the same target point in the reference image block and the image block to be fused, resulting in unreasonable setting of the fusion weights of the image blocks to be fused.

[0114] Based on the above findings, the embodiment of the present invention starts from the root cause of ghosting, obtains a deviation correction parameter for each pixel, adjusts the difference between each pixel in the image block to be fused and the pixel corresponding to the same scene position in the reference image block, corrects the similarity error between the current pixel in the image block to be fused and the pixel corresponding to the same scene position as the current pixel in the reference image block, obtains the true similarity between the current pixel in the image block to be fused and the pixel corresponding to the same scene position as the current pixel in the reference image block, ensures that the fusion weight of the image block to be fused is obtained based on the true similarity between each pixel in the image block to be fused and the pixel corresponding to the same scene position in the reference image block, thereby solving the problem of ghosting in the fused image.

[0115] The image fusion method proposed in the embodiment of the present invention is applied to an electronic device terminal, which may be a computer, a computer cluster, a mobile phone, or a server, etc. Figure 2 is a flowchart of the steps of the image fusion method proposed in an embodiment of the present invention. Figure 2 As shown, the image fusion steps are as follows:

[0116] Step S110: Obtain a reference image block and an image block to be fused.

[0117] The reference image block is the current image block in the reference frame for image fusion operation. The image block to be fused is the image block in the image frame to be fused that matches the reference image block. There is only one image block in an image frame to be fused that matches the reference image block. There can be multiple image frames to be fused, and the number of image blocks to be fused is the same as the number of image frames to be fused.

[0118] Step S111: Obtain the deviation correction parameter of each pixel in the image block to be fused, and use the deviation correction parameter of each pixel to adjust the difference between each pixel in the image block to be fused and the pixel corresponding to the same scene position in the reference image block, so as to obtain the fusion weight of the image block to be fused corresponding to the adjusted difference.

[0119] Pixel A in the image block to be fused and pixel B in the reference image block correspond to point C at the same scene position. The chrominance (UV) difference value of pixel A and pixel B can be calculated to obtain the difference between each pixel in the image block to be fused and the pixel corresponding to the same scene position in the reference image block. The three primary colors (RGB) difference value of pixel A and pixel B can also be calculated to obtain the difference between each pixel in the image block to be fused and the pixel corresponding to the same scene position in the reference image block.

[0120] The alignment and registration process can be that for each reference image block, a matching image block to be fused can be found in each of the remaining frames of the image, or that for each pixel point of each reference image block, a matching pixel point can be found in the image block to be fused that matches it. For example, image block 1 to be fused and reference image block 1 correspond to the cat ear shooting target, image block 1 to be fused is the image block that matches reference image block 1, pixel point A in image block 1 to be fused and pixel point B in reference image block 1 correspond to position point C of the cat ear shooting target, then pixel point A in image block 1 to be fused is a pixel point that matches pixel point B in reference image block 1.

[0121] The factors affecting the difference between the pixel points in the image block to be fused and the pixel points in the reference image block to which it matches can be calculated to obtain the deviation correction parameters of the pixel points. For example, the deviation correction parameters of the pixel point A in the image block to be fused are obtained based on the factors that cause the similarity between the pixel point A and the pixel point B in the reference image block that matches the pixel point A to be enlarged, reduced or deviated.

[0122] The embodiment of the present invention proposes an optional method of using the deviation correction parameter of each pixel to adjust the difference between each pixel in the image block to be fused and the pixel corresponding to the same scene position in the reference image block.

[0123] Step S111 - 1 : obtaining a difference matrix composed of the difference between each pixel in the image block to be fused and the pixel corresponding to the same scene position in the reference image block.

[0124] In an example of the present application, the image block to be fused consists of pixel 1, pixel 2, and pixel 3, and the reference image block consists of pixel 4, pixel 5, and pixel 6. The difference matrix obtained includes elements: the difference between pixel 1 and pixel 4, the difference between pixel 2 and pixel 5, and the difference between pixel 3 and pixel 6.

[0125] Step S111 - 2: Substitute the deviation correction parameter into formula (1) to adjust the difference matrix.

[0126] Among them, d pix represents the difference matrix, sigma pix represents the deviation correction parameter, Indicates the fusion weight of the image block to be fused corresponding to the adjusted difference.

[0127] Step S112: fusing the image block to be fused and the reference image block according to the fusion weight of the image block to be fused.

[0128] The above steps S110 to S112 may be executed in parallel for the image blocks to be fused in different image frames to be fused.

[0129] The embodiment of the present invention obtains a deviation correction parameter that affects the similarity between the pixel points of the image block to be fused and the pixel points of the reference image block, uses the deviation correction parameter to adjust the difference between the pixel points of the image block to be fused and the pixel points of the reference image block, restores the real difference between the pixel points of the image block to be fused and the pixel points of the reference image block, and sets a fusion weight that conforms to the real difference between the pixel points of the image block and the pixel points of the reference image block according to the real difference between the pixel points of the image block to be fused and the pixel points of the reference image block, so that the image fusion will not be affected by the registration deviation and the moving target, thereby ensuring that the image fusion of the pixel points of the image block to be fused and the pixel points of the reference image block is more accurate, and solving the problem of ghosting in the fused image.

[0130] Another embodiment of the present invention finds that factors affecting the difference between the pixel points in the image block to be fused and the pixel points in the reference image block to be matched include image brightness and the alignment accuracy of the image block to be fused and the reference image block during the alignment and registration process. Therefore, an alignment deviation correction parameter and a pixel brightness correction parameter are obtained, and the deviation correction parameter is determined based on the comparison result of the alignment deviation correction parameter and the pixel brightness correction parameter.

[0131] Another embodiment of the present invention provides an image fusion method comprising the following steps:

[0132] Step S120: Obtain a reference image block and an image block to be fused.

[0133] Step S121: obtaining an alignment deviation correction parameter and a pixel brightness correction parameter for each pixel in the image block to be fused.

[0134] During the alignment and registration process of the image block to be fused and the reference image block, alignment deviation may occur. For example, during the alignment and registration process, the electronic device terminal mistakenly aligns the pixel point A corresponding to the scene position 1 in the image block to be fused with the pixel point A' corresponding to the scene position 2 in the reference image block, and calculates the difference between the pixel point A and the pixel point A'. However, the difference between the pixel point A and the pixel point A' cannot represent the similarity between the image block to be fused and the reference image block showing the same scene position point. Therefore, an alignment deviation correction parameter can be obtained as a deviation correction parameter to adjust the difference between each pixel point in the image block to be fused and the pixel point corresponding to the same scene position in the reference image block.

[0135] The brightness of different pixels corresponding to the same scene position in different image blocks to be fused is different, and the brightness will affect the difference between two different pixels. For example, when calculating the difference of pixels based on color coding YUV, Y in YUV represents the brightness of the pixel (Luminance), and UV represents the chrominance of the pixel (Chrominance). The brightness value of two pixels will affect the chrominance difference value of the two pixels. A larger brightness may amplify the chrominance difference value of the two pixels, and a smaller difference value may reduce the chrominance difference value of the two pixels. Therefore, a pixel brightness correction parameter can be obtained as a deviation correction parameter to adjust the difference between each pixel in the image block to be fused and the pixel corresponding to the same scene position in the reference image block, so as to correct the difference between the pixel in the image block to be fused and the pixel in the reference image block after enlargement or reduction caused by brightness to the difference between the pixel in the real image block to be fused and the pixel in the reference image block.

[0136] Step S122: taking the larger one between the alignment deviation correction parameter and the pixel brightness correction parameter as the deviation correction parameter.

[0137] According to other embodiments of the present application, the difference between each pixel in the image block to be fused and the pixel corresponding to the same scene position in the reference image block is adjusted, and the calculation method of the fusion weight of the image block to be fused corresponding to the adjusted difference is known. The larger the difference between the pixel in the image block to be fused and the pixel corresponding to the same scene position in the reference image block, the smaller the fusion weight; the larger the deviation correction parameter, the larger the adjusted difference, and the smaller the fusion weight of the image block to be fused. Therefore, the larger of the alignment deviation correction parameter and the pixel brightness correction parameter is taken as the deviation correction parameter to reduce the weight of the image block to be fused. Without introducing complex calculations, the pixel similarity calculation error caused by the alignment deviation or the pixel similarity calculation error caused by the pixel brightness can be directly eliminated to ensure that dissimilar image information will not be superimposed on the reference image block.

[0138] Step S123: Obtain the deviation correction parameter of each pixel in the image block to be fused, and use the deviation correction parameter of each pixel to adjust the difference between each pixel in the image block to be fused and the pixel corresponding to the same scene position in the reference image block, so as to obtain the fusion weight of the image block to be fused corresponding to the adjusted difference.

[0139] Step S124: fusing the image block to be fused and the reference image block according to the fusion weight of the image block to be fused.

[0140] Another embodiment of the present invention proposes an optional method of obtaining an alignment deviation correction parameter as a deviation correction parameter to adjust the difference between each pixel in the image block to be fused and the pixel corresponding to the same scene position in the reference image block. The deviation correction parameter includes an alignment deviation correction parameter, and obtaining the deviation correction parameter for each pixel in the image block to be fused includes:

[0141] Step S210: obtaining an alignment deviation correction parameter for each pixel in the image block to be fused according to an alignment deviation occurring during the alignment process between the reference image block and the image block to be fused.

[0142] The embodiment of the present invention can obtain the alignment deviation correction parameter based on the alignment deviation between the image block to be fused and the reference image block obtained directly in the alignment and registration process, without the need for complex calculations.

[0143] Step S220: using the alignment deviation correction parameter of each pixel point to adjust the difference between each pixel point in the image block to be fused and the pixel point corresponding to the same scene position in the reference image block.

[0144] By adjusting the difference between each pixel in the image block to be fused and the pixel corresponding to the same scene position in the reference image block with the alignment deviation correction parameter, the difference calculation error caused by the matching error can be eliminated when the pixel of the image to be fused and the pixel in the reference image block have a matching error. For example, pixel A of the image block to be fused corresponds to scene position A, and pixel B of the reference image block corresponds to scene position B. During the alignment and registration process of the image block to be fused and the reference image block, an alignment deviation occurs, and pixel A is incorrectly matched with pixel B. Regardless of the similarity between pixel A and pixel B, and whether the difference between pixel A and pixel B is large, the information of pixel A cannot be superimposed on pixel B. Therefore, when obtaining the deviation of matching pixel A with pixel B during the alignment and registration process of the image block to be fused and the reference image block, the difference between pixel A and pixel B is adjusted with the deviation, and the image block to be fused can obtain a sufficiently small fusion weight, thereby ensuring that the information of pixel A will not be superimposed on pixel B.

[0145] Another embodiment of the present invention proposes an optional method for obtaining alignment deviation correction parameters. Figure 3 Schematic diagram of an image area in one embodiment of the present invention. Figure 3 As shown, the image frame 32 to be fused is divided into a plurality of image blocks during the image segmentation process. The image block 31 to be fused is an image block in the image frame 32 to be fused. The image blocks near the image block to be fused and the image block to be fused form an image region 33. The domain range of the image block to be fused can be preset. Figure 3 The area is set to 3×3. Figure 3 As shown in the example, the image blocks near the image block to be fused are image blocks within a 3×3 domain. The image blocks near the image block 32 to be fused include image blocks 34, 35, 36, 37, 38, 39, 40, and 41.

[0146] The steps to obtain the alignment deviation correction parameters are as follows:

[0147] Step S310: calculating the alignment vector difference strength of two image blocks with the largest alignment vector difference in the image region where the image block to be fused is located; wherein the alignment vector difference strength represents the overall alignment accuracy of all image blocks in the image region.

[0148] After an image is aligned with another image, if a partial image of one image is accurately aligned with a partial image corresponding to the same target object in another image, the alignment vectors of all image blocks in the partial image of one image change smoothly without significant differences. Based on the above findings, the embodiment of the present invention uses the difference in alignment vectors of two image blocks with the largest difference in alignment vectors in an image region as the intensity of the difference in alignment vectors that characterizes the image region, thereby using the difference in alignment vectors of two image blocks with the largest difference in alignment vectors in an image region to characterize the overall alignment accuracy of all image blocks in the image region.

[0149] The number of the image regions where the image blocks to be fused are located may be one or more, which are automatically generated according to the received instructions. Figure 4 is another schematic diagram of an image area in another embodiment of the present invention. After receiving the parameters 3×3, horizontal division, and vertical division input by the user, the electronic device terminal generates the following Figure 4 The image range shown is a 3×3 horizontal area 41 and an image range is a 3×3 vertical area 42 . The horizontal area 41 and the vertical area 42 are both image areas where the image block 43 to be fused is located.

[0150] In another embodiment of the present invention, the image region where the image block to be fused is located includes a horizontal region and a vertical region, and an optional method for calculating the alignment vector difference strength of two image blocks with the largest alignment vector difference in the image region where the image block to be fused is located is proposed, and the steps are as follows:

[0151] Step S310 - 1 : sorting the horizontal displacements of all the image blocks in the horizontal region to obtain the image block with the maximum horizontal displacement and the image block with the minimum horizontal displacement in the horizontal region.

[0152] The horizontal displacement of an image block is the horizontal component of the alignment vector during the image block alignment process.

[0153] After sorting the horizontal displacements of all image blocks in the horizontal region, an image block i with the largest horizontal displacement in the horizontal region with an image range of 3×3 and an image block j with the smallest horizontal displacement in the horizontal region with an image range of 3×3 are obtained.

[0154] Step S310 - 2 : sorting the vertical displacements of all the image blocks in the vertical region to obtain the image block with the maximum vertical displacement and the image block with the minimum vertical displacement in the vertical region.

[0155] The vertical displacement of the image block is the vertical component of the alignment vector in the image block alignment process. After sorting the vertical displacements of all image blocks in the horizontal region, the image block p with the largest vertical displacement in the horizontal region with an image range of 3×3 and the image block g with the smallest horizontal displacement in the horizontal region with an image range of 3×3 are obtained.

[0156] Step S310 - 3 : Calculate the difference between the maximum horizontal displacement and the minimum horizontal displacement to obtain the alignment vector difference strength of the horizontal area.

[0157] The difference between the maximum horizontal displacement and the minimum horizontal displacement is calculated using formula (3):

[0158] Among them, M x is the alignment vector difference strength of the horizontal region, υ x (i) is the horizontal component of the alignment vector of the image block i with the maximum horizontal displacement in the horizontal region of the image range 3×3, and the elements of the set N3 include all the image blocks in the horizontal region of the image range 3×3; υ x (j) is the horizontal component of the alignment vector of image block j with the minimum horizontal displacement in the horizontal region of the image range 3×3.

[0159] Step S310 - 4 : Calculate the difference between the maximum vertical displacement and the minimum vertical displacement to obtain the alignment vector difference strength of the vertical area.

[0160] The difference between the maximum vertical displacement and the minimum vertical displacement is calculated using formula (4):

[0161] Among them, M y is the alignment vector difference strength in the vertical region, υ y (p) is the vertical component of the alignment vector of the image block p with the maximum vertical displacement in the vertical region of the image range of 3×3, and the elements of the set N3 include all the image blocks in the vertical region of the image range of 3×3; υ y (g) is the vertical component of the alignment vector of the image block g with the minimum vertical displacement in the vertical region of the image range 3×3.

[0162] Step S310 - 5 : taking the square root of the sum of the square of the alignment vector difference strength of the horizontal region and the square of the alignment vector difference strength of the vertical region to obtain an average value of the alignment vector difference strength of the horizontal region and the alignment vector of the vertical region.

[0163] The average of the alignment vector difference strength of the horizontal region and the alignment vector of the vertical region can be used as: the average of the alignment vector difference strength of all image regions corresponding to the image block to be fused. The average of the alignment vector difference strength of the horizontal region and the alignment vector of the vertical region can also be used as the overall alignment vector difference strength of the image block to be fused relative to all image regions.

[0164] Formula (5) is used to take the square root of the sum of the square of the alignment vector difference strength of the horizontal region and the square of the alignment vector difference strength of the vertical region to obtain the average value of the alignment vector difference strength of the horizontal region and the alignment vector of the vertical region:

[0165] Wherein, M represents the overall alignment vector difference strength of all image regions where the image blocks to be fused are located.

[0166] Step S320: Substitute the alignment vector difference strength of the horizontal region and the average value of the alignment vector of the vertical region into a preset alignment curve to obtain an alignment deviation correction parameter for each pixel in the image block to be fused.

[0167] Figure 5 It is a schematic diagram of a preset alignment curve in an example of the present invention. Figure 5 The horizontal axis is M, the vertical axis is the alignment deviation correction parameter sigma2, and v1, v2, and v3 are specific values ​​of sigma2. Figure 5 Where thr1 represents the minimum alignment deviation that needs to be corrected, thr4 represents the maximum alignment deviation that can be corrected; thr2 and thr3 are node values ​​set according to experience, and multiple node values ​​such as thr2 and thr3 can be set according to image resolution, image contrast, etc.

[0168] The alignment vector difference strength of the horizontal region where the image block to be fused is located and the average value M of the alignment vector of the vertical region are brought into Figure 5 The alignment curve shown is used to obtain the alignment deviation correction parameter sigma2 of the image block to be fused, and the alignment deviation correction parameter sigma2 is used as the alignment deviation correction parameter of all pixels in the image block to be fused.

[0169] The embodiment of the present application calculates the alignment vector difference of the adjacent image blocks of the image to be fused, obtains the alignment vector difference strength of the image region where the image to be fused is located, uses the alignment vector difference strength to characterize the alignment accuracy of the image region where the image to be fused is located, and then brings the alignment vector difference strength into a pre-set alignment curve to obtain an alignment deviation correction parameter. Without complex calculations, the alignment deviation correction parameter for correcting the "similarity calculation deviation between the pixel points in the image block to be fused and the pixel points in the reference image block caused by the alignment registration deviation" can be obtained.

[0170] Another embodiment of the present invention proposes an optional method of obtaining a pixel brightness correction parameter as a deviation correction parameter to adjust the difference between each pixel in the image block to be fused and the pixel corresponding to the same scene position in the reference image block. The deviation correction parameter includes a pixel brightness correction parameter, and obtaining the deviation correction parameter of each pixel in the image block to be fused includes:

[0171] Step S410: Obtain the brightness information value of each pixel in the image block to be fused.

[0172] Based on the brightness information value of each pixel in the reference image block, filtering calculation may be performed on each pixel in the image block to be fused in turn to obtain the brightness value of each pixel in the image block to be fused.

[0173] Step S420: Substitute the brightness information value of each pixel into a brightness curve preset according to the scene sensitivity value to obtain the pixel brightness correction parameter of each pixel in the image block to be fused.

[0174] Figure 6 is a schematic diagram of a brightness curve preset in an example of the present invention, Figure 6 The horizontal axis is the brightness information value, which is expressed in lumens (lum), and lum measures the lighting intensity of the light source. Figure 6 The vertical axis is the pixel brightness correction parameter sigma1. t0 is the minimum brightness information value of the pixel set according to the scene sensitivity value, and t1 is the maximum brightness information value of the pixel set according to the scene sensitivity value. r0 is the pixel brightness correction parameter corresponding to t0, and r1 is the pixel brightness correction parameter corresponding to t1.

[0175] Step S430: using the pixel brightness correction parameter of each pixel to adjust the difference between each pixel in the image block to be fused and the pixel corresponding to the same scene position in the reference image block.

[0176] In the embodiment of the present application, a brightness curve representing the correspondence between brightness information values ​​and pixel brightness correction parameters is set according to the scene sensitivity value. After the brightness information value of the pixel in the image to be fused is obtained, the brightness curve is substituted. The pixel brightness correction parameters can be directly obtained without complicated calculations, so that the pixel brightness correction parameters are used to correct the "similarity calculation error between the pixel in the image block to be fused and the pixel in the reference image block" caused by the pixel brightness.

[0177] Another embodiment of the present invention proposes to calculate the overall similarity of the reference image block and the image block to be fused based on the method of other embodiments to calculate the pixel similarity of the reference image block and the image block to be fused, and use the overall similarity to adjust the fusion weight obtained based on the pixel similarity to obtain the comprehensive fusion weight of the image block to be fused, so as to avoid the fusion weight calculation error caused by the situation that the image block to be fused and the reference image block are not images of the same scene target, but the similarity of individual pixels is extremely high, thereby avoiding the situation where the image information of other scene targets is erroneously superimposed on the reference image block.

[0178] Step S510: Calculate the difference between the average value of pixels in the image block to be fused and the average value of pixels in the reference image block to obtain the pixel mean difference of the image block to be fused.

[0179] Taking the calculation of the average value of pixels in the image block to be fused as an example, the calculation method may be to calculate the UV value or RGB value of each pixel in the image block to be fused, and then find the average value of the UV value or RGB value of all pixels.

[0180] Formula (6) can be used to calculate the pixel mean difference of the image block to be fused:

[0181] d blk =|mean(ref_blk)-mean(cur_blk)| (6); where mean(ref_blk) represents the average value of the pixels in the image block to be fused, mean(cur_blk) represents the difference between the average values ​​of the pixels in the reference image block, and d blk Represents the pixel mean difference of the image block to be fused.

[0182] Step S520: setting the image block brightness correction parameter of the image block to be fused according to the scene brightness information value.

[0183] This application uses an example of sigma blk Represents the image block brightness correction parameter. According to the scene brightness information value, the configuration principle of setting the image block brightness correction parameter of the image block to be fused is that the higher the brightness information value of the current scene, the higher the sigma blk The smaller.

[0184] Step S530: using a preset image block brightness correction parameter to adjust the pixel mean difference of the image block to be fused, to obtain a similarity weight of the image block to be fused, wherein the image block brightness correction parameter is inversely proportional to the scene brightness information value.

[0185] Formula (2) can be used to adjust the pixel mean difference of the image block to be fused using the preset image block brightness correction parameter to obtain the similarity weight of the image block to be fused; the image block brightness correction parameter is substituted into formula (2) to adjust the pixel mean difference of the image block to be fused.

[0186] Among them, d blk Represents the pixel mean difference, sigma blk represents the brightness correction parameter of the image block, Represents the similarity weight of the image block to be fused.

[0187] Step S540: using the image block similarity weight to adjust the fusion weight.

[0188] Formula (7) can be used to adjust the fusion weight with the image block similarity weight to obtain the adjusted fusion weight. In one example of the present invention, the adjusted fusion weight can be used as the comprehensive fusion weight, and the image block to be fused and the reference image block are fused according to the comprehensive fusion weight.

[0189] in, Represents the adjusted fusion weight.

[0190] Step S550: fusing the image block to be fused and the reference image block according to the adjusted fusion weight of the image block to be fused.

[0191] Figure 7 is a flowchart of an exemplary method for performing image fusion in the present invention. Figure 7 As shown, the steps to perform image fusion are as follows:

[0192] Step A: Acquire multiple frames of images for the target scene;

[0193] Step B: preprocessing multiple frames of images;

[0194] Step C: determining a reference image frame from multiple image frames;

[0195] Step D: performing image segmentation on the reference image frame and other image frames to obtain a plurality of reference image blocks and a plurality of image blocks to be fused corresponding to each of the other image frames.

[0196] Step E: For each other image frame, align and register the image blocks to be fused obtained by segmenting each other image frame with the reference image blocks, and determine the reference image blocks that match each image block to be fused.

[0197] Step F: Calculate the alignment vector difference strength of the image area where the image block to be fused is located, and bring it into the stacking area curve to obtain the alignment deviation correction parameter;

[0198] Step G: Obtain the brightness information value of each pixel in each image block to be fused, substitute it into the brightness curve, and obtain the brightness correction parameter of the pixel;

[0199] Step H: Take the larger of the pixel brightness correction parameter and the alignment deviation correction parameter as the deviation correction parameter, correct the difference between the pixels of the image block to be fused and the reference image block in one-to-one correspondence, calculate the point-by-point similarity between the image block to be fused and the reference image block, and obtain the fusion weight

[0200] Step I: Obtain the pixel mean difference of the image block to be fused, use the image block brightness correction parameter set according to the scene brightness information, adjust the pixel mean difference, and obtain the image block similarity weight

[0201] Step J: Fusion of image block similarity weights and fusion weight Get the comprehensive fusion weight of each image block to be fused, that is, the adjusted fusion weight

[0202] Step K: According to the adjusted fusion weight The image block to be fused and the reference image block are fused to obtain a fused single-frame image, and the single-frame image is output.

[0203] Figure 8 is a functional module diagram of the image fusion device proposed in an embodiment of the present invention, Figure 8 As shown, the image fusion device provided in the embodiment can be used to implement the present invention Figure 1-Figure 7 The technical solution of the method embodiment shown, its implementation principle and technical effects can be further referred to the relevant description in the method embodiment.

[0204] In this embodiment, the obtaining module 81 is used to obtain the reference image block and the image block to be fused;

[0205] An adjustment module 82 is used to obtain a deviation correction parameter for each pixel in the image block to be fused, and to adjust the difference between each pixel in the image block to be fused and the pixel corresponding to the same scene position in the reference image block using the deviation correction parameter of each pixel, so as to obtain a fusion weight of the image block to be fused corresponding to the adjusted difference;

[0206] The fusion module 83 is used to fuse the image block to be fused and the reference image block according to the fusion weight of the image block to be fused.

[0207] Fig. 9 A structural schematic diagram of an electronic terminal device provided in an embodiment of the present invention, the electronic terminal device 200 includes a processor 201, a memory 202, and a computer program stored in the memory 202 and executable on the processor 201, the processor 201 implements the steps in the aforementioned method embodiment when executing the program, the electronic terminal device provided in the embodiment can be used to execute the technical solution of the method embodiment shown above, and its implementation principle and technical effects can be further referred to the relevant description in the method embodiment, which will not be repeated here.

[0208] Fig.10 The hardware structure diagram of the electronic terminal device provided in the embodiment of the present invention is shown in the attached Fig.10 The electronic terminal device may include at least one processor; and at least one memory in communication with the processor, wherein: the memory stores program instructions that may be executed by the processor, and the processor calls the program instructions to execute the instructions of this specification. Figure 1 to Figure 7 The illustrated embodiment provides an image fusion method.

[0209] The terminal device may be a smart electronic device such as a smart phone, a tablet computer or a laptop computer. This embodiment does not limit the form of the terminal device.

[0210] For example, Fig.10 Taking a smart phone as an example, a schematic diagram of the structure of a terminal device is shown. Figure 8 As shown, the terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0211] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0212] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0213] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0214] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0215] The processor 110 executes various functional applications and data processing by running the programs stored in the internal memory 121, such as implementing the present application Figure 1 to Figure 7 The illustrated embodiment provides an image fusion method.

[0216] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0217] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the terminal device 100. While the charging management module 140 is charging the battery 142, it may also power the terminal device 100 through the power management module 141.

[0218] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0219] The wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0220] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0221] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0222] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0223] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0224] In some embodiments, the antenna 1 of the terminal device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0225] The terminal device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0226] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0227] The terminal device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0228] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0229] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0230] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the terminal device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0231] Video codecs are used to compress or decompress digital videos. The terminal device 100 may support one or more video codecs. Thus, the terminal device 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0232] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0233] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0234] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the terminal device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the terminal device 100 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor.

[0235] The terminal device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0236] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0237] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The terminal device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0238] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the ear.

[0239] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The terminal device 100 can be provided with at least one microphone 170C. In other embodiments, the terminal device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the terminal device 100 can also be provided with three, four or more microphones 170C to realize the collection of sound signals, noise reduction, identification of sound sources, realization of directional recording function, etc.

[0240] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0241] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The terminal device 100 may receive key input and generate key signal input related to user settings and function control of the terminal device 100.

[0242] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0243] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0244] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the terminal device 100 by inserting the SIM card interface 195 or pulling it out from the SIM card interface 195. The terminal device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the terminal device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100.

[0245] The present application provides a non-transitory computer-readable storage medium storing computer instructions, which enable the computer to execute the instructions of the present invention. Figure 1 to Figure 7 The illustrated embodiment provides an image fusion method.

[0246] The above-mentioned non-transient computer-readable storage medium can adopt any combination of one or more computer-readable media. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, - but not limited to - electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive lists) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (read only memory, ROM), erasable programmable read-only memories (erasable programmable read only memory, EPROM) or flash memory, optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing programs, which can be used by instruction execution systems, devices or devices or used in combination with them.

[0247] Computer-readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0248] The program code embodied on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, radio frequency (RF), etc., or any suitable combination of the foregoing.

[0249] Computer program code for performing the operations of the present specification may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0250] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0251] In the description of the embodiments of the present invention, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0252] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this specification, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0253] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of this specification includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of this specification belong.

[0254] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0255] It should be noted that the terminals involved in the embodiments of the present application may include but are not limited to personal computers (PCs), personal digital assistants (PDAs), wireless handheld devices, tablet computers, mobile phones, MP3 players, MP4 players, etc.

[0256] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0257] In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0258] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes.

[0259] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

Claims

1. An image fusion method, characterized in that: The method comprises: Obtaining a reference image block and an image block to be fused; Obtaining a deviation correction parameter for each pixel in the image block to be fused, and adjusting the difference between each pixel in the image block to be fused and a pixel corresponding to the same scene position in the reference image block using the deviation correction parameter of each pixel, to obtain a fusion weight of the image block to be fused corresponding to the adjusted difference, wherein the fusion weight is negatively correlated with the adjusted difference; The image block to be fused and the reference image block are fused according to the fusion weight of the image block to be fused.

2. The method according to claim 1, characterized in that The obtaining of the deviation correction parameter of each pixel in the image block to be fused includes: Obtaining an alignment deviation correction parameter and a pixel brightness correction parameter for each pixel in the image block to be fused; The larger one between the alignment deviation correction parameter and the pixel brightness correction parameter is taken as the deviation correction parameter.

3. The method according to claim 1, characterized in that: The deviation correction parameter includes an alignment deviation correction parameter, and obtaining the deviation correction parameter of each pixel in the image block to be fused includes: Obtaining an alignment deviation correction parameter for each pixel in the image block to be fused according to an alignment deviation occurring during the alignment process between the reference image block and the image block to be fused; The step of adjusting the difference between each pixel in the image block to be fused and a pixel corresponding to the same scene position in the reference image block by using the deviation correction parameter of each pixel includes: The alignment deviation correction parameter of each pixel is used to adjust the difference between each pixel in the image block to be fused and the pixel corresponding to the same scene position in the reference image block.

4. The method according to claim 3, characterized in that The image block to be fused is an image block in the image frame to be fused, image blocks near the image block to be fused and the image block to be fused form an image region, and according to the alignment deviation between the reference image block and the image block to be fused in the alignment process, an alignment deviation correction parameter of each pixel in the image block to be fused is obtained, including: Calculating the alignment vector difference strength of two image blocks with the largest alignment vector difference in the image region where the image block to be fused is located; wherein the alignment vector difference strength represents the overall alignment accuracy of all image blocks in the image region; The average value of the alignment vector difference strengths of all image regions corresponding to the image block to be fused is substituted into a preset alignment curve to obtain an alignment deviation correction parameter for each pixel in the image block to be fused.

5. The method according to claim 1, characterized in that The deviation correction parameter includes a pixel brightness correction parameter, and obtaining the deviation correction parameter of each pixel in the image block to be fused includes: Obtaining the brightness information value of each pixel in the image block to be fused; Substituting the brightness information value of each pixel into a brightness curve pre-set according to the scene sensitivity value, and obtaining the pixel brightness correction parameter of each pixel in the image block to be fused; The step of adjusting the difference between each pixel in the image block to be fused and a pixel corresponding to the same scene position in the reference image block by using the deviation correction parameter of each pixel includes: The pixel brightness correction parameter of each pixel is used to adjust the difference between each pixel in the image block to be fused and the pixel corresponding to the same scene position in the reference image block.

6. The method according to claim 1, characterized in that The method further comprises: Calculating the difference between the average value of pixels in the image block to be fused and the average value of pixels in the reference image block to obtain the pixel mean difference of the image block to be fused; According to the scene brightness information value, setting the image block brightness correction parameter of the image block to be fused; Using a preset image block brightness correction parameter, adjusting the pixel mean difference of the image block to be fused, to obtain a similarity weight of the image block to be fused, wherein the image block brightness correction parameter is inversely proportional to the scene brightness information value; Using the image block similarity weight, adjusting the fusion weight; According to the fusion weight of the image block to be fused, the image block to be fused and the reference image block are fused, including: The image block to be fused and the reference image block are fused according to the adjusted fusion weight of the image block to be fused.

7. The method according to claim 4, characterized in that The image region where the image block to be fused is located includes a horizontal region and a vertical region; the step of calculating the alignment vector difference strength of two image blocks with the largest alignment vector difference in the image region where the image block to be fused is located includes: Sorting the horizontal displacements of all the image blocks in the horizontal region to obtain an image block with a maximum horizontal displacement and an image block with a minimum horizontal displacement in the horizontal region; sorting the vertical displacements of all the image blocks in the vertical region to obtain an image block with a maximum vertical displacement and an image block with a minimum vertical displacement in the vertical region; Calculating the difference between the maximum horizontal displacement and the minimum horizontal displacement to obtain the alignment vector difference strength of the horizontal area; Calculating the difference between the maximum vertical displacement and the minimum vertical displacement to obtain the alignment vector difference strength of the vertical area; Substituting the average value of the alignment vector difference strength of all image regions corresponding to the image block to be fused into a preset alignment curve to obtain the alignment deviation correction parameter of each pixel in the image block to be fused, including: Taking a square root of the sum of the square of the alignment vector difference strength of the horizontal region and the square of the alignment vector difference strength of the vertical region to obtain an average value of the alignment vector difference strength of the horizontal region and the alignment vector of the vertical region; The alignment vector difference strength of the horizontal region and the average value of the alignment vector of the vertical region are brought into a preset alignment curve to obtain an alignment deviation correction parameter for each pixel in the image block to be fused.

8. The method according to claim 1, characterized in that The method further comprises: Obtaining a difference matrix composed of differences between each pixel in the image block to be fused and a pixel corresponding to the same scene position in the reference image block; Using the deviation correction parameter of each pixel point, adjusting the difference between each pixel point in the image block to be fused and the pixel point corresponding to the same scene position in the reference image block, and obtaining the fusion weight of the image block to be fused corresponding to the adjusted difference value, including: Substitute the deviation correction parameter into formula (1) to adjust the difference matrix; Among them, d pix represents the difference matrix, sigma pix represents the deviation correction parameter, Indicates the fusion weight of the image block to be fused corresponding to the adjusted difference.

9. The method according to claim 6, characterized in that Using the preset image block brightness correction parameter, adjusting the pixel mean difference of the image block to be fused to obtain the similarity weight of the image block to be fused, including: Substitute the image block brightness correction parameter into formula (2) to adjust the pixel mean difference of the image block to be fused; Among them, d blk Represents the pixel mean difference, sigma blk represents the brightness correction parameter of the image block, Represents the similarity weight of the image block to be fused.

10. An image fusion device, characterized in that: The device comprises: An acquisition module, used for acquiring a reference image block and an image block to be fused; A first adjustment module is used to obtain a deviation correction parameter for each pixel in the image block to be fused, and to adjust the difference between each pixel in the image block to be fused and a pixel corresponding to the same scene position in the reference image block using the deviation correction parameter of each pixel, so as to obtain a fusion weight of the image block to be fused corresponding to the adjusted difference, wherein the fusion weight is negatively correlated with the adjusted difference; A fusion module is used to fuse the image block to be fused and the reference image block according to the fusion weight of the image block to be fused.

11. A terminal device, comprising: at least one processor; as well as at least one memory in communication with the processor, wherein: The memory stores program instructions executable by the processor, and the processor can execute the method according to any one of claims 1 to 9 by calling the program instructions. 12 . A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions enable a computer to execute the method according to claim 1 .

Images