Image fusion method, device, electronic device and readable storage medium

Abstract

The present application provides an image fusion method, device, electronic device and readable storage medium, which relate to the field of image processing technology. The method includes: obtaining each target pixel value combination in M ​​images to be fused, each target pixel value combination includes pixel values ​​of M pixels corresponding to each other; obtaining a weight dictionary determined in advance by a neural network model, the weight dictionary contains fusion weights corresponding to different pixel value combinations; determining the fusion weights corresponding to each target pixel value combination according to the weight dictionary; fusing the M images based on the fusion weights corresponding to each target pixel value combination to obtain a fused target image. Compared with the method of manually setting weights through experience, the weights obtained in this scheme in the above manner are more reasonable and have stronger robustness, so that better image fusion effects can be obtained.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image fusion method, device, electronic device and readable storage medium. Background Art

[0002] High-Dynamic Range (HDR), also known as wide dynamic range technology, HDR images can provide more dynamic range and image details than ordinary images. In some application scenarios, images of the same scene with different exposure times can be obtained, and the parts with better details in the images obtained at each exposure time are fused into the final HDR image with a certain weight, which can better reflect the visual effects in the real environment.

[0003] There are many algorithms for HDR image synthesis, among which linear fusion is a very common and mature algorithm. However, when the input image has a large difference in brightness, the synthesized image tends to appear gray, and the overall effect is not natural enough. The reason is that the weights for fusion of bright and dark frames are set manually based on experience, and the weights obtained in this way are not robust enough, resulting in poor synthesized image effects. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image fusion method, device, electronic device and readable storage medium to improve the problem in the prior art that the fusion weights manually set are not robust enough, resulting in poor fusion effect.

[0005] In a first aspect, an embodiment of the present application provides an image fusion method, the method comprising: obtaining each target pixel value combination in M ​​images to be fused, each target pixel value combination comprising pixel values ​​of M corresponding pixels, wherein M is an integer greater than or equal to 2, and the M corresponding pixels respectively belong to the M images and have a corresponding relationship in their positions in the images to which they belong; obtaining a weight dictionary determined in advance by a neural network model, the weight dictionary containing fusion weights corresponding to different pixel value combinations; determining the fusion weights corresponding to each target pixel value combination according to the weight dictionary; and fusing the M images based on the fusion weights corresponding to each target pixel value combination to obtain a fused target image.

[0006] In the above implementation process, a corresponding weight dictionary is obtained for different pixel value combinations through a neural network model, and then the fusion weights corresponding to each target pixel value combination in the M images can be obtained according to the weight dictionary, and then the images are fused based on the fusion weights. Compared with the method of manually setting weights through experience, the fusion weights obtained in the above method in the present application are more reasonable and have stronger robustness, thereby obtaining better image fusion effects.

[0007] Optionally, the obtaining a weight dictionary pre-determined by a neural network model includes:

[0008] Take i1 to i respectively M From 0 to N, multiple pixel value combinations (i1,i2,...,i M ), where N is the number of gray levels of the image minus one;

[0009] Input the M monochrome images corresponding to each pixel value combination in the multiple pixel value combinations into the neural network model to obtain the fusion weight corresponding to each pixel value combination, wherein the fusion weight corresponding to the pixel value combination (i1, i2, ..., i M ) The pixel value of the jth monochrome image in the M monochrome images corresponding to is i j , j is 1 to M;

[0010] The weight dictionary is determined according to the fusion weight corresponding to each pixel value combination.

[0011] In the above implementation process, a weight dictionary corresponding to multiple pixel value combinations is obtained through a trained neural network model, so that the obtained weights are more robust and have a wider range of uses. In addition, the pre-obtained weight dictionary can be pre-stored in an electronic device (such as a mobile terminal), so that the electronic device does not need to run the neural network model, which can reduce the computational complexity of the electronic device and is more friendly to electronic devices with poor computing power.

[0012] Optionally, inputting the M monochrome images corresponding to each pixel value combination of the multiple pixel value combinations into the neural network model to obtain a fusion weight corresponding to each pixel value combination includes:

[0013] Inputting M monochrome images corresponding to each pixel value combination of the multiple pixel value combinations into the neural network model to obtain a weight image corresponding to each pixel value combination;

[0014] The pixel value mean of the weight image corresponding to each pixel value combination is used as the fusion weight corresponding to the corresponding pixel value combination.

[0015] In the above implementation process, the fusion weight is obtained by calculating the mean of the pixel values ​​in the weight image, so that the weights with different values ​​in the weight image can be integrated to obtain a more reasonable weight.

[0016] Optionally, when M=2, the weight image is 1, the fusion weight corresponding to the pixel value combination is the fusion weight corresponding to one pixel value in the pixel value combination, and the fusion weight corresponding to another pixel value in the pixel value combination is 1-the fusion weight corresponding to the pixel value combination. In this way, the output weight image is a two-dimensional image with a small amount of data.

[0017] Optionally, when M is greater than or equal to 2, the weight image is M sub-weight images of the same size, and the sum of the pixel values ​​of the M corresponding pixels in the M sub-weight images is 1; wherein the M corresponding pixels in the M sub-weight images belong to the M sub-weight images respectively and have a corresponding relationship in their respective positions in the images to which they belong. In this way, the output weight images are multiple images, so that the fusion weight corresponding to each pixel value in each pixel value combination can be quickly found later.

[0018] Optionally, the neural network model is trained in the following manner:

[0019] A plurality of groups of sample images are obtained, each group of sample images comprising M sub-sample images;

[0020] Obtain the standard weight image corresponding to each group of sample images;

[0021] Training an initial neural network model according to the sample image and the standard weight image;

[0022] When it is determined that the parameters characterizing the completion of the training of the initial neural network model meet the preset requirements, the trained neural network model is obtained.

[0023] In the above implementation process, the neural network model is trained using training images, so that the neural network model can have higher accuracy when predicting fusion weights.

[0024] Optionally, obtaining a standard weight image corresponding to each group of sample images includes:

[0025] Acquire each sample pixel value combination in the M sub-sample images, wherein the sample pixel value combination includes pixel values ​​of M corresponding sample pixel points, where M is an integer greater than or equal to 2; the M corresponding pixel points belong to the M sub-sample images respectively and have corresponding positions in the images to which they belong;

[0026] Obtain a standard HDR image obtained by fusion of the M sub-sample images;

[0027] For each sample pixel value combination, determining a standard fusion weight corresponding to the sample pixel value combination according to the pixel value of the pixel point corresponding to the sample pixel value combination in the standard HDR image;

[0028] A standard weight image corresponding to the sample image is determined according to the standard fusion weights corresponding to each combination of sample pixel values.

[0029] In the above implementation process, the fusion weights corresponding to the respective pixel value combinations when fusing M sub-sample images are calculated based on the standard HDR image, so as to obtain more accurate label data when training the neural network model.

[0030] Optionally, determining the fusion weights corresponding to each target pixel value combination according to the weight dictionary includes:

[0031] Searching and obtaining the initial weights corresponding to each target pixel value combination from the weight dictionary to obtain an initial weight image, wherein the initial weight image includes each initial weight;

[0032] Performing filtering on the initial weight image to obtain a filtered weight image;

[0033] The fusion weights corresponding to each target pixel value combination are obtained from the filtered weight image.

[0034] In the above implementation process, filtering the initial weight image can remove the mid-high frequency information in the image and preserve the edge information in the image, which can effectively avoid the problem that the mid-high frequency changes affect the image details of the fused image.

[0035] Optionally, fusing the M images based on fusion weights corresponding to the target pixel value combinations to obtain a fused target image includes:

[0036] Determine the fusion weights corresponding to the M pixels in the target pixel value combination according to the fusion weights corresponding to the target pixel value combination;

[0037] Performing weighted fusion on the M pixel points according to the fusion weights corresponding to the M pixel points, to obtain pixel values ​​of fused pixel points corresponding to the M pixel points;

[0038] According to the pixel value of each fused pixel point, the fused target image is determined.

[0039] In the above implementation process, by obtaining the fusion weight of each pixel point, the rapid fusion of M images can be achieved.

[0040] In a second aspect, an embodiment of the present application provides an image fusion device, the device comprising:

[0041] A pixel value acquisition module, used to acquire each target pixel value combination in the M images to be fused, each of the target pixel value combinations including pixel values ​​of M corresponding pixels, wherein M is an integer greater than or equal to 2, and the M corresponding pixels respectively belong to the M images and have corresponding positions in the respective images;

[0042] A weight dictionary acquisition module is used to acquire a weight dictionary determined in advance by a neural network model, wherein the weight dictionary contains fusion weights corresponding to different pixel value combinations;

[0043] A weight search module, used to determine the fusion weight corresponding to each target pixel value combination according to the weight dictionary;

[0044] The image fusion module is used to fuse the M images based on the fusion weights corresponding to the target pixel value combinations to obtain a fused target image.

[0045] Optionally, the weight dictionary acquisition module is used to respectively obtain i1 to i M From 0 to N, multiple pixel value combinations (i1,i2,...,i M ), where N is the number of gray levels of the image minus one; inputting the M monochrome images corresponding to each pixel value combination in the multiple pixel value combinations into the neural network model to obtain the fusion weight corresponding to each pixel value combination, where the pixel value combination (i1, i2, ..., i M ) The pixel value of the jth monochrome image in the M monochrome images corresponding to is i j , j is 1 to M; the weight dictionary is determined according to the fusion weight corresponding to each pixel value combination.

[0046] Optionally, the weight dictionary acquisition module is used to input M monochrome images corresponding to each pixel value combination of the multiple pixel value combinations into the neural network model to obtain a weight image corresponding to each pixel value combination; and use the pixel value mean of the weight image corresponding to each pixel value combination as the fusion weight corresponding to the corresponding pixel value combination.

[0047] Optionally, when M=2, the weight image is 1, the fusion weight corresponding to the pixel value combination is the fusion weight corresponding to one pixel value in the pixel value combination, and the fusion weight corresponding to another pixel value in the pixel value combination is 1-the fusion weight corresponding to the pixel value combination.

[0048] Optionally, when M is greater than or equal to 2, the weight image is M sub-weight images of the same size, and the sum of the pixel values ​​of M corresponding pixels in the M sub-weight images is 1; wherein, the M corresponding pixels in the M sub-weight images respectively belong to the M sub-weight images and have a corresponding relationship in their positions in the images to which they belong.

[0049] Optionally, the device further comprises:

[0050] A model training module is used to obtain multiple groups of sample images, each group of sample images includes M sub-sample images; obtain standard weight images corresponding to each group of sample images; train an initial neural network model according to the sample images and the standard weight images; when it is determined that the parameters characterizing the completion of the training of the initial neural network model meet preset requirements, obtain the trained neural network model.

[0051] Optionally, the model training module is used to obtain each sample pixel value combination in the M sub-sample images, the sample pixel value combination including pixel values ​​of M corresponding sample pixel points, wherein M is an integer greater than or equal to 2; the M corresponding pixel points respectively belong to the M sub-sample images and have a corresponding relationship in the positions in the respective images to which they belong; obtain a standard HDR image obtained after the M sub-sample images are fused; for each sample pixel value combination, determine a standard fusion weight corresponding to the sample pixel value combination according to the pixel value of the pixel point corresponding to the sample pixel value combination in the standard HDR image; determine a standard weight image corresponding to the sample image according to the standard fusion weights corresponding to each sample pixel value combination.

[0052] Optionally, the weight search module is used to search the weight dictionary for initial weights corresponding to each target pixel value combination to obtain an initial weight image, which includes each initial weight; filter the initial weight image to obtain a filtered weight image; and obtain the fusion weights corresponding to each target pixel value combination from the filtered weight image.

[0053] Optionally, the image fusion module is used to determine the fusion weights corresponding to each of the M pixel points in the target pixel value combination according to the fusion weights corresponding to the target pixel value combination; perform weighted fusion on the M pixel points according to the fusion weights corresponding to each of the M pixel points to obtain pixel values ​​of fused pixel points corresponding to the M pixel points; and determine the fused target image according to the pixel values ​​of each fused pixel point.

[0054] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the method provided in the first aspect are performed.

[0055] In a fourth aspect, an embodiment of the present application provides a readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the method provided in the first aspect are performed.

[0056] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the embodiments of the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 A flowchart of an image fusion method provided in an embodiment of the present application;

[0059] Figure 2 A schematic diagram of the structure of a convolutional neural network model provided in an embodiment of the present application;

[0060] Figure 3 A schematic diagram of the structure of a coding layer provided in an embodiment of the present application;

[0061] Figure 4 A schematic diagram of the structure of a decoding layer provided in an embodiment of the present application;

[0062] Figure 5 A schematic diagram of the structure of an attention module provided in an embodiment of the present application;

[0063] Figure 6 A schematic diagram of comparison before and after filtering an initial weight map provided in an embodiment of the present application;

[0064] Figure 7 A structural block diagram of an image fusion device provided in an embodiment of the present application;

[0065] Figure 8 A schematic diagram of the structure of an electronic device for executing an image fusion method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present application.

[0067] The embodiment of the present application provides an image fusion method, which obtains a corresponding weight dictionary for different pixel value combinations through a neural network model, and then obtains the fusion weights corresponding to each target pixel value combination in M ​​images according to the weight dictionary, and then performs image fusion based on the fusion weights. Compared with the method of manually setting weights through experience, the fusion weights obtained in the above method in the present application are more reasonable and have stronger robustness, thereby obtaining a better image fusion effect.

[0068] Please refer to Figure 1 , Figure 1 A flowchart of an image fusion method provided in an embodiment of the present application, the method comprising the following steps:

[0069] Step S110: Obtain each target pixel value combination in the M images to be fused; each target pixel value combination includes pixel values ​​of M corresponding pixel points, where M is an integer greater than or equal to 2; the M corresponding pixel points belong to the M images respectively and have a corresponding relationship in their positions in the images to which they belong.

[0070] The M images to be fused are images with basically the same field of view but different brightness. For example, multiple images are obtained by shooting the same position using different shooting parameters. When there are objects of different brightness in the field of view (such as a lamp and a black cat), when the shooting parameters of other images are the same and only the exposure time parameter is different, in the image obtained with a longer exposure time, the lamp is overexposed and the black cat is properly exposed; in the image obtained with a shorter exposure time, the lamp is properly exposed and the black cat is underexposed. In order to obtain the visual effect of both the lamp and the black cat being properly exposed, it is hoped that the two images can be fused, and the lamp object in the fused image mainly uses the information in the image obtained with a short exposure time, and the black cat object in the fused image mainly uses the information in the image obtained with a long exposure time. It can be seen that in order to obtain a good fused image effect, it is necessary to determine the fusion weight of each pixel of the image to be fused according to the pixel value of each pixel of the image to be fused.

[0071] Among them, M images may refer to images taken under different shooting parameters (such as exposure time, gain) for the same scene, and the same scene may refer to the same target object contained in the image, such as shooting the same person at the same position and at the same angle. Of course, the same position and the same angle do not mean the absolute same position and the same angle. In the actual process, due to factors such as hand shaking during shooting, the angle or position may be slightly offset. For such a slightly offset scene, it is also considered to be an image taken of the same scene. After obtaining M images, in order to fuse the M images, the corresponding pixels in the M images can be found first. The corresponding pixels can be understood as M pixels at corresponding positions in the M images. Taking two images as an example, the way to obtain corresponding pixels can be to align image 1 with image 2, and each corresponding pixel in the two images can be found by alignment. It can be understood that due to the different shooting parameters of Image 1 and Image 2, the pixel values ​​of corresponding pixels in Image 1 and Image 2 may be different. For example, the pixel value of a pixel in Image 1 is greater than the pixel value of the corresponding pixel in Image 2. At this time, Image 1 can be called a bright frame image and Image 2 can be called a dark frame image. Alternatively, when the average pixel value of Image 1 is greater than the average pixel value of Image 2, Image 1 is called a bright frame image and Image 2 is called a dark frame image.

[0072] When fusing M images with different pixel values, such as fusing a bright frame image with a dark frame image, in order to preserve the highlight information in the bright frame image and the detail information in the dark frame image as much as possible, it is necessary to set respective fusion weights for the corresponding pixels in the M images.

[0073] In order to obtain the fusion weight, it is necessary to first obtain each target pixel value combination in the M images, and then obtain the fusion weight corresponding to each target pixel value combination. Among them, the target pixel value combination includes the pixel values ​​of the corresponding M pixels. For example, taking two images as an example, pixel 1 is a pixel in image 1, pixel 2 is a pixel in image 2, and pixel 1 and pixel 2 are mutually corresponding pixels. Then the target pixel value combination corresponding to "pixel 1, pixel 2" includes the pixel value of pixel 1 and the pixel value of pixel 2. For example, if the pixel value of pixel 1 is 50 and the pixel value of pixel 2 is 52, then the target pixel value combination can be expressed as (50,52). It should be noted here that if both image 1 and image 2 are monochrome images, since the pixel values ​​of all pixels in the monochrome image are the same, in this case, the target pixel value combination of each mutually corresponding pixel in image 1 and image 2 can be expressed as (50,52), that is, all mutually corresponding pixels correspond to the same target pixel value combination.

[0074] It is understandable that the image to be fused can be obtained by shooting, or can be obtained by image fusion, for example, by the image fusion method of the embodiment of the present application. If three images are to be fused, the three images can be directly fused, or two of the images can be fused first to obtain a fused intermediate image, and then the remaining image and the fused intermediate image are fused to obtain the final fused image.

[0075] Step S120: Obtain a weight dictionary determined in advance by a neural network model.

[0076] Among them, the weight dictionary includes fusion weights corresponding to different pixel value combinations, such as the fusion weight corresponding to the pixel value combination (23, 24) is (0.8, 0.2), which means that the fusion weight corresponding to the pixel value 23 is 0.8, and the fusion weight corresponding to the pixel value 24 is 0.2. The weight dictionary contains fusion weights corresponding to multiple such pixel value combinations. In other embodiments, in the weight dictionary, the fusion weight corresponding to the pixel value combination (23, 24) can also be 0.8, which means the fusion weight corresponding to one of the pixel value 23 and the pixel value 24, and the fusion weight corresponding to the other pixel value can be calculated according to the fusion weight in the weight dictionary, for example, the weight corresponding to the pixel value 23 is 0.8, and the weight corresponding to the pixel value 24 is 1-0.8=0.2.

[0077] That is to say, the fusion weights stored in the above-mentioned weight dictionary may refer to the fusion weight combinations corresponding to each pixel value combination. For example, if the pixel value range of the image to be fused is 0-255, and there are 2 images to be fused, there are 256*256 or 256*256 / 2 pixel value combinations, and each pixel value combination corresponds to a fusion weight. The fusion weights stored in the weight dictionary may be the fusion weights corresponding to each pixel value in the pixel value combination. Alternatively, the weight dictionary may also store the fusion weights corresponding to a pixel value in the pixel value combination, and the fusion weights of other pixel values ​​in the pixel value combination may be calculated based on the fusion weights in the weight dictionary. For example, if the fusion weight corresponding to the pixel value of 50 is 0.8, then the 0.8 is the fusion weight of the pixel point with a pixel value of 50 corresponding to one of the images when performing image fusion, then the fusion weights of the corresponding pixel points in the other images may be calculated based on the fusion weights of the pixel points of the aforementioned image (the specific implementation process is described in detail in the subsequent embodiments).

[0078] It can be understood that the specific expression form of the weight dictionary can be flexibly set based on actual needs, and different expression forms can be set under different needs.

[0079] The weight dictionary may be obtained through a trained neural network model, and the neural network model may refer to a network model such as a convolutional neural network model and a recurrent neural network model.

[0080] When training a neural network model, M sub-sample images can be used as a group of sample images, and the standard HDR image obtained after the fusion of the M sub-sample images can be used as the standard HDR image of the group of sample images. In this way, the pixel values ​​of each corresponding sample pixel in the M sub-sample images and the pixel values ​​of the pixels corresponding to the sample pixel value combination in the fused standard HDR image can be known, so that the fusion weights of each corresponding pixel in the M sub-sample images can be calculated. For example, pixel 1 in sub-sample image 1 and pixel 2 in sub-sample image 2 are corresponding pixels, and the corresponding pixel after the fusion of these two pixels is pixel 3 in the standard HDR image. Then, the pixel values ​​of pixel 1, pixel 2 and pixel 3 are obtained respectively, and the formula for calculating the fusion weight can be: pixel value of pixel 1*x+pixel value of pixel 2*(1-x)=pixel value of pixel 3, where x is the fusion weight corresponding to pixel 1, and 1-x is the fusion weight corresponding to pixel 2. In this way, the standard fusion weight corresponding to each sample pixel value combination in the M sub-sample images can be obtained. The standard weight image corresponding to the sample image can be determined according to the standard fusion weight corresponding to each sample pixel value combination. Subsequently, the initial neural network model is trained according to the sample image and the standard weight image; when it is determined that the parameters representing the completion of the training of the initial neural network model meet the preset requirements, the trained neural network model is obtained.

[0081] After such training, after M monochrome images are input into the neural network model, the neural network can output the fusion weights corresponding to the pixel value combinations corresponding to the M monochrome images. The fusion weights obtained by inputting the M monochrome images corresponding to all possible pixel value combinations into the neural network model are summarized to obtain a weight dictionary. It can be understood that the number of all possible pixel value combinations is determined by the number of grayscale levels of the image pixel values ​​and the number of images to be fused. If the pixel value range of the image to be fused is 0-255, and there are 2 images to be fused, then the number of all possible pixel value combinations is 256*256 or 256*256 / 2. The fusion weights obtained by inputting the M monochrome images corresponding to all possible pixel value combinations into the neural network model are summarized to obtain a weight dictionary, which can ensure that the fusion weight corresponding to the target pixel value combination can be queried in the weight dictionary.

[0082] Step S130: determining the fusion weights corresponding to the target pixel value combinations according to the weight dictionary.

[0083] After obtaining each target pixel value combination in M ​​images, the target pixel value combination can be used as a query index to find the fusion weight corresponding to the target pixel value combination in the weight dictionary. For example, if the target pixel value combination is (50,52), the fusion weight corresponding to the pixel value combination (50,52) can be found from the weight dictionary. The weight dictionary can store the fusion weight of each pixel value in the pixel value combination or the fusion weight of one pixel value in the pixel value combination. In this case, the fusion weights of other pixel values ​​in the pixel value combination can be calculated based on the fusion weight of this pixel value combination. For example, the weight dictionary contains fusion weights corresponding to each pixel value in the pixel value combination, such as the fusion weight corresponding to the pixel value combination (50,52) is (0.2,0.8), which means that the fusion weight corresponding to the pixel value of 50 is 0.2, and the fusion weight corresponding to the pixel value of 52 is 0.8. Alternatively, the weight dictionary may also store the fusion weight corresponding to a pixel point in the pixel value combination, where a pixel point can be set to be the previous pixel point in the pixel value combination or the next pixel point, and can be set to be a larger pixel point in the pixel value combination or a smaller pixel point in the pixel value combination. For example, if the fusion weight corresponding to the pixel value combination (50, 52) in the weight dictionary is 0.2, if the fusion weight recorded in the weight dictionary is set to be the fusion weight of the previous pixel in the pixel value combination, it means that the fusion weight corresponding to the pixel value of 50 is 0.2, and then the fusion weight of the pixel value of 52 is obtained by calculation to be 0.8; or, if the fusion weight recorded in the weight dictionary is set to be the fusion weight of the next pixel in the pixel value combination, it means that the fusion weight corresponding to the pixel value of 52 is 0.2, and the fusion weight corresponding to the pixel value of 50 is 1-0.2=0.8.

[0084] Therefore, according to the above method, the fusion weight corresponding to each target pixel value combination can be obtained according to the weight dictionary.

[0085] Step S140: fusing the M images based on the fusion weights corresponding to the target pixel value combinations to obtain a fused target image.

[0086] After obtaining the fusion weights corresponding to each target pixel value combination, the pixel points corresponding to each target pixel value combination can be fused based on the fusion weights. For example, for the corresponding pixel points (pixel point 1, pixel point 2), the corresponding target pixel value combination is (50, 52), and the fusion weight corresponding to the target pixel value combination is (0.7, 0.3). Fusion of two images refers to fusion of the corresponding pixel points of the two images. The calculation method for fusing pixel point 1 and pixel point 2 can be, for example, the weighted sum of the pixel value of pixel point 1 and the pixel value of pixel point 2, expressed as: 50*0.7+52*0.3=50.6, 50.6 is the pixel value of the pixel point at the position in the target image obtained after fusion. The fusion methods for other corresponding pixel points can be fused in the above manner, and no examples are given here.

[0087] Of course, there are other ways to fuse images based on fusion weights. For example, the calculation method of pixel values ​​during fusion can also be: A*a*b1+B*(1-a)*b2, where A and B represent the pixel values ​​of two corresponding pixels, a represents the fusion weight corresponding to the pixel with pixel value A, 1-a represents the fusion weight corresponding to the pixel with pixel value B, and b1 and b2 represent weight coefficients, which can be set according to requirements. For example, in some application scenarios, if it is necessary to retain certain areas in the bright frame image as much as possible, corresponding weight coefficients can be set for the pixels in these areas to increase the weight of the area. Of course, for certain areas, the weight can also be reduced by the weight coefficient, so that the weight corresponding to each pixel can be adjusted according to requirements.

[0088] It can be understood that the above example is for the scenario of fusing two images. If three images or more images are to be fused, the same is true. For example, when fusing three images, two images can be fused first to obtain a fused image and then fused with another image. That is to say, in this scenario, the fusion weight stored in the weight dictionary can be the fusion weight corresponding to the pixel value combination of two corresponding pixels. For example, the pixel value combination of a corresponding pixel in the three images is (50, 55, 40). At this time, the pixel points corresponding to the pixel value of 50 and the pixel value of 55 can be fused first. Then, according to the weight dictionary, the fusion weight corresponding to the pixel value combination of (50, 55) is (0.2, 0.8). Then, based on the above fusion method, the two pixel values ​​(50, 55) are fused, and the fused pixel value obtained is 54. When fusion is performed again, the fusion weight corresponding to the pixel value combination (54, 40) is obtained according to the weight dictionary, and then fusion is performed again in the above method to achieve the fusion of three pixels. Alternatively, if the weight dictionary stores the fusion weight corresponding to the pixel value combination of three pixel points, such as (0.2, 0.3, 0.5), the fusion method can be, for example, 50*0.2+55*0.5+40*0.5=57.5, then 57.5 is the pixel value of the corresponding pixel point after fusion. In this way, multiple images can be fused only once.

[0089] Therefore, for the fusion of more than three images, the final target image can be obtained by fusion of two images or by fusion of three images at a time.

[0090] In the above implementation process, a corresponding weight dictionary is obtained for different pixel value combinations through a neural network model, and then the fusion weights corresponding to each target pixel value combination in the M images can be obtained according to the weight dictionary, and then the images are fused based on the fusion weights. Compared with the method of manually setting weights through experience, the fusion weights obtained in the above method in the present application are more reasonable and have stronger robustness, thereby obtaining better image fusion effects.

[0091] In some implementations, the weight dictionary is obtained in step S120 by taking i1 to i2 respectively. M From 0 to N, multiple pixel value combinations (i1,i2,...,i M ), where N is the number of gray levels of the image minus one, and then M monochrome images corresponding to each pixel value combination in the multiple pixel value combinations are input into the neural network model to obtain the fusion weight corresponding to each pixel value combination, where the pixel value combination (i1, i2, ..., i M ) The pixel value of the jth monochrome image in the M monochrome images corresponding to is i j, j is 1 to M, and then the weight dictionary is determined according to the fusion weight corresponding to each pixel value combination.

[0092] In the embodiments of the present application, an RGB image is used as an example for explanation, and of course, the image is not limited to other types of images, such as grayscale images, etc. For an RGB image, the number of grayscale levels is 256, that is, it includes 256 grayscale levels, and the maximum value of N is 255, so that for M monochrome images, there are 256 to the Mth power, or 256 to the Mth power / M factorial, so for each pixel value combination, the corresponding fusion weight can be obtained through the neural network model, and these fusion weights can form a weight dictionary.

[0093] Among them, a monochrome image can be understood as an image in which the pixel values ​​of all pixels in the image are the same. For example, if a monochrome image contains 100 pixels, the pixel values ​​of these 100 pixels are the same, such as 1. In this way, M monochrome images can be input into the neural network model according to different combinations of pixel values. For example, when M is 2, its pixel value combinations can include (0,0), (0,1), ... (255,255), etc., and then the M monochrome images corresponding to these pixel value combinations are input into the neural network model, so that the fusion weights corresponding to different pixel value combinations can be obtained.

[0094] Take two monochrome images as an example. For example, the size of the first monochrome image is 100*100, and the pixel values ​​of all pixels are 250. The size of the second monochrome image is 100*100, and the pixel values ​​of all pixels are 10. When these two images are input into the neural network model, the fusion weight corresponding to the pixel value combination (250, 10) can be obtained according to the output result of the neural network model. In this way, 256 2 The fusion weight under the combination of pixel values.

[0095] It should be noted here that when the fusion weights corresponding to the pixel value combination (250,10) and the pixel value combination (10,250) are different (for example, (0.9,0.1) and (0.1,0.9) respectively), 256 2 The fusion weights under the pixel value combination are as follows; when the fusion weights corresponding to the pixel value combination (250,10) and the pixel value combination (10,250) are the same (for example, both are (0.9,0.1)), 256 2 / Fusion weights under 2 pixel value combinations.

[0096] In the above implementation process, a weight dictionary corresponding to multiple pixel value combinations is obtained through a trained neural network model, so that the obtained weights are more robust and have a wider range of uses. In addition, the pre-obtained weight dictionary can be pre-stored in an electronic device (such as a mobile terminal), so that the electronic device does not need to run the neural network model, which can reduce the computational complexity of the electronic device and is more friendly to electronic devices with poor computing power.

[0097] In some embodiments, the weight dictionary output by the neural network model can also be represented in the form of an image, or obtained based on the image output by the model, such as inputting M monochrome images corresponding to each pixel value combination in a plurality of pixel value combinations into the neural network model, obtaining a weight image corresponding to each pixel value combination, and the pixel value in the weight image is the fusion weight corresponding to the pixel value combination. In other words, the weight dictionary can be obtained based on the weight image, so that what is stored in the weight dictionary is the fusion weight in the weight image, or the weight image can also be directly used as the weight dictionary.

[0098] When two monochrome images corresponding to a certain pixel value combination are input into the neural network model, the output result can be a weighted image, and the mean pixel value of the weighted image is used as the fusion weight corresponding to the pixel value combination. In one example, the first monochrome image (size is 100*100, the pixel value of all pixels is 250) and the second monochrome image (size is 100*100, the pixel value of all pixels is 10) corresponding to the pixel value combination (250,10) are input into the neural network model, and a weighted image of size 100*100 and the pixel value of all pixels is 0.9 is obtained. The mean pixel value of the weighted image 0.9 is used as the fusion weight corresponding to the pixel value combination (250,10). At this time, in the weight dictionary, the fusion weight corresponding to the pixel value combination (250,10) is 0.9. In another example, the first monochrome image (size 100*100, all pixel values ​​are 250) and the second monochrome image (size 100*100, all pixel values ​​are 10) corresponding to the pixel value combination (250, 10) are input into the neural network model, and two weighted images of size 100*100 are obtained, in which all pixel values ​​on one weighted image are 0.9, and all pixel values ​​on the other weighted image are 0.1. The pixel value mean (0.9, 0.1) of the weighted image is used as the fusion weight corresponding to the pixel value combination (250, 10). At this time, in the weight dictionary, the fusion weight corresponding to the pixel value combination (250, 10) is (0.9, 0.1).

[0099] In order to facilitate image fusion, the above-mentioned M monochrome images have the same size. For example, before inputting the M monochrome images into the neural network model, the image size of the Mth monochrome image can be adjusted to be consistent, such as 100*100. In this way, the output of the neural network model can also be a single-channel floating-point image of size 100*100, that is, a weight image. In this way, each pixel in the floating-point image represents the fusion weight when the M monochrome images are fused. Since images of real scenes are used in the training process of the neural network model, the weight image output by the neural network model contains semantic information, resulting in that not every pixel in the output weight image has the same pixel value. Therefore, the pixel mean of the weight image corresponding to each pixel value combination can also be used as the fusion weight corresponding to the corresponding pixel value combination.

[0100] That is to say, in the process of obtaining the weight dictionary, the pixel value combination (i1, i2, ..., i M ) corresponding to the M monochrome images are input into the trained neural network model to obtain the pixel value combination (i1,i2,...,i M ), and then calculate the mean of each pixel value in the weighted image to obtain the pixel mean, which is the pixel value combination (i1,i2,...,i M ) corresponds to the fusion weight.

[0101] For example, when the size of a monochrome image is 100*100, the size of each weighted image obtained is also 100*100, that is, each weighted image contains multiple pixels. Then, the pixel value of each pixel can be obtained, and then the pixel value average can be calculated to obtain the pixel mean, which is the fusion weight under the corresponding pixel value combination.

[0102] In the above implementation process, the fusion weight is obtained by calculating the mean of the pixel values ​​in the weight image, so that the weights with different values ​​in the weight image can be integrated to obtain a more reasonable weight.

[0103] In other embodiments, the method for obtaining the fusion weight corresponding to each pixel value combination can also be to first obtain the pixel value of each pixel point in the weight image, and then count the number of pixel points with the same pixel value. If the proportion of pixel points with the same pixel value exceeds a preset value, the same pixel value is determined as the fusion weight of the corresponding pixel value combination.

[0104] Among them, the preset value can be set according to actual needs, such as 70%. For example, if the pixel values ​​of more than 70% of the pixels in the weight image corresponding to the pixel value combination (10, 20) are 0.6, then the pixel value 0.6 is considered to be the fusion weight corresponding to the pixel value combination.

[0105] In some embodiments, the weight image output by the neural network model may be 1 or M. When the weight image is 1, the pixel value of the pixel point in the weight image may represent the fusion weight corresponding to one pixel value in the pixel value combination, and the fusion weight corresponding to another pixel value in the pixel value combination is 1-the fusion weight corresponding to the pixel value combination. The weight image data output in this way is small in size and easy to store.

[0106] Alternatively, when M is greater than or equal to 2, the weight image is M sub-weight images of the same size, and the sum of the pixel values ​​of the M corresponding pixels in the M sub-weight images is 1, wherein the M corresponding pixels in the M sub-weight images belong to the M sub-weight images respectively and have a corresponding relationship in their respective positions in the images to which they belong. In this way, multiple weight images are output to facilitate the subsequent rapid search for the fusion weight corresponding to each pixel value in the pixel value combination.

[0107] The following is an introduction to the training process of the neural network model.

[0108] Taking the neural network model as a convolutional neural network model as an example, the structure of the convolutional neural network model can adopt the structure of the existing convolutional neural network model. In the embodiment of the present application, in order to improve the network performance, all convolution operations in the model can be designed as depthwise convolutions, and an attention mechanism can be added to the model to increase the robustness of the model to the ghost problem in the image. Figure 2 As shown in the figure, in the model, the attention map output by the attention module is multiplied by the coding vector output by the coding layer after the pooling layer, and the obtained value is input into the subsequent coding layer. This can increase the weight of the ghost area in the image, improve the attention to the ghost area, and facilitate the subsequent extraction of valid pixels in the ghost area.

[0109] The structural diagram of the coding layer is as follows: Figure 3 As shown, the structural diagram of the decoding layer is as follows Figure 4 As shown in Figure 2, the structural diagram of the attention module is as follows: Figure 5 shown.

[0110] In the training stage, each group of sample images contains M sub-sample images. Taking the example that each group of sample images contains two sub-sample images, for two sub-sample images with different brightness (for the convenience of description, they are called bright frame images and dark frame images), the predetermined HDR standard image can be used as the standard to infer the standard fusion weights corresponding to the combination of each sample pixel value in the two sub-sample images, thereby obtaining a standard weight image, and the standard weight image is used as the gt corresponding to the two sub-sample images. The HDR standard image is the image obtained after the fusion of the two sub-sample images. A specific inversion method is: Bright (bright frame image) * Mask (standard weight image) + Dark (dark frame image) * (1-Mask) = HDR standard image, so Mask = (HDR-Dark) / (Bright-Dark). For example, if the pixel value of a certain pixel in the bright frame image is 100, the pixel value of the corresponding pixel in the dark frame image is 50, and the pixel value of the corresponding pixel in the HDR standard image is 80, then the fusion weight of the pixel with a pixel value of 100 in the bright frame image can be calculated as 0.6, and the fusion weight of the pixel with a pixel value of 50 in the dark frame image can be calculated as 0.4. In this way, the standard fusion weight of each pixel in the sample image is obtained, and a standard weight image is formed according to the standard fusion weight of each pixel.

[0111] Then the two sub-sample images can be used as training images of the convolutional neural network model, and the standard weight images obtained above can be used as label data in the training process of the convolutional neural network model to train the convolutional neural network model. After a certain stage of training, if the value of the loss function is less than the specified threshold or the number of iterations reaches the threshold, it means that the parameters characterizing the completion of the training of the convolutional neural network model meet the preset requirements, then it is determined that the convolutional neural network model has been trained.

[0112] In addition, it should be noted that when the inversely derived Mask standard weight image is single-channel (1 image), the weight image output by the neural network model is also single-channel; when the inversely derived Mask standard weight image is multi-channel (multiple images), the weight image output by the neural network model is also multi-channel. If you want to make the inversely derived Mask standard weight image multi-channel (multiple images), you can limit Bright (bright frame image) * Mask1 (standard weight image 1) + Dark (dark frame image) * Mask2 (standard weight image 2) = HDR standard image, and then limit the relationship between Mask1 and Mask2 (for example, limit Mask1 + Mask2 to equal all pixel values ​​1). In this way, multiple standard weight images can be inverted based on the bright frame image, dark frame image and HDR standard image.

[0113] In the above implementation process, by obtaining the weight dictionary in the offline process, the problem of the convolutional neural network model being too slow and unable to be real-time in actual application is avoided, and it can be applied to electronic devices with poor computing power.

[0114] In some embodiments, in order to achieve a better fusion effect, in the process of obtaining the fusion weights corresponding to each target pixel value combination according to the weight dictionary, the initial weights corresponding to each target pixel value combination can be first searched from the weight dictionary, and these initial weights constitute an initial weight image. It is understandable that the size of the target image is the same as the size of the M images to be fused. For a first target pixel point in the target image, there are M pixels corresponding to the first target pixel point in the M images, and these M pixels constitute a target pixel value combination. The size of the initial weight image is the same as the size of the target image. The fusion weight queried in the weight dictionary with this target pixel value combination as the index is the pixel value of the pixel point corresponding to the first target pixel point in the initial weight image. It is understandable that the target pixel value combination corresponding to multiple target pixels in the target image may be the same, and there may be multiple pixels with the same pixel value in the initial weight image.

[0115] Among them, taking two images as an example, the method of obtaining the initial weight image can be: for example, each pixel point in the first image can be traversed, the corresponding pixel point in the second image can be found, the target pixel value combination of the two pixel points can be obtained, and then the initial weight corresponding to each target pixel value combination can be found from the weight dictionary in turn. The initial weight is the pixel value of the pixel point corresponding to the position of the two pixel points in the initial weight image.

[0116] Then, the initial weight image is filtered to obtain the filtered weights, and then the fusion weights corresponding to each target pixel value combination are obtained from the filtered weight image.

[0117] The initial weight image is filtered. Guided filtering can be used here. The significance of filtering is to remove high-mid-frequency information and preserve the edge information of the initial weight image, so that the changes in the mid-high frequency in the initial weight image will not affect the image details in the final fused image, such as Figure 6 As shown, Figure 6 This is a comparison diagram before and after filtering the initial weighted image. The image on the left is the initial weighted image before filtering, and the image on the right is the weighted image obtained after filtering. The specific filtering process is not introduced in detail here, and the relevant implementation process in the prior art can be referred to for details.

[0118] In the above implementation process, filtering the initial weight image can remove the mid-high frequency information in the image and preserve the edge information in the image, which can effectively avoid the problem that the mid-high frequency changes affect the image details of the fused image.

[0119] In some embodiments, in the process of fusing M images based on fusion weights to obtain a target image, the fusion weights corresponding to the M pixels in the target pixel value combination can be determined according to the fusion weights corresponding to the target pixel value combination, and then the M pixels are weightedly fused according to the fusion weights corresponding to the M pixels to obtain the pixel values ​​of the fused pixels corresponding to the M pixels, and the fused target image is determined according to the pixel values ​​of each fused pixel.

[0120] For example, for image 1 and image 2, the target pixel value combination includes the pixel values ​​of pixel 1 and pixel 2 that correspond to each other. The fusion weights corresponding to pixel 1 and pixel 2 can be obtained respectively by the above method, and then the pixel values ​​of these two pixels are weighted and summed according to the fusion weights. The obtained pixel value is the pixel value of the corresponding fused pixel in the target image. In this way, the pixel value of each fused pixel in the target image can be obtained.

[0121] According to the above method, when fusion is performed, the pixel values ​​of the corresponding pixels are weighted and summed to obtain the pixel values ​​of the corresponding pixels in the target image, so that the fusion of M images can be achieved.

[0122] In the above implementation process, by obtaining the weight of each pixel point, the rapid fusion of M images can be achieved.

[0123] In some embodiments, in order to facilitate the search for the weights corresponding to each pixel in any image, each pixel in any image can be traversed, and then the fusion weights corresponding to each pixel can be searched from the weight dictionary in turn, so that the fusion weights of each pixel can be obtained quickly and orderly, avoiding the problem of repeatedly searching for fusion weights and doing useless work.

[0124] Please refer to Figure 7 , Figure 7 1 is a structural block diagram of an image fusion device 100 provided in an embodiment of the present application. The device 100 may be a module, program segment or code on an electronic device. It should be understood that the device 100 is similar to the above-mentioned Figure 1 The method embodiment corresponds to and can be executed Figure 1 The various steps involved in the method embodiment and the specific functions of the device 100 can be found in the above description. To avoid repetition, the detailed description is appropriately omitted here.

[0125] Optionally, the device 100 includes:

[0126] The pixel value acquisition module 110 is used to acquire each target pixel value combination in the M images to be fused, each of which includes pixel values ​​of M corresponding pixels, where M is an integer greater than or equal to 2, and the M corresponding pixels respectively belong to the M images and have corresponding positions in the images to which they belong;

[0127] A weight dictionary acquisition module 120 is used to acquire a weight dictionary determined in advance by a neural network model, wherein the weight dictionary contains fusion weights corresponding to different pixel value combinations;

[0128] A weight search module 130, configured to determine the fusion weight corresponding to each target pixel value combination according to the weight dictionary;

[0129] The image fusion module 140 is used to fuse the M images based on the fusion weights corresponding to the target pixel value combinations to obtain a fused target image.

[0130] Optionally, the weight dictionary acquisition module 120 is used to respectively obtain i1 to i M From 0 to N, multiple pixel value combinations (i1,i2,...,i M ), where N is the number of gray levels of the image minus one; inputting the M monochrome images corresponding to each pixel value combination in the multiple pixel value combinations into the neural network model to obtain the fusion weight corresponding to each pixel value combination, where the pixel value combination (i1, i2, ..., i M ) The pixel value of the jth monochrome image in the M monochrome images corresponding to is i j , j is 1 to M; the weight dictionary is determined according to the fusion weight corresponding to each pixel value combination.

[0131] Optionally, the weight dictionary acquisition module 120 is used to input M monochrome images corresponding to each pixel value combination of the multiple pixel value combinations into the neural network model to obtain a weight image corresponding to each pixel value combination; and use the pixel value mean of the weight image corresponding to each pixel value combination as the fusion weight corresponding to the corresponding pixel value combination.

[0132] Optionally, when M=2, the weight image is 1, the fusion weight corresponding to the pixel value combination is the fusion weight corresponding to one pixel value in the pixel value combination, and the fusion weight corresponding to another pixel value in the pixel value combination is 1-the fusion weight corresponding to the pixel value combination.

[0133] Optionally, when M is greater than or equal to 2, the weight image is M sub-weight images of the same size, and the sum of the pixel values ​​of M corresponding pixels in the M sub-weight images is 1; wherein, the M corresponding pixels in the M sub-weight images respectively belong to the M sub-weight images and have a corresponding relationship in their positions in the images to which they belong.

[0134] Optionally, the device 100 further includes:

[0135] A model training module is used to obtain multiple groups of sample images, each group of sample images includes M sub-sample images; obtain standard weight images corresponding to each group of sample images; train an initial neural network model according to the sample images and the standard weight images; when it is determined that the parameters characterizing the completion of the training of the initial neural network model meet preset requirements, obtain the trained neural network model.

[0136] Optionally, the model training module is used to obtain each sample pixel value combination in the M sub-sample images, the sample pixel value combination including pixel values ​​of M corresponding sample pixel points, wherein M is an integer greater than or equal to 2; the M corresponding pixel points respectively belong to the M sub-sample images and have a corresponding relationship in the positions in the images to which they belong; obtain a standard HDR image obtained after the M sub-sample images are fused; for each sample pixel value combination, determine a standard fusion weight corresponding to the sample pixel value combination according to the pixel value of the pixel point corresponding to the sample pixel value combination in the standard HDR image; determine a standard weight image corresponding to the sample image according to the standard fusion weights corresponding to each sample pixel value combination.

[0137] Optionally, the weight search module 130 is used to search the weight dictionary for initial weights corresponding to each target pixel value combination to obtain an initial weight image, which includes each initial weight; filter the initial weight image to obtain a filtered weight image; and obtain fusion weights corresponding to each target pixel value combination from the filtered weight image.

[0138] Optionally, the image fusion module 140 is used to determine the fusion weights corresponding to each of the M pixel points in the target pixel value combination according to the fusion weights corresponding to the target pixel value combination; perform weighted fusion on the M pixel points according to the fusion weights corresponding to each of the M pixel points to obtain pixel values ​​of fused pixel points corresponding to the M pixel points; and determine the fused target image according to the pixel values ​​of each fused pixel point.

[0139] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0140] Please refer to Figure 8 , Figure 8 A structural diagram of an electronic device for executing an image fusion method provided in an embodiment of the present application, the electronic device may include: at least one processor 210, such as a CPU, at least one communication interface 220, at least one memory 230 and at least one communication bus 240. Among them, the communication bus 240 is used to realize direct connection and communication between these components. Among them, the communication interface 220 of the device in the embodiment of the present application is used to communicate signaling or data with other node devices. The memory 230 can be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 230 can optionally also be at least one storage device located away from the aforementioned processor. Computer-readable instructions are stored in the memory 230. When the computer-readable instructions are executed by the processor 210, the electronic device executes the above-mentioned Figure 1 The method process is shown.

[0141] Understandably, Figure 1 The structure shown is for illustration only, and the electronic device may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0142] The present application provides a readable storage medium, wherein when the computer program is executed by a processor, the computer program is executed as follows: Figure 1 The method process in the method embodiment shown is executed by the electronic device.

[0143] The present embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining each target pixel value combination in M ​​images to be fused, each of the target pixel value combination includes pixel values ​​of M corresponding pixels, wherein M is an integer greater than or equal to 2, and the M corresponding pixels belong to the M images respectively and have a corresponding relationship in the positions in the images to which they belong; obtaining a weight dictionary determined in advance by a neural network model, the weight dictionary including fusion weights corresponding to different pixel value combinations; determining the fusion weights corresponding to each of the target pixel value combinations according to the weight dictionary; and fusing the M images based on the fusion weights corresponding to each of the target pixel value combinations to obtain a fused target image.

[0144] In summary, the embodiments of the present application provide an image fusion method, device, electronic device and readable storage medium, which obtain corresponding weight dictionaries for different pixel value combinations through a neural network model, and then obtain the fusion weights corresponding to each target pixel value combination in M ​​images according to the weight dictionary, and then perform image fusion based on the fusion weights. Compared with the method of manually setting weights through experience, the fusion weights obtained in the above method in the present application are more reasonable and have stronger robustness, thereby obtaining better image fusion effects.

[0145] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0146] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0147] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image fusion method, characterized in that: The method comprises: Obtain each target pixel value combination in the M images to be fused, each of the target pixel value combinations including pixel values ​​of M corresponding pixels, wherein M is an integer greater than or equal to 2, and the M corresponding pixels respectively belong to the M images and have corresponding positions in the respective images; Obtaining a weight dictionary determined in advance by a neural network model, wherein the weight dictionary contains fusion weights corresponding to different pixel value combinations; Determine the fusion weight corresponding to each target pixel value combination according to the weight dictionary; Fusing the M images based on the fusion weights corresponding to the target pixel value combinations to obtain a fused target image; Among them, the weight dictionary is determined in the following way: The step of obtaining a weight dictionary determined in advance by a neural network model includes: Take i1 to i respectively M From 0 to N, multiple pixel value combinations (i1, i2, ..., i M ), where N is the number of gray levels of the image minus one; Input the M monochrome images corresponding to each pixel value combination in the multiple pixel value combinations into the neural network model to obtain the fusion weight corresponding to each pixel value combination, wherein the fusion weight corresponding to the pixel value combination (i1, i2, ..., i M ) The pixel value of the jth monochrome image in the M monochrome images corresponding to is i j , j is 1 to M; The weight dictionary is determined according to the fusion weight corresponding to each pixel value combination.

2. The method according to claim 1, characterized in that The step of inputting M monochrome images corresponding to each pixel value combination of the multiple pixel value combinations into the neural network model to obtain a fusion weight corresponding to each pixel value combination includes: Inputting M monochrome images corresponding to each pixel value combination in the multiple pixel value combinations into the neural network model to obtain a weight image corresponding to each pixel value combination; The pixel value mean of the weight image corresponding to each pixel value combination is used as the fusion weight corresponding to the pixel value combination.

3. The method according to claim 2, characterized in that When M=2, the weight image is 1, the fusion weight corresponding to the pixel value combination is the fusion weight corresponding to one pixel value in the pixel value combination, and the fusion weight corresponding to another pixel value in the pixel value combination is 1-the fusion weight corresponding to the pixel value combination.

4. The method according to claim 2, characterized in that: When M is greater than or equal to 2, the weight image is M sub-weight images of the same size, and the sum of the pixel values ​​of the M corresponding pixels in the M sub-weight images is 1; wherein, the M corresponding pixels in the M sub-weight images respectively belong to the M sub-weight images and have a corresponding relationship in their positions in the images to which they belong.

5. The method according to claim 1, characterized in that The neural network model is trained in the following way: A plurality of groups of sample images are obtained, each group of sample images comprising M sub-sample images; Obtain the standard weight image corresponding to each group of sample images; Training an initial neural network model according to the sample image and the standard weight image; When it is determined that the parameters characterizing the completion of the training of the initial neural network model meet the preset requirements, the trained neural network model is obtained.

6. The method according to claim 5, characterized in that The step of obtaining a standard weight image corresponding to each group of sample images includes: Acquire each sample pixel value combination in the M sub-sample images, wherein the sample pixel value combination includes pixel values ​​of M corresponding sample pixel points, where M is an integer greater than or equal to 2; the M corresponding pixel points belong to the M sub-sample images respectively and have corresponding positions in the images to which they belong; Obtain a standard HDR image obtained by fusion of the M sub-sample images; For each sample pixel value combination, determining a standard fusion weight corresponding to the sample pixel value combination according to the pixel value of the pixel point corresponding to the sample pixel value combination in the standard HDR image; A standard weight image corresponding to the sample image is determined according to the standard fusion weights corresponding to each combination of sample pixel values.

7. The method according to any one of claims 1 to 6, characterized in that: The determining, according to the weight dictionary, the fusion weights corresponding to the target pixel value combinations includes: Searching and obtaining the initial weights corresponding to each target pixel value combination from the weight dictionary to obtain an initial weight image, wherein the initial weight image includes each initial weight; Performing filtering on the initial weight image to obtain a filtered weight image; The fusion weights corresponding to each target pixel value combination are obtained from the filtered weight image.

8. The method according to any one of claims 1 to 6, characterized in that: The fusing the M images based on the fusion weights corresponding to the target pixel value combinations to obtain a fused target image includes: Determine the fusion weights corresponding to the M pixels in the target pixel value combination according to the fusion weights corresponding to the target pixel value combination; Performing weighted fusion on the M pixel points according to the fusion weights corresponding to the M pixel points, to obtain pixel values ​​of fused pixel points corresponding to the M pixel points; According to the pixel value of each fused pixel point, the fused target image is determined.

9. An image fusion device, characterized in that: The device comprises: A pixel value acquisition module, used to acquire each target pixel value combination in the M images to be fused, each of the target pixel value combinations including pixel values ​​of M corresponding pixels, wherein M is an integer greater than or equal to 2, and the M corresponding pixels respectively belong to the M images and have corresponding positions in the respective images; A weight dictionary acquisition module is used to acquire a weight dictionary determined in advance by a neural network model, wherein the weight dictionary contains fusion weights corresponding to different pixel value combinations; A weight search module, used to determine the fusion weight corresponding to each target pixel value combination according to the weight dictionary; An image fusion module, used for fusing the M images based on fusion weights corresponding to each target pixel value combination to obtain a fused target image; Wherein, the weight dictionary acquisition module is specifically used for: The step of obtaining a weight dictionary determined in advance by a neural network model includes: Take i1 to i respectively M From 0 to N, multiple pixel value combinations (i1, i2, ..., i M ), where N is the number of gray levels of the image minus one; Input the M monochrome images corresponding to each pixel value combination in the multiple pixel value combinations into the neural network model to obtain the fusion weight corresponding to each pixel value combination, wherein the fusion weight corresponding to the pixel value combination (i1, i2, ..., i M ) The pixel value of the jth monochrome image in the M monochrome images corresponding to is ij, where j is 1 to M; The weight dictionary is determined according to the fusion weight corresponding to each pixel value combination.

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 8 is executed.

11. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is executed.

Images