Image fusion reconstruction method and system based on deep learning

Abstract

The invention relates to an image fusion reconstruction method and system based on deep learning, and relates to the technical field of image fusion reconstruction. The method comprises the following steps: acquiring a source image set and preprocessing the source image set; performing feature extraction on the source image set and collecting edge coordinates of the source image set to obtain an image feature set and an edge coordinate set; determining a feature fusion proportion set of each group according to the image feature set; adjusting the image feature set to the same plane dimension according to the edge coordinate set in combination with the fusion proportion set; and fusing the image feature set according to the feature fusion proportion of each group of images to obtain a fused image. According to the method, based on the advantages of deep learning and a traditional image processing technology, effective fusion of non-registered images is realized through the steps of feature extraction, edge coordinate recording, feature fusion proportion determination, image position adjustment and the like, and the quality and detail retention of the fused images are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion and reconstruction, and in particular to an image fusion and reconstruction method and system based on deep learning. Background Art

[0002] Image fusion reconstruction is a technology that integrates the useful information of multiple images. Through fusion reconstruction, the limitations of a single image can be overcome, such as noise interference, insufficient resolution, and limited viewing angle in some images. Combining the complementary information in different images can obtain a clearer, more comprehensive, and more diagnostic or analytical image, and generate a new image that combines the advantages of each original image.

[0003] In multi-source image fusion, due to differences in shooting angles, distances or equipment, there may be misalignment between source images, that is, there are differences in spatial positions between source images, and they are not precisely aligned to the same coordinate system, which may cause ghosting, blurring or artifacts in the fused image, seriously affecting the image quality and the effects of subsequent tasks. Summary of the invention

[0004] Based on this, it is necessary to provide an image fusion and reconstruction method and system based on deep learning that can achieve effective fusion of non-registered images to address the above problems.

[0005] The present invention provides an image fusion reconstruction method based on deep learning, the method comprising:

[0006] Collecting a source image set and preprocessing the source image set;

[0007] Extract features from the source image set and collect its edge coordinates to obtain an image feature set and an edge coordinate set;

[0008] Determine the fusion weight set of each group of features according to the image feature set;

[0009] The image feature set is adjusted to the same plane dimension according to the edge coordinate set combined with the fusion weight set;

[0010] The image feature sets are fused according to the fusion weight of each group of image features to obtain a fused image.

[0011] In one embodiment, the step of extracting features from a source image set and recording its edge coordinates to obtain an image feature set and an edge coordinate set includes:

[0012] Import the source image into the 2D screen;

[0013] Perform edge detection on the source image in the two-dimensional picture to obtain an edge detection result;

[0014] The source image endpoints are identified based on the edge detection results, and the edge coordinate set of the source image is obtained through analysis.

[0015] In one embodiment, the step of determining the fusion weight of each set of features according to the image feature set includes:

[0016] Determine a first fusion weight of each group of features based on the pixel value of each group of features;

[0017] Determining a second fusion weight of each group of features based on a position of each group of features in the source image;

[0018] The third fusion weight of each group of features is determined based on the resolution of each group of features.

[0019] In one embodiment, the step of determining the first fusion weight of each group of features based on the pixel value of each group of features includes:

[0020] Calculate the pixel value of each group of image features in the image feature set;

[0021] Calculate the feature pixel ratio according to the pixel value of each group of image features;

[0022] The inverse of the image feature pixel ratio is taken as the first fusion weight of each group of image features.

[0023] In one embodiment, the step of determining the second fusion weight of each group of features based on the position of each group of features in the source image further includes:

[0024] According to the position of each group of features in the image in the image feature set;

[0025] Taking the center of the source image as the starting point, calculate the distance of each group of image features from the starting point;

[0026] The second fusion weight of each group of image features is determined according to the distance that each group of image features deviates from the starting point.

[0027] In one embodiment, the step of determining the third fusion weight of each group of features based on the resolution of each group of features further includes:

[0028] According to the resolution of features in the image feature set;

[0029] The third fusion weight of each group of image features is determined according to the resolution of each group of image features.

[0030] In one embodiment, the step of adjusting the image feature set to the same plane dimension according to the edge coordinate set includes:

[0031] transforming edge coordinates of the first set of source images according to a desired layout of image features in the first set of source images in the initial target image to obtain a fused edge coordinate set;

[0032] According to the common image features of the subsequent source image and the real-time target image, determining the starting fusion coordinate points of a group of endpoints of the subsequent source image in the target image;

[0033] According to the difference between the starting fusion coordinate point and the endpoint coordinates corresponding to the subsequent source image, the edge coordinates corresponding to the subsequent source image are converted into the coordinate system of the target image to obtain the initial fusion edge coordinate set;

[0034] The initial fused edge coordinate set is corrected to obtain the target fused edge coordinate set.

[0035] In one embodiment, the step of correcting the initial fused edge coordinate set to obtain the target fused edge coordinate set includes:

[0036] Obtaining initial fusion coordinates of each group of image features in the source image according to the initial fusion edge coordinate set;

[0037] Comparing the initial fusion coordinates of each group of image features in the subsequent source image with the fusion coordinates of the same image features in the real-time target image to obtain the fusion coordinate differences of each group of image features;

[0038] The initial fused edge coordinate set is corrected according to the fused coordinate difference of each group of image features to obtain the target fused edge coordinate set of the subsequent source image.

[0039] In one embodiment, the step of fusing the image feature sets according to the fusion weights of the image feature sets to obtain a fused image includes:

[0040] According to the first fusion weight, the second fusion weight and the third fusion, a fusion pixel value set of each group of image features on a single group of source images is calculated;

[0041] A fused image is obtained according to the fused pixel values ​​of each group of image features on the source image.

[0042] The present invention also provides an image fusion reconstruction system based on deep learning, the system comprising:

[0043] An image acquisition module, used for acquiring a source image set and preprocessing the source image set;

[0044] An analysis module is used to extract features from a source image set and collect its edge coordinates to obtain an image feature set and an edge coordinate set;

[0045] A calculation module, used for determining a fusion weight set of each set of features according to an image feature set;

[0046] An initial fusion module is used to adjust the image feature set to the same plane dimension according to the edge coordinate set combined with the fusion weight set;

[0047] The assignment module is used to fuse the image feature set according to the fusion weight of each group of image features to obtain a fused image.

[0048] The above-mentioned image fusion and reconstruction method based on deep learning combines the advantages of deep learning and traditional image processing technology. Through the steps of feature extraction, edge coordinate recording, feature fusion weight determination and image position adjustment, it realizes the effective fusion of non-registered images and improves the quality and detail retention of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 This is a flow chart of the image fusion reconstruction method based on deep learning proposed by the present invention;

[0051] Figure 2 This is a structural diagram of the deep learning-based image fusion and reconstruction system proposed in the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] In view of the above problems, the following Figure 1-Figure 2 The present invention describes the image fusion reconstruction method and system based on deep learning.

[0054] like Figure 1 As shown, in one embodiment, a deep learning-based image fusion reconstruction method includes:

[0055] Step S100: collecting a source image set and preprocessing the source image set.

[0056] Specifically, according to the application scenario and image quality requirements, select appropriate imaging equipment, and perform pre-processing operations such as denoising, white balance adjustment, and contrast enhancement on the collected source images to improve image quality.

[0057] Step S200 , extracting features from a source image set and collecting edge coordinates thereof to obtain an image feature set and an edge coordinate set.

[0058] Specifically, convolutional neural networks and Transformer are used to extract multi-scale features of the image, and edge detection algorithms are used to record the edge coordinates of the image.

[0059] Optionally, the edge detection algorithm is a Canny operator.

[0060] It should be noted that the convolutional neural network is trained based on a pre-trained model using feature-annotated training samples obtained from public data sets related to the application scenario. For example, you can use keras.applications.vgg16.VGG16 in Keras to load the VGG16 model, and in Keras, set include_top=False to exclude the top classifier and import feature-annotated samples for training.

[0061] The Transformer model can effectively extract the global features of the image through the self-attention mechanism, helping the model to better understand the overall structure and contextual information of the image.

[0062] Step S300, determining the fusion weight set of each group of features according to the image feature set.

[0063] Step S400, adjusting the image feature set to the same plane dimension according to the edge coordinate set combined with the fusion weight set.

[0064] Specifically, a blank image with the same size as the input image is created as the target image to store the real-time fusion result. At the same time, the edge coordinate set is used to adjust the source image to the same plane dimension as the target image to reduce the geometric distortion between the images.

[0065] Step S500, fusing the image feature sets according to the fusion weights of each set of image features to obtain a fused image.

[0066] Specifically, source images are collected by using appropriate imaging equipment, and denoising, white balance adjustment, contrast enhancement and other operations are performed on the collected source images to improve image quality. Convolutional neural networks and Transformer models are used to extract multi-scale features of images, and edge detection algorithms are used in combination to record edge coordinates of images. The proportion of each group of image features in the fusion process is obtained according to the image feature set analysis, and a blank image of the same size as the input image is created as the target image. The source image is adjusted to the same plane dimension as the target image according to the edge coordinate set, and the source image set is fused to the target image according to the determined fusion proportion. During this process, the target image stores the fusion result in real time.

[0067] In one embodiment, the steps of extracting features from a source image set and recording its edge coordinates to obtain an image feature set and an edge coordinate set include:

[0068] Step S210: import the source image into the two-dimensional picture.

[0069] Step S220, performing edge detection on the source image in the two-dimensional picture to obtain an edge detection result.

[0070] Step S230, identifying the endpoints of the source image according to the edge detection result, and analyzing to obtain the edge coordinate set of the source image.

[0071] Specifically, the image is imported into a two-dimensional screen, the imported source image is grayed, and then appropriate threshold parameters are set, and the Canny operator is applied for edge detection. Scan and analyze row by row and column by column based on the obtained edge detection results, starting from the upper left corner of the image, and traversing each pixel point in turn in units of pixels. When an edge pixel point is encountered, for example, a point with a value of 255 in the Canny operator detection result, the surrounding pixels are checked.

[0072] The eight-neighborhood method is used to determine the endpoints, that is, if an edge pixel has only one or a few adjacent edge pixels in its eight-neighborhood, then the point is likely to be an endpoint. For example, if an edge pixel has only one adjacent edge pixel above, below, to the left, to the right, and in the four diagonal directions, then the point is likely to be an endpoint. Then, based on the identified endpoint, the coordinate position of the endpoint in the image is recorded, and the edge coordinate set of the source image is obtained.

[0073] Optionally, the two-dimensional screen is an image fusion software, such as Adobe Photoshop, Fusion, etc.

[0074] In one embodiment, the step of determining the fusion weight of each set of features according to the image feature set includes:

[0075] Step S310: determining a first fusion weight of each group of features based on the pixel value of each group of features.

[0076] Step S320: determining a second fusion weight of each group of features based on the position of each group of features in the source image.

[0077] Step S330: determining the third fusion weight of each group of features based on the resolution of each group of features.

[0078] In this embodiment, the step of determining the first fusion weight of each group of features based on the pixel value of each group of features includes:

[0079] Step S312, calculating the pixel value of each group of image features in the image feature set.

[0080] Step S314, calculating the feature pixel ratio according to the pixel values ​​of each group of image features.

[0081] Step S316, taking the inverse of the image feature pixel ratio as the first fusion weight of each group of image features.

[0082] Specifically, the pixel value of each pixel in the source image corresponding to each group of image features is calculated and accumulated to obtain the total pixel value. For each group of image features, the pixel value of each pixel is accumulated to obtain the feature pixel sum. The feature pixel sum is divided by the pixel sum to obtain the pixel ratio of each group of image features. The inverse of the pixel ratio of each group of image features is used as the first fusion ratio of the group of image features.

[0083] Specifically, by taking the reciprocal as the fusion weight, image features with relatively low pixel values ​​can have a relatively higher weight during fusion, avoiding the fusion result from being excessively biased towards areas with high pixel values, thereby achieving a balanced fusion of features with different intensities.

[0084] In this embodiment, the step of determining the second fusion weight of each group of features based on the position of each group of features in the source image further includes:

[0085] Step S322, based on the position of each group of features in the image feature set.

[0086] Step S324, taking the center of the source image as the starting point, calculating the distance of each group of image features from the starting point.

[0087] Step S326, determining the second fusion weight of each group of image features according to the distance that each group of image features deviates from the starting point.

[0088] Specifically, the center coordinates of the source image and the edge coordinates of each group of image features on the source image are determined according to the edge coordinate set of the source image, and then a calculation formula is used to obtain the distance between the center of each group of image features and the center of the source image, that is, the distance that each group of image features deviates from the starting point.

[0089] For example, the coordinates of the upper left corner (Xs, Ys) and lower right corner (Xe, Ye) of the rectangular area corresponding to a group of image features, then the coordinates of the center point of each group of image features are

[0090] Then the distance calculation formula of this group of image features from the center of the source image is: Wherein, D is the distance that the group of image features deviates from the center of the source image, Xc is the horizontal coordinate of the center of the source image, and Yc is the vertical coordinate of the center of the source image.

[0091] Then the second fusion weight calculation formula of this group of image features is: Wherein, B2 is the second fusion weight value, and D is the distance that the group of image features deviates from the center of the source image.

[0092] It should be noted that the central area of ​​a general image is often the visual focus or the area where important information is more likely to be concentrated. Therefore, the farther the feature is from the center, the lower its importance may be. Therefore, the proportion of image features that are far away is lower.

[0093] In this embodiment, the step of determining the third fusion weight of each group of features based on the resolution of each group of features further includes:

[0094] Step S332, based on the resolution of the features in the image feature set.

[0095] Step S334, determining the third fusion weight of each group of image features according to the resolution of each group of image features.

[0096] Specifically, the calculation formula for the third fusion ratio is B3=F x ×F y , where B3 is the third fusion weight value, F x is the number of pixels of the set of image features in the horizontal direction, F y is the number of pixels of this set of image features in the vertical direction.

[0097] It should be noted that high-resolution features can provide more detail information, so they are given a relatively high weight during fusion to improve the detail clarity of the fused image.

[0098] In one embodiment, the step of adjusting the image feature set to the same plane dimension according to the edge coordinate set includes:

[0099] Step S410 , transforming edge coordinates of the first group of source images according to the desired layout of image features in the first group of source images in the initial target image to obtain a fused edge coordinate set.

[0100] Step S420, determining starting fusion coordinate points of a group of endpoints of the subsequent source image in the target image according to the common image features of the subsequent source image and the real-time target image.

[0101] Step S430 , according to the difference between the starting fusion coordinate point and the endpoint coordinates corresponding to the subsequent source image, the edge coordinates corresponding to the subsequent source image are converted into the coordinate system of the target image to obtain an initial fusion edge coordinate set.

[0102] Step S440: Correct the initial fused edge coordinate set to obtain a target fused edge coordinate set.

[0103] In this embodiment, the step of correcting the initial fused edge coordinate set to obtain the target fused edge coordinate set includes:

[0104] Step S440, obtaining initial fusion coordinates of each group of image features in the source image according to the initial fusion edge coordinate set.

[0105] Step S440 , comparing the initial fusion coordinates of each group of image features in the subsequent source image with the fusion coordinates of the same image features in the real-time target image to obtain the fusion coordinate differences of each group of image features.

[0106] Step S440, correcting the initial fused edge coordinate set according to the fused coordinate difference of each group of image features to obtain a target fused edge coordinate set of a subsequent source image.

[0107] Specifically, based on a specific application scenario, the desired layout of the initial target image is clarified, and for the first group of source images, the edge coordinates of this group of source images are transformed according to the positional relationship of their internal image features under the above desired layout, so that their edge coordinates can be adapted to the coordinate system of the initial target image, thereby obtaining a fused edge coordinate set, and the first group of source images are fused with the initial target image to obtain a real-time target image.

[0108] For subsequent source images, by finding out the common image features between the subsequent source images and the real-time target image, a set of starting fusion coordinate points of endpoints of the subsequent source images in the real-time target image are determined based on these common image features. By first calculating the difference between the starting fusion coordinate points and the endpoint coordinates corresponding to the subsequent source image, the edge coordinates corresponding to the subsequent source image are transformed into the coordinate system of the real-time target image to obtain the initial fusion edge coordinates.

[0109] Subsequently, the coordinate information corresponding to each group of image features in the source image is extracted from the initial fusion edge coordinate set, thereby obtaining the initial fusion coordinates of each group of image features. For each group of image features in the subsequent source image, the same image features in the real-time target image are found, and then the initial fusion coordinates of the group of image features in the subsequent source image are compared with the fusion coordinates of the same image features in the real-time target image, and their differences in the directions of each coordinate axis are calculated to obtain the fusion coordinate difference of each group of image features. According to the fusion coordinate difference, the initial fusion edge coordinate set is corrected to finally obtain the target fusion edge coordinate set.

[0110] In one embodiment, the step of fusing the image feature sets according to the fusion weights of the image feature sets to obtain a fused image includes:

[0111] Step S510, calculating and obtaining a fusion pixel value set of each group of image features on a single group of source images according to the first fusion weight, the second fusion weight and the third fusion.

[0112] Specifically, the calculation formula for the fused pixel values ​​of each group of image features on a single group of source images is W=F×B1+F×B2+F×B3, wherein W is the fused pixel value of a group of image features on a single group of source images, F is a group of image feature pixel values ​​on a single group of source images, B1 is the first fusion weight value of the group of image features, B2 is the second fusion weight value of the group of image features, and B3 is the third fusion weight value of the group of image features.

[0113] Step S520, obtaining a fused image according to the fused pixel values ​​of each group of image features on the source image.

[0114] Specifically, the calculated fused pixel value sets of each group of image features on the single group of source images are assigned to the pixel positions in the real-time target image in a one-to-one correspondence manner to obtain a fused image.

[0115] like Figure 2 As shown, in one embodiment, the present invention further provides an image fusion reconstruction system based on deep learning, the system comprising:

[0116] The image acquisition module 10 is used to acquire a source image set and pre-process the source image set.

[0117] The analysis module 20 is used to extract features from the source image set and collect its edge coordinates to obtain an image feature set and an edge coordinate set.

[0118] The operation module 30 is used to determine the fusion weight set of each group of features according to the image feature set.

[0119] The initial fusion module 40 is used to adjust the image feature set to the same plane dimension according to the edge coordinate set combined with the fusion weight set.

[0120] The value assignment module 50 is used to fuse the image feature set according to the fusion weight of each group of image features to obtain a fused image.

[0121] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] The above-described embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. An image fusion reconstruction method based on deep learning, characterized in that: The method comprises: Collecting a source image set and preprocessing the source image set; Extract features from the source image set and collect its edge coordinates to obtain an image feature set and an edge coordinate set; Determine the fusion weight set of each group of features according to the image feature set; The image feature set is adjusted to the same plane dimension according to the edge coordinate set combined with the fusion weight set; The image feature sets are fused according to the fusion weight of each group of image features to obtain a fused image.

2. The image fusion reconstruction method based on deep learning according to claim 1, characterized in that: The steps of extracting features from a source image set and recording its edge coordinates to obtain an image feature set and an edge coordinate set include: Import the source image into the 2D screen; Perform edge detection on the source image in the two-dimensional picture to obtain an edge detection result; The source image endpoints are identified based on the edge detection results, and the edge coordinate set of the source image is obtained through analysis.

3. The image fusion reconstruction method based on deep learning according to claim 1, characterized in that: The steps of determining the fusion weight of each set of features according to the image feature set include: Determine a first fusion weight of each group of features based on the pixel value of each group of features; Determining a second fusion weight of each group of features based on a position of each group of features in the source image; The third fusion weight of each group of features is determined based on the resolution of each group of features.

4. The image fusion reconstruction method based on deep learning according to claim 3, characterized in that: The step of determining a first fusion weight of each group of features based on the pixel value of each group of features includes: Calculate the pixel value of each group of image features in the image feature set; Calculate the feature pixel ratio according to the pixel value of each group of image features; The inverse of the image feature pixel ratio is taken as the first fusion weight of each group of image features.

5. The image fusion reconstruction method based on deep learning according to claim 3, characterized in that: The step of determining the second fusion weight of each group of features based on the position of each group of features in the source image also includes: According to the position of each group of features in the image in the image feature set; Taking the center of the source image as the starting point, calculate the distance of each group of image features from the starting point; The second fusion weight of each group of image features is determined according to the distance that each group of image features deviates from the starting point.

6. The image fusion reconstruction method based on deep learning according to claim 3, characterized in that: The step of determining the third fusion weight of each group of features based on the resolution of each group of features also includes: According to the resolution of features in the image feature set; The third fusion weight of each group of image features is determined according to the resolution of each group of image features.

7. The image fusion reconstruction method based on deep learning according to claim 1, characterized in that: The step of adjusting the image feature set to the same plane dimension according to the edge coordinate set comprises: transforming edge coordinates of the first set of source images according to a desired layout of image features in the first set of source images in the initial target image to obtain a fused edge coordinate set; According to the common image features of the subsequent source image and the real-time target image, determining the starting fusion coordinate points of a group of endpoints of the subsequent source image in the target image; According to the difference between the starting fusion coordinate point and the endpoint coordinates corresponding to the subsequent source image, the edge coordinates corresponding to the subsequent source image are converted into the coordinate system of the target image to obtain the initial fusion edge coordinate set; The initial fused edge coordinate set is corrected to obtain the target fused edge coordinate set.

8. The image fusion reconstruction method based on deep learning according to claim 7, characterized in that: The step of correcting the initial fused edge coordinate set to obtain the target fused edge coordinate set includes: Obtaining initial fusion coordinates of each group of image features in the source image according to the initial fusion edge coordinate set; Comparing the initial fusion coordinates of each group of image features in the subsequent source image with the fusion coordinates of the same image features in the real-time target image to obtain the fusion coordinate differences of each group of image features; The initial fused edge coordinate set is corrected according to the fused coordinate difference of each group of image features to obtain the target fused edge coordinate set of the subsequent source image.

9. The image fusion reconstruction method based on deep learning according to claim 1, characterized in that: The step of fusing the image feature set according to the fusion weight of each group of image features to obtain a fused image includes: According to the first fusion weight, the second fusion weight and the third fusion, a fusion pixel value set of each group of image features on a single group of source images is calculated; A fused image is obtained according to the fused pixel values ​​of each group of image features on the source image.

10. An image fusion and reconstruction system based on deep learning, applied to an image fusion and reconstruction method based on deep learning according to any one of claims 1 to 9, characterized in that: The system comprises: An image acquisition module, used for acquiring a source image set and preprocessing the source image set; An analysis module is used to extract features from a source image set and collect its edge coordinates to obtain an image feature set and an edge coordinate set; A calculation module, used for determining a fusion weight set of each set of features according to an image feature set; An initial fusion module is used to adjust the image feature set to the same plane dimension according to the edge coordinate set combined with the fusion weight set; The assignment module is used to fuse the image feature set according to the fusion weight of each group of image features to obtain a fused image.

Images