Multi-scale feature adaptive weighting-based multivariate field-of-view image fusion method

Abstract

The invention discloses a multivariate view field image fusion method based on multi-scale feature adaptive weighting, which comprises the steps of adaptive weighting processing of a multivariate scale feature image, rotation transformation and distortion correction of the multivariate scale feature image, deep learning fusion of the multivariate scale feature image, and lightweight processing of a deep learning process. The method is used for realizing image fusion of multiple scenes in a complex terrain environment, weights can be dynamically allocated to scene feature images of multiple scales, fusion of multiple view field images can be better realized through affine transformation and distortion correction, the fusion process is realized through a double-branch deep learning network, and a lightweight network structure design has a good application prospect. And the image fusion efficiency is further improved on the premise of ensuring the multi-element view field image fusion precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and image processing, and particularly relates to a multi-element field of view image fusion method based on multi-scale feature adaptive weighting, which is suitable for multi-scene image fusion in complex terrain environment. BACKGROUND

[0002] Traditional image fusion methods have obvious limitations in processing images with multi-element field of view differences. When facing multi-source images with different resolutions, viewing angles or spectral ranges, these methods often have difficulty in effectively coordinating the feature differences between images. Resolution differences can cause edge blurring or loss of details in the fusion results, while viewing angle differences can cause target deformation or spatial misplacement, and spectral range differences can easily cause noise amplification or color distortion. Due to the lack of adaptability to multi-element field of view differences, traditional fusion methods usually use simple weighted averaging or threshold segmentation strategies, which cannot accurately preserve high-frequency details and low-frequency structures, resulting in discontinuity or artifacts in the visual fusion image. In addition, these methods often rely on manually designed parameters or rules, which are difficult to meet the complex and variable scene requirements, further limiting the quality and reliability of the fusion effect.

[0003] Although the pyramid decomposition-based method can achieve image feature extraction through multi-scale analysis, its computational complexity significantly limits its practical application. This method needs to construct and process multiple levels of pyramid structure, and each layer of filtering and downsampling operation will introduce additional computational burden, resulting in a significant increase in processing time for high-resolution images, which is difficult to meet the real-time requirements. At the same time, although deep learning models show potential in feature learning and adaptive fusion, their training process relies on a large amount of labeled data, which not only has high cost, but also may introduce errors due to subjective factors. In addition, deep learning models are easily affected by changes in data distribution, and their generalization performance decreases significantly when facing new types of scenes or sensor data, which may cause feature extraction bias or fusion result distortion.

[0004] These two methods each have limitations that are difficult to overcome, making it difficult for them to perform effectively in complex and variable practical applications. It is necessary to explore new fusion strategies to balance computational efficiency, data dependency and generalization ability. There is no lightweight deep learning fusion method based on multi-scale feature weighting in the field. SUMMARY

[0005] The application aims to provide a multi-scale feature adaptive weighting-based multi-view field image fusion method, so as to realize the dynamic allocation of weights to multiple scale scene feature images, and better realize the fusion of multiple view field images through affine transformation and distortion correction, and the fusion process is realized through a double-branch deep learning network, and the lightweight network structure design further improves the efficiency of image fusion under the premise of ensuring the accuracy of multi-view field image fusion, so as to solve the problems in the above background technology.

[0006] To achieve the above purpose, the application provides the following technical scheme: a multi-scale feature adaptive weighting-based multi-view field image fusion method, comprising the following steps: Step one: for view field images of different angles and different scales, image analysis and processing of texture levels are carried out, and on the basis of a gray level co-occurrence matrix, various texture features of different scale view field images are calculated, including energy parameters, entropy parameters, contrast parameters and correlation parameters, and then the weights of the images of various scales are calculated according to the parameters; Step two: rotation transformation correction and distortion correction are carried out on view field images with angle rotation and lens distortion; Step three: multi-layer convolution processing and pooling processing are carried out on the view field images whose weights have been calculated in step one and whose rotation transformation correction and distortion correction have been carried out in step two, so as to realize the fusion of multiple view fields; Step four: the fused image in step three is subjected to lightweight processing.

[0007] Preferably, the energy parameter in step one reflects the uniformity of the texture of different scale images, the greater the energy parameter, the rougher the uniformity of the image texture, the smaller the energy parameter, the more delicate the uniformity of the image texture, and the calculation of the energy parameter is: ; The entropy parameter reflects the richness of the texture of different scale images, the greater the entropy parameter, the richer the image texture, the smaller the entropy parameter, the sparser the image texture, and the calculation of the entropy parameter is: ; The contrast parameter reflects the definition of the texture of different scale images, the greater the contrast parameter, the clearer the image texture, the smaller the contrast parameter, the more blurred the image texture, and the calculation of the contrast parameter is: ; The correlation parameter reflects the correlation between the textures of different scale images, the greater the correlation parameter, the greater the correlation between the image textures, the smaller the correlation parameter, the smaller the correlation between the image textures, and the calculation of the correlation parameter is: .

[0008] Preferably, in step two, the image region after rotation transformation is determined based on the four corner points of the multi-scale image that has undergone rotation transformation; Based on the expected position of the corrected image region, the angle of rotation transformation is obtained, and a definite coordinate relationship exists between the two regions. The relationship between the coordinates of each pixel at the transformation position and the rotation angle and image width is as follows: ; Based on the above relationship, the coordinates of the four corner points of the multi-scale image after rotation transformation can be accurately calculated, and then the extreme value span of the image region after rotation transformation can be calculated to obtain the maximum region that needs to be corrected. Based on the correspondence between image regions before and after correction, the image boundary pixels are marked, and then pixel correction is performed one by one starting from the boundary. The relationship between the corrected pixel coordinates and the pixel coordinates that have undergone rotation transformation is as follows: ; Based on the above relationship, by performing the inverse solution, the relationship between the pixel coordinates that underwent the rotation transformation and the corrected pixel coordinates can be obtained as follows: ; The coordinates of the curvature center are calculated based on the magnitude of the curvature produced by the distorted image region. , ), calculate the minimum radius of curvature. and maximum radius of curvature Record the starting angle of curvature occurrence and termination angle Distortion correction is performed using the relationship between polar coordinates and world coordinates, as follows: .

[0009] Preferably, in step three, a VGG-19 CNN network is used to perform deep learning-level multi-scale feature image fusion. The deep learning network contains a total of 16 convolutional layers, 5 pooling layers, and 3 fully connected layers. The first convolutional layer contains 64 3x3 pixel filters; The second convolutional layer contains 64 3x3 pixel filters; The third convolutional layer contains 128 3x3 pixel filters; The fourth convolutional layer contains 128 3x3 pixel filters; The fifth convolutional layer contains 256 3x3 pixel filters; The sixth convolutional layer contains 256 3x3 pixel filters; The seventh convolutional layer contains 256 3x3 pixel filters; The eighth convolutional layer contains 256 3x3 pixel filters; The ninth convolutional layer contains 512 3x3 pixel filters; The tenth convolutional layer contains 512 3x3 pixel filters; The eleventh convolutional layer contains 512 3x3 pixel filters; The twelfth convolutional layer contains 512 3x3 pixel filters; The thirteenth convolutional layer contains 512 3x3 pixel filters; The fourteenth convolutional layer contains 512 3x3 pixel filters; The fifteenth convolutional layer contains 512 3x3 pixel filters; The sixteenth convolutional layer contains 512 3x3 pixel filters; The first pooling layer uses max pooling with a 2*2 pixel template; The second pooling layer uses max pooling with a 2x2 pixel template; The third pooling layer uses max pooling with a 2x2 pixel template; The fourth pooling layer uses max pooling with a 2*2 pixel template; The fifth pooling layer uses max pooling with a 2x2 pixel template.

[0010] Preferably, in step four, MobileNet is used to replace VGG-19 in the original deep learning network to perform lightweight processing on the deep learning fusion network. The convolution process of MobileNet includes depthwise convolution and pointwise convolution. The calculation of depthwise convolution involves the horizontal width parameter, vertical height parameter, third dimension parameter, horizontal width kernel parameter, and vertical height kernel parameter of the multi-scale image, and is calculated as follows: ; The calculation of pointwise convolution involves the horizontal width parameter, vertical height parameter, third dimension parameter, horizontal width kernel parameter, vertical height kernel parameter, and third dimension kernel parameter of the multi-scale image, and is calculated as follows: .

[0011] Compared with the prior art, the beneficial effects of the present invention are: First, in this invention, various texture features of field images at different scales are calculated, including energy parameters, entropy parameters, contrast parameters, and correlation parameters. Then, the weights of images at each scale are calculated based on these parameters, providing weight configurations for images at different scales for subsequent deep fusion of multi-scale field images. This allows images at scales with richer texture features to occupy a higher proportion in the fused image, thereby improving the fusion quality of the field images.

[0012] Secondly, rotation transformation correction and distortion correction are performed on field images with angle rotation and lens distortion. This can eliminate the influence of rotation and distortion on multi-scale feature images, restore scale images that are more consistent with the actual field of view, provide more accurate input for subsequent multi-scale field image deep fusion, and improve the fusion quality of field images.

[0013] Third, for field-of-view images of different angles and scales that have already undergone rotation transformation correction and distortion correction, and the corresponding weights of each scale image, CNN deep fusion based on the VGG framework is adopted. This can give full play to the advantages of deep learning processing such as convolution processing, pooling processing, fully connected processing, and activation processing, and form a better field-of-view fusion image.

[0014] Fourth, a lightweight processing method is used for deep learning fusion networks for multi-scale feature images. MobileNet is used instead of VGG-19 in the original deep learning network. The total computation of the convolution process of MobileNet is the sum of the computation of depthwise convolution and pointwise convolution, which is significantly lower than the computation of conventional convolution, thus achieving the effect of lightweight processing of deep learning process. Attached Figure Description

[0015] Figure 1 This is a flowchart of the adaptive weighting process for multi-scale feature images according to the present invention.

[0016] Figure 2 This is a flowchart of the rotation transformation correction process for multi-scale feature images according to the present invention.

[0017] Figure 3 This is a flowchart of the distortion correction process for multi-scale feature images according to the present invention.

[0018] Figure 4 This is a flowchart of the deep learning fusion process for multi-scale feature images according to the present invention.

[0019] Figure 5 This is a flowchart of the conventional convolution processing in the deep learning process of this invention.

[0020] Figure 6 This is a lightweight processing block diagram of the deep learning process of this invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The multi-scale feature adaptive weighting multi-field image fusion method includes adaptive weighting processing of multi-scale feature images, rotation transformation and distortion correction of multi-scale feature images, deep learning fusion of multi-scale feature images, and lightweight processing of the deep learning process. The adaptive weighted processing of multi-scale feature images, such as Figure 1 As shown, texture-level image analysis and processing are performed on field-of-view images at different angles and scales. Here, based on the gray-level co-occurrence matrix, various texture features of field-of-view images at different scales are calculated, including energy parameters, entropy parameters, contrast parameters, and correlation parameters. Then, the weights of images at each scale are calculated based on these parameters.

[0023] The adaptive weighted processing of multi-scale feature images takes as input images of different angles and scales, and outputs as the weights corresponding to each scale image.

[0024] The aforementioned rotation transformation and distortion correction of multi-scale feature images are performed on field-of-view images that exhibit angular rotation and lens distortion.

[0025] like Figure 2 As shown, rotation transformation correction determines the area to be rotated based on the four corner points of the image, and then performs forward or inverse kinematics based on the coordinate relationship of each pixel point within the rectangular area before and after rotation to recover the pixel point coordinates before the rotation transformation.

[0026] like Figure 3 As shown, distortion correction targets image regions exhibiting curvature. It determines the center coordinates of the curvature, then identifies the corresponding angle based on the boundary of the curvature image, and finally completes the distortion correction using the relationship between polar coordinates and world coordinates. The input to the rotation transformation and distortion correction process for multi-scale feature images are field-of-view images at different scales, and the output is the correction result for each scale image.

[0027] The deep learning fusion of multi-scale feature images, such as Figure 4As shown, multi-layer convolution and pooling processes are applied to images of different angles and scales that have already undergone rotation transformation and distortion correction, along with the weights corresponding to each scale image, to achieve multi-field image fusion. The first layer of convolution and pooling processes includes 64 filters, the second layer includes 128 filters, the third layer includes 256 filters, the fourth layer includes 512 filters, and the fifth layer includes 512 filters.

[0028] After five layers of convolution and pooling, a fused image is formed through fully connected processing and activation processing. The input to the deep learning fusion process of multi-scale feature images is the corrected multi-scale field of view image and its corresponding weights, and the output is the fusion result of the multi-scale feature images.

[0029] The lightweight processing of the deep learning process, such as Figure 5 As shown, a lightweight processing method is applied to the deep learning fusion network for fused images, thereby improving fusion efficiency and reducing computational load.

[0030] Here, MobileNet is used instead of VGG-19 in the original deep learning network. This leverages the advantage of MobileNet's low computational cost for depthwise separable convolution. The convolution process in MobileNet includes both depthwise convolution and pointwise convolution, which significantly reduces the computational cost compared to the conventional convolution in VGG-19.

[0031] A multi-field image fusion method based on multi-scale feature adaptive weighting includes the following steps: Step 1: Based on the feature images at each scale, perform statistical calculations of the gray-level co-occurrence matrix to obtain four parameters that affect the weights of images at different scales in the fusion result, where: The energy parameter reflects the uniformity of image texture at different scales. A larger energy parameter indicates a coarser image texture uniformity, while a smaller energy parameter indicates a finer image texture uniformity. The energy parameter is calculated as follows:

[0032] The entropy parameter reflects the richness of image texture at different scales. A larger entropy parameter indicates richer image texture, while a smaller entropy parameter indicates sparser image texture. The entropy parameter is calculated as follows:

[0033] The contrast parameter reflects the sharpness of image texture at different scales. A larger contrast parameter indicates a sharper image texture, while a smaller contrast parameter indicates a blurrier image texture. The contrast parameter is calculated as follows:

[0034] The correlation parameter reflects the association between image textures at different scales. A larger correlation parameter indicates a stronger association between image textures, while a smaller correlation parameter indicates a weaker association between image textures. The correlation parameter is calculated as follows:

[0035] Step 2: Determine the image region after rotation transformation based on the four corner points of the multi-scale image that has undergone rotation transformation; Based on the expected position of the corrected image region, the angle of rotation transformation is obtained. A definite coordinate relationship exists between the two regions. The relationship between the coordinates of each pixel at the transformation position and the rotation angle and image width is as follows:

[0036] Based on the above relationship, the coordinates of the four corner points of the multi-scale image after rotation transformation can be accurately calculated, and then the extreme span of the image region after rotation transformation can be calculated to obtain the maximum region that needs to be corrected.

[0037] Based on the correspondence between image regions before and after correction, the image boundary pixels are marked, and then pixel correction is performed one by one starting from the boundary. The relationship between the corrected pixel coordinates and the pixel coordinates that have undergone rotation transformation is as follows:

[0038] Based on the above relationship, by performing the inverse solution, the relationship between the pixel coordinates that underwent the rotation transformation and the corrected pixel coordinates can be obtained as follows:

[0039] The coordinates of the curvature center are calculated based on the magnitude of the curvature produced by the distorted image region. , ), calculate the minimum radius of curvature. and maximum radius of curvature Record the starting angle of curvature occurrence and termination angle Distortion correction is performed using the relationship between polar coordinates and world coordinates, as follows:

[0040] Step 3: Based on the weights of the images at each scale in Step 1 and the field-of-view images that have undergone rotation transformation correction and distortion correction in Step 2, a VGG-19 CNN network is used to perform multi-scale convolution and pooling processing on deep learning level multi-scale features to achieve multi-field image fusion. The deep learning network contains a total of 16 convolutional layers, 5 pooling layers, and 3 fully connected layers. The first convolutional layer contains 64 3x3 pixel filters; The second convolutional layer contains 64 3x3 pixel filters; The third convolutional layer contains 128 3x3 pixel filters; The fourth convolutional layer contains 128 3x3 pixel filters; The fifth convolutional layer contains 256 3x3 pixel filters; The sixth convolutional layer contains 256 3x3 pixel filters; The seventh convolutional layer contains 256 3x3 pixel filters; The eighth convolutional layer contains 256 3x3 pixel filters; The ninth convolutional layer contains 512 3x3 pixel filters; The tenth convolutional layer contains 512 3x3 pixel filters; The eleventh convolutional layer contains 512 3x3 pixel filters; The twelfth convolutional layer contains 512 3x3 pixel filters; The thirteenth convolutional layer contains 512 3x3 pixel filters; The fourteenth convolutional layer contains 512 3x3 pixel filters; The fifteenth convolutional layer contains 512 3x3 pixel filters; The sixteenth convolutional layer contains 512 3x3 pixel filters; The first pooling layer uses max pooling with a 2x2 pixel template; The second pooling layer uses max pooling with a 2x2 pixel template; The third pooling layer uses max pooling with a 2x2 pixel template; The fourth pooling layer uses max pooling with a 2*2 pixel template; The fifth pooling layer uses max pooling with a 2x2 pixel template.

[0041] Step 4: Replace VGG-19 in the original deep learning network with MobileNet to take advantage of the low computational cost of MobileNet's depthwise separable convolution. The convolution process of MobileNet includes both depthwise convolution and pointwise convolution. The calculation of depthwise convolution involves the horizontal width parameter, vertical height parameter, third dimension parameter, horizontal width kernel parameter, and vertical height kernel parameter of the multi-scale image, and is calculated as follows:

[0042] The calculation of pointwise convolution involves the horizontal width parameter, vertical height parameter, third dimension parameter, horizontal width kernel parameter, vertical height kernel parameter, and third dimension kernel parameter of the multi-scale image, and is calculated as follows: .

[0043] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-field image fusion method based on multi-scale feature adaptive weighting, characterized in that: Includes the following steps: Step 1: Based on the field images at different angles and scales, perform texture-level image analysis and processing. On the basis of the gray-level co-occurrence matrix, calculate various texture features of the field images at different scales, including energy parameters, entropy parameters, contrast parameters, and correlation parameters. Then, calculate the weights of the images at each scale based on these parameters. Step 2: Perform rotation transformation correction and distortion correction based on the field of view image showing angular rotation and lens distortion; Step 3: Based on the weights of the images at each scale in Step 1 and the field-of-view images that have undergone rotation transformation correction and distortion correction in Step 2, perform multi-layer convolution and pooling processing to achieve multi-field-of-view image fusion. Step 4: Lightweight processing of the fused image from Step 3.

2. The multi-field image fusion method based on multi-scale feature adaptive weighting according to claim 1, characterized in that: The energy parameter in step one reflects the uniformity of image texture at different scales. A larger energy parameter indicates a coarser uniformity of image texture, while a smaller energy parameter indicates a finer uniformity of image texture. The energy parameter is calculated as follows: ； The entropy parameter reflects the richness of image texture at different scales. A larger entropy parameter indicates richer image texture, while a smaller entropy parameter indicates sparser image texture. The entropy parameter is calculated as follows: ； The contrast parameter reflects the sharpness of image texture at different scales. A larger contrast parameter indicates a sharper image texture, while a smaller contrast parameter indicates a blurrier image texture. The contrast parameter is calculated as follows: 。 3. The correlation parameter reflects the association between image textures at different scales. A larger correlation parameter indicates a stronger association between image textures, while a smaller correlation parameter indicates a weaker association. The correlation parameter is calculated as follows: 。 4. The multi-field image fusion method based on multi-scale feature adaptive weighting according to claim 2, characterized in that: In step two, the image region after rotation transformation is determined based on the four corner points of the multi-scale image that has undergone rotation transformation. Based on the expected position of the corrected image region, the angle of rotation transformation is obtained. A definite coordinate relationship exists between the two regions. The relationship between the coordinates of each pixel at the transformation position and the rotation angle and image width is as follows: ； Based on the above relationship, the coordinates of the four corner points of the multi-scale image after rotation transformation can be accurately calculated, and then the extreme value span of the image region after rotation transformation can be calculated to obtain the maximum region that needs to be corrected. Based on the correspondence between image regions before and after correction, the image boundary pixels are marked, and then pixel correction is performed one by one starting from the boundary. The relationship between the corrected pixel coordinates and the pixel coordinates that have undergone rotation transformation is as follows: ； Based on the above relationship, by performing the inverse solution, the relationship between the pixel coordinates that underwent the rotation transformation and the corrected pixel coordinates can be obtained as follows: ； The coordinates of the curvature center are calculated based on the curvature magnitude of the distorted image region. , ), calculate the minimum radius of curvature and maximum radius of curvature Record the starting angle of curvature occurrence and termination angle Distortion correction is performed using the relationship between polar coordinates and world coordinates, as follows: 。 5. The multi-field image fusion method based on multi-scale feature adaptive weighting according to claim 3, characterized in that: In step three, a VGG-19 CNN network is used to perform deep learning-level multi-scale feature image fusion. The deep learning network contains a total of 16 convolutional layers, 5 pooling layers, and 3 fully connected layers. The first convolutional layer contains 64 3x3 pixel filters; The second convolutional layer contains 64 3x3 pixel filters; The third convolutional layer contains 128 3x3 pixel filters; The fourth convolutional layer contains 128 3x3 pixel filters; The fifth convolutional layer contains 256 3x3 pixel filters; The sixth convolutional layer contains 256 3x3 pixel filters; The seventh convolutional layer contains 256 3x3 pixel filters; The eighth convolutional layer contains 256 3x3 pixel filters; The ninth convolutional layer contains 512 3x3 pixel filters; The tenth convolutional layer contains 512 3x3 pixel filters; The eleventh convolutional layer contains 512 3x3 pixel filters; The twelfth convolutional layer contains 512 3x3 pixel filters; The thirteenth convolutional layer contains 512 3x3 pixel filters; The fourteenth convolutional layer contains 512 3x3 pixel filters; The fifteenth convolutional layer contains 512 3x3 pixel filters; The sixteenth convolutional layer contains 512 3x3 pixel filters; The first pooling layer uses max pooling with a 2x2 pixel template; The second pooling layer uses max pooling with a 2x2 pixel template; The third pooling layer uses max pooling with a 2x2 pixel template; The fourth pooling layer uses max pooling with a 2*2 pixel template; The fifth pooling layer uses max pooling with a 2x2 pixel template.

6. The multi-field image fusion method based on multi-scale feature adaptive weighting according to claim 4, characterized in that: In step four, MobileNet is used to replace VGG-19 in the original deep learning network to perform lightweight processing on the deep learning fusion network. The convolution process of MobileNet includes depthwise convolution and pointwise convolution. The calculation of depthwise convolution involves the horizontal width parameter, vertical height parameter, third dimension parameter, horizontal width kernel parameter, and vertical height kernel parameter of the multi-scale image, and is calculated as follows: ； The calculation of pointwise convolution involves the horizontal width parameter, vertical height parameter, third dimension parameter, horizontal width kernel parameter, vertical height kernel parameter, and third dimension kernel parameter of the multi-scale image, and is calculated as follows: 。

Images