Different-size visible light image fusion method and system

Abstract

The invention discloses a different-size visible light image fusion method, and the method comprises the steps: obtaining a plurality of visible light images, determining a candidate region according to the task correlation, and obtaining a union set and an intersection of task regions through a union / intersection builder; carrying out bidirectional candidate geometric transformation, multistage pyramid refinement and sub-pixel optimization on any two images to generate a candidate reference image; respectively calculating a task information loss minimization decision function and a minimum stretching / undistorted completion decision function, and selecting a reference map; determining a global scale map and a local scale map on the reference coordinate system, and dynamically cutting the global scale map and the local scale map to generate an optimal cutting window; and constructing a Laplacian pyramid, introducing a task weight into the calculation content weight of each layer of each dynamically cut image, carrying out layer-by-layer fusion and reconstruction to obtain a fused image, and outputting a fusion result of the visible light images with different sizes. According to the method, the image alignment precision, the information fidelity, the fusion efficiency and the environment robustness are all improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and image processing, in particular to a visible light image size fusion technology, which is particularly suitable for visible light image fusion from different resolutions, different field angles, different imaging devices and different shooting angles. The technology can be applied to intelligent monitoring, unmanned aerial vehicle aerial photography, industrial detection and other scenes requiring high-precision image synthesis and information integration. BACKGROUND

[0002] At present, in the fields of unmanned aerial vehicle aerial photography, intelligent monitoring, industrial detection, etc., multi-source fusion of visible light images has become an important means to improve scene perception capability. However, in actual application, the visible light images collected by different camera devices (including lenses of different models and different focal lengths) have significant differences in resolution, field angle, pixel density and shooting angle, etc. This size difference makes it difficult for images to correspond directly in spatial position and geometric shape, and direct fusion will cause geometric misplacement, deformation and detail loss, etc. Therefore, how to realize high-precision visible light size image fusion while maintaining detail fidelity and geometric consistency has become a technical problem to be solved.

[0003] Existing visible light image fusion methods of varying sizes can be broadly categorized into three types: data-level fusion, feature-level fusion, and decision-level fusion. The principles and applications of these three methods are described below: Data-level fusion (early fusion): This method directly fuses the raw data from the sensors. Generally, data-level fusion first spatially aligns images from different modalities using geometric transformations or interpolation methods, and then fuses the image data. While this method is simple and intuitive, direct fusion of the original images often leads to information loss due to significant differences in resolution, spectral response, etc., especially under the influence of irregular light intensity and environmental factors, making it difficult to effectively preserve detailed information. For example, traditional bilinear interpolation or nearest-neighbor interpolation methods are prone to information distortion when handling differences in image size. Feature-level fusion (deep fusion): Unlike data-level fusion, feature-level fusion typically extracts features from images from each sensor using neural networks (such as convolutional neural networks, CNNs), and then fuses them in the feature space. This method effectively improves adaptability to complex scenes by extracting and fusing features from different modalities. A typical approach involves extracting multi-scale features using methods such as Convolutional Neural Networks (CNNs) and pyramid feature extraction, then performing weighted fusion, stitching, or attention-based fusion in the feature space, and finally reconstructing the fused image using a decoder. These methods can mitigate the impact of resolution differences on fusion to some extent and enhance adaptability to complex scenes by utilizing deep features. However, size alignment is usually still required before feature fusion; if the registration accuracy is insufficient, the fusion result will still suffer from blurring and misalignment. Decision-level fusion (post-fusion): In this type of method, each image independently performs tasks such as object detection and segmentation, and then the task outputs are fused. For example, the detection results of two images are merged after coordinate transformation, or the final result is generated at the segmentation level through logical operations, voting, or other strategies. This type of method has lower computational cost and higher flexibility, but because it loses original pixel-level information, it cannot improve detail in the fused result, and the final accuracy is easily affected by the detection errors of a single image.

[0004] While the existing three types of methods have achieved the fusion of visible light images of different sizes to some extent, they still have the following limitations when facing the actual image fusion requirements of multi-resolution and multi-field-of-view differences, especially for application scenarios such as edge / cloud intelligence:

[0005] 1. Lack of task-related drive: Most existing image fusion methods adopt a unified global processing flow, without prioritizing and optimizing key information regions based on specific task requirements (such as detection, recognition, etc.). This may result in the weakening or even loss of details in task-related regions during the fusion process.

[0006] 2. Insufficient deformation control: Existing fusion methods mostly rely on simple scaling, that is, when unifying images of different sizes to the same size, they generally rely on simple global scaling, which lacks constraints on the geometric proportions of the target, easily causing deformation distortion and affecting the accuracy of subsequent visual tasks.

[0007] 3. Single resolution scaling strategy: Existing methods mostly use a globally consistent scaling ratio without adaptively adjusting the resolution according to the importance of the region. This results in wasted computing resources in non-critical areas, while details in critical areas are damaged due to insufficient resolution.

[0008] 4. The cropping method is not intelligent enough: Most methods use fixed windows or simple boundary cropping, and fail to dynamically generate the optimal cropping range based on the union and intersection of the task area, which can easily lead to the target being truncated or the background being redundant.

[0009] 5. Insufficient alignment accuracy and robustness: Even with the introduction of pyramid hierarchical search or affine transformation, it is still difficult to achieve high-precision global and local alignment when there are large scale differences, viewpoint changes or geometric distortions in the image. This can easily lead to problems such as local misalignment, texture mismatch and fusion artifacts.

[0010] To address these issues, existing research has proposed various approaches to image fusion, such as frequency-domain-based fusion algorithms, multimodal sensor collaborative processing frameworks, and cross-modal fusion strategies for infrared and visible light. These methods have shown some effectiveness in improving image quality and enhancing multi-source sensing capabilities, but overall, cross-modal fusion remains the primary research focus, lacking systematic exploration of visible light image fusion for images of the same modality but different sizes.

[0011] Existing methods typically focus on image alignment, feature extraction and fusion, and post-processing of the fusion results. However, these methods often rely on globally uniform scaling or geometric transformations, failing to adequately consider target deformation control and the priority preservation of task-relevant regions, easily leading to detail loss or target scale distortion. Furthermore, resolution scaling strategies are mostly simplistic, failing to adaptively adjust according to the importance of different regions in the scene, resulting in insufficient resolution in critical areas while wasting computational resources in non-critical areas. In addition, existing cropping methods are generally simple, making it difficult to dynamically generate the optimal fusion region, easily causing target truncation or redundant background. A more prominent issue lies in alignment accuracy and robustness. Although some methods introduce multi-scale pyramid search or affine transformations, it is still difficult to guarantee global and local consistency in the presence of large scale differences, viewpoint changes, or even geometric distortions, ultimately leading to misalignment, texture mismatch, or artifacts. These problems not only affect the visual quality of the fused image but also weaken its application in downstream tasks (such as detection, recognition, and tracking).

[0012] In summary, existing technologies still have significant shortcomings in addressing the problem of fusion of visible light images of different sizes, lacking a comprehensive method that can simultaneously achieve task-driven processing, adaptive resolution adjustment, intelligent cropping, and high-precision geometric alignment. How to achieve efficient and robust fusion applicable to complex scenarios while ensuring detail fidelity and geometric consistency remains a core technical challenge that urgently needs to be addressed in this field.

[0013] Although image fusion technology has been widely used in various image processing fields such as computer vision and multi-view imaging, existing technologies still have many problems when handling the fusion of "visible light images of different sizes," failing to meet the requirements for fusion accuracy, target fidelity, and safe region cropping. Specifically, existing technologies have the following problems and shortcomings:

[0014] 1) Lack of task-relevance-driven fusion strategy: Existing fusion methods mostly adopt uniform weighting processing of the entire image, ignoring the importance of key task regions (such as target regions), resulting in blurred details or information loss in these regions after fusion. This is mainly because fusion algorithms focus on visual effects or overall balance, without incorporating the needs of downstream tasks (such as object detection) into the fusion decision. This invention constructs a target union / intersection-oriented fusion decision function to ensure minimal information loss in task regions and prioritize the preservation of details in key regions.

[0015] 2) Lack of deformation control mechanism: Common scale unification strategies often employ simple global scaling, leading to deformation and aspect ratio distortion in the target region, thus affecting the performance of the recognition algorithm. This is because existing methods do not constrain the target's geometric structure, nor do they use deformation as an evaluation metric. This invention introduces a "minimum stretching decision function," selecting the image with the lowest deformation cost as a benchmark, and using non-distortion information completion to maintain the consistency of the target structure.

[0016] 3) Single resolution scaling strategy: Traditional methods often use uniform interpolation to process the entire image, which fails to guarantee details in key areas and wastes computational resources in irrelevant areas. The root cause of the problem lies in the lack of an identification and response mechanism for the importance of regions. This invention proposes a "global + local partition scaling strategy," which uses high-precision interpolation for the task area while using a low-resource scaling method for the background area, achieving a balance between efficiency and quality.

[0017] 4) Lack of intelligent cropping methods: Current cropping methods commonly used in fusion workflows employ fixed windows or bounding boxes, which easily truncate the target or introduce redundant background, reducing the practicality of fusion. This is mainly due to the lack of ability to dynamically crop using the task region structure. This invention achieves a balance between complete target coverage and maximum background removal by dynamically generating the optimal cropping window based on the target union / intersection and its importance graph.

[0018] 5) Insufficient Alignment Accuracy and Geometric Robustness: Even with the introduction of multi-level pyramid registration or affine transformation strategies in some techniques, local misalignment, texture mismatch, or fusion artifacts still occur when faced with significant scale differences, viewpoint distortion, or perspective distortion. This is because these methods lack multi-level fine-grained optimization mechanisms and adaptive adjustments for local deformations. To address this, this invention constructs a fine-grained alignment process of "multi-level pyramid + affine transformation + local optimization search," achieving sub-pixel-level alignment and improving global and local geometric consistency and robustness.

[0019] Therefore, addressing the four major shortcomings of visible light images of different sizes—inaccurate alignment, insufficient control of geometric deformation, severe information loss, and lack of intelligence in cropping strategies—is of great practical significance. There is an urgent need to research a novel task-driven method and system for the fusion and alignment of visible light images of different sizes. Summary of the Invention

[0020] To address the problems of inaccurate alignment of visible light images of different sizes, insufficient control of geometric deformation, severe information loss, and lack of intelligence in cropping strategies in existing technologies, a task-driven visible light image fusion and alignment method and system for edge / cloud applications are proposed.

[0021] In a first aspect, embodiments of this application provide a method for fusing visible light images of different sizes, the method comprising:

[0022] Task region extraction and importance mapping steps: acquire multiple visible light images, determine candidate regions based on task relevance, generate a task weight map, and use a union / intersection builder to obtain the union and intersection of task regions; for any two images, perform bidirectional candidate geometric transformation and multi-level pyramid refinement and sub-pixel optimization to generate a candidate baseline map.

[0023] Dual decision function evaluation and benchmark selection steps: Calculate the decision function that minimizes task information loss and the decision function that minimizes stretching / non-distortion completion for each candidate benchmark map, and select the benchmark map;

[0024] Resolution scaling and dynamic cropping steps: Based on the pixel density of the reference map and the task weight map, determine the global scale map and the local scale map on the reference coordinate system, use high-precision interpolation in the task area and lightweight interpolation in the non-task area, and dynamically crop the global scale map and the local scale map to generate the optimal cropping window.

[0025] Multi-scale weighted fusion and reconstruction steps: Construct a Laplacian pyramid, calculate the content weight of each layer of each dynamically cropped image and introduce the task weight, perform layer-by-layer fusion and reconstruction to obtain the fused image, and output the fusion result of the visible light image of different sizes.

[0026] In this embodiment of the invention, the above-mentioned task region extraction and importance mapping steps include:

[0027] Input acquisition steps: Acquire two or more visible light images;

[0028] Task region extraction and importance mapping steps: Obtain task region masks on each image through task model or algorithm, calculate importance weight map for each image, and use union / intersection builder to obtain the union and intersection of task regions in a unified reference coordinate system;

[0029] Candidate geometric alignment step: For any two images, generate bidirectional candidate geometric transformations;

[0030] Multi-level pyramid refinement and sub-pixel optimization steps: The calculation results of bidirectional candidate geometric transformations are coarse-to-fine optimized from top to bottom in each layer of the Gaussian / Laplacian pyramid to minimize alignment error, and sub-pixel interpolation and small-range parameter search are used for fine-tuning in the last layer.

[0031] In this embodiment of the invention, the above-mentioned dual decision function evaluation and benchmark graph selection steps include:

[0032] Union / Intersection builder calculation steps: On a unified coordinate system of all images, perform union and AND logical operations on the input task mask to form the union and intersection of task regions; the union of task regions is the coverage area of ​​the task regions of all images, used for dynamic cropping to ensure that all task targets are covered; the intersection of task regions is the common area of ​​the task regions of all images, used to evaluate reliability / alignment accuracy and assist the decision function in selecting the reference image.

[0033] The calculation steps for the decision function to minimize task information loss are as follows: When any image is selected as the reference image, the evaluation focuses on maximizing the task information that can be retained before fusion within the intersection of task regions and minimizing information loss.

[0034] The calculation steps for the minimum stretch / non-distortion completion decision function are as follows: When any image is selected as the reference image, the deformation cost imposed on the non-reference image for alignment is evaluated, and isotropic, area-conserving, and low-deformation completion is encouraged.

[0035] Reference image selection steps: Determine the reference image by combining targets, and transform and register the non-reference images to the reference coordinate system.

[0036] In this embodiment of the invention, the above-mentioned resolution scaling and dynamic cropping steps include:

[0037] Resolution scaling steps: Based on the pixel density of the baseline map and the task weight map, determine the global scale and local scale maps on the baseline coordinate system. High-precision interpolation is used for the task area, and lightweight interpolation is used for the non-task area.

[0038] Dynamic cropping steps: Based on the union and intersection of task regions and error boundaries, an optimal cropping window is generated to ensure the integrity of the target and reduce redundant background.

[0039] In this embodiment of the invention, the above-mentioned multi-level pyramid refinement steps include:

[0040] The steps for constructing a multi-layer pyramid are as follows: Each image involved in the registration is sampled separately to form a pyramid from coarse to fine. The Gaussian / Laplacian pyramid is used to obtain the image pairs and their pixel domains for each layer, which serve as input for subsequent layer-by-layer optimization.

[0041] Coarse-layer initial estimation steps: Feature point extraction and matching (ORB / SIFT / AKAZE, etc.) are performed in the coarsest layer, combined with outlier removal (RANSAC) to obtain the initial geometric transformation. The aforementioned It can be one of similarity, affine, homography (perspective), or thin plate spline (TPS) or a cascade thereof, depending on the geometric complexity of the scene. For the coarse layer of the multi-layer pyramid, feature point extraction and feature matching are performed, outliers are removed, and the initial geometric transformation matrix is ​​obtained;

[0042] Optimization steps from coarse to fine layers: In the first... layer Using the convergent transformation parameters of the previous layer as initial values ​​for this layer, the image to be registered is mapped to the reference coordinate system through this transformation. In the intersection domain... Internal calculation of weighted alignment error objective function

[0043]

[0044] in For pixel weights, For gradient operators; : respectively represent the first Aspect (candidate benchmark) and the first Input visible light image; : vector Norms are used to improve robustness to changes in illumination and local occlusion. To be Mapped to Geometric transformations of a coordinate system, whose parameter vectors are denoted as... At each pyramid level, at least one numerical optimization method, such as gradient descent, Gauss-Newton, or Levenberg-Marquardt, is used to minimize... To update It is generated and continuously updated by the "candidate geometry alignment (S4 step) - multi-level pyramid refinement (S5 step)" process until the convergence threshold is met. Or a maximum number of iterations limit.

[0045] The finest layer subpixel optimization steps are as follows: receive the converged transformation results, use high-precision interpolation in the highest resolution layer, and search for each alignment parameter until the fine alignment reaches the preset threshold.

[0046] Local refinement steps: If non-rigid differences exist, constraints are applied to avoid excessive deformation;

[0047] Output the final transformation step: obtain the globally optimal geometric transformation matrix.

[0048] In this embodiment of the invention, the above-mentioned task information loss minimization decision function calculation step includes:

[0049] (1) Intersection region selection: After unifying the symbols of the two images to be registered, the intersection task region is obtained based on the task mask, and the intersection is mapped to the reference coordinate system, denoted as . ;

[0050] (2) Definition of detail intensity: For the potential amount of detail or structure that can be preserved in the image of the intersection task region, in... For each pixel Define details / structural strength

[0051]

[0052] in, For the first Image; For gradient operators; For the Laplace operator; The weights satisfy the local entropy. .

[0053] (3) Resampling attenuation modeling: Using a resampling attenuation factor, a differentiable attenuation function for the resampling scale is constructed, preserving the predictable attenuation:

[0054]

[0055] in, For local equivalent scale, The local equivalent rotation angle (both are derived from the transformation) (obtained by Jacobi polar decomposition). , .

[0056] (4) Calculation of task importance weights: The task weights are determined based on the task model output and image statistics. :

[0057]

[0058] in For task confidence (detection / segmentation / saliency probability); To normalize to The operator; the weights satisfy In the absence of a task model, it can be made .

[0059] (5) Upper bound of detail to be retained and actual amount to be retained: for each pixel definition

[0060]

[0061]

[0062] in, Indicates the reference coordinates Mapping to Image Geometric transformations, .

[0063] (6) Decision function construction: Based on the principle of minimizing the loss of intersection information, the decision function that minimizes the loss of task information is constructed as follows:

[0064]

[0065] in, To prevent zero constant, Index for candidate benchmark maps, For another graph index; The reference coordinates are pixels; For the task intersection on the baseline map The set of pixels on; Defined as above. For each pixel in the intersection region, calculate the non-negative difference between "ideal reachable detail" and "details preserved after resampling," and then average this difference according to task weights. A smaller value indicates a higher selection rate. The smaller the loss of task-related information when using it as a baseline.

[0066] In this embodiment of the invention, the calculation step of the minimum stretching / non-distortion completion decision function includes:

[0067] (1) Calculation of local deformation measurement: Let the image be Registration to reference map The geometric transformation is Its pixel The Jacobian matrix at point is:

[0068]

[0069] Singular value decomposition yields Define the cost of local deformation:

[0070]

[0071] in The first term measures anisotropic stretching, and the second term measures local area change. (This is a local area scale factor), and the third term measures the smoothness of the deformation field. The local deformation metric is calculated based on the Jacobian and singular values ​​of the image transform.

[0072] (2) Equivalent scale and rotation extraction: Local equivalent scale and rotation are obtained from polar decomposition.

[0073]

[0074] in This represents the rotated portion of Jacobi; and It can also be used for resampling-fidelity modeling.

[0075] (3) Decision function based on the principle of minimizing deformation cost: in the task intersection region Above, construct the minimum stretch / non-distortion completion decision function:

[0076]

[0077] in, For the cost of deformation The weighted normal average on the chart, the smaller the value, the better when the chart is selected. When using a reference and registering other images to its coordinate system, the completion process is closer to an isotropic, area-conserving, and smooth "non-distortion" transformation. Deformation-sensitive weights are used to emphasize regions sensitive to geometric distortion.

[0078]

[0079] As the weight of task importance, To normalize the input to The operator. Suitable for task-free models. , To prevent zero constant. To "put the first Transform the image to the first When the image is in pixels The cost of local deformation at the point of origin. It uses the singular values ​​of the transformed Jacobian. To measure three things: whether it is isotropic, whether the area is conserved, and whether the deformation is smooth:

[0080]

[0081] in: For the tradeoff coefficients; the first term is the anisotropy penalty: when This term is zero when scaling proportionally / purely rotating; the greater the deviation, the greater the penalty. The second term is the area preservation penalty: when... The value is zero when the local area remains constant, used to suppress density distortion caused by excessive magnification / compression; the third term is a smoothing penalty. for The spatial gradient (obtained at the current pyramid level) constrains the spatial variation of the deformation field to be smooth, avoiding folding, tearing, and jagged edges. Represents a local approximate rigid body (without stretching, scaling, and smooth deformation); The larger the value, the stronger the required stretching / compression / torsion, and the greater the geometric distortion.

[0082] Secondly, embodiments of this application provide a heterogeneous visible light image fusion system, employing the heterogeneous visible light image fusion method described above. The system includes:

[0083] Task Region Extraction and Importance Mapping Module: Acquire multiple visible light images, determine candidate regions based on task relevance, generate a task weight map, and use a union / intersection builder to obtain the union and intersection of task regions; for any two images, perform bidirectional candidate geometric transformation and multi-level pyramid refinement and sub-pixel optimization to generate a candidate baseline map;

[0084] Dual Decision Function Evaluation and Benchmark Map Selection Module: Calculates the decision function that minimizes task information loss and the decision function that minimizes stretching / non-distortion completion for candidate benchmark maps, and selects the benchmark map;

[0085] Resolution scaling and dynamic cropping module: Based on the pixel density of the reference map and the task weight map, the global scale map and local scale map are determined on the reference coordinate system. High-precision interpolation is used in the task area and lightweight interpolation is used in the non-task area. The global scale map and local scale map are dynamically cropped to generate the optimal cropping window.

[0086] Multi-scale weighted fusion and reconstruction module: Constructs a Laplacian pyramid, calculates the content weights for each layer of each dynamically cropped image, introduces task weights, performs layer-by-layer fusion and reconstruction to obtain a fused image, and outputs the fusion results of visible light images of different sizes.

[0087] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a method for fusing heterogeneous visible light images.

[0088] Fourthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the heterogeneous visible light image fusion method as described above.

[0089] Compared with existing technologies, it has the following outstanding advantages:

[0090] 1. The method and system of this invention propose an image alignment method for images of different sizes based on minimum deformation constraints.

[0091] The primary technical point of this invention lies in constructing an alignment decision function with minimum deformation constraints. Addressing the problem that existing methods, relying solely on global scaling or affine transformations, easily introduce geometric distortion, this method establishes a deformation cost model during the alignment process. This model quantitatively evaluates the degree of deformation for different alignment schemes and automatically selects the scheme with the minimum deformation as the optimal alignment path. This approach not only ensures the consistency of the overall structure at the macroscopic level but also reduces the risk of distortion at the object's edges and contours at the microscopic level.

[0092] Technical effects:

[0093] 1) Achieve high-precision alignment of images across different sizes, avoiding the stretching and compression caused by traditional scaling methods;

[0094] 2) Maintain the geometric proportions of the target object to the greatest extent possible to ensure the reliability of subsequent target detection and recognition;

[0095] 3) Improve the robustness of the alignment process in complex environments, enabling the system to be stably applied to various task scenarios.

[0096] 2. The method and system of this invention propose a task-driven information retention and selection mechanism.

[0097] This invention introduces a task-related decision function into the image fusion process, performing union and intersection analysis on the target regions in the input images. In the intersection region, by calculating information entropy and semantic contribution, the fusion scheme with the least information loss is automatically selected to ensure that information in task-related regions is preserved first. In the union region, an adaptive weighting mechanism is adopted to maximize the utilization of complementary information from different images.

[0098] Technical effects:

[0099] 1) Ensure that information in key target areas is not discarded, and improve the effectiveness of the fusion results for the task;

[0100] 2) Significantly improves accuracy and robustness in downstream tasks such as detection, recognition, and segmentation;

[0101] 3) Implement task-driven information optimization to avoid the loss of key task information caused by the "average treatment" in traditional methods.

[0102] 3. The method and system of this invention propose a region-adaptive resolution scaling mechanism.

[0103] To address the problem that existing technologies generally employ global uniform scaling while neglecting the differences in information across different regions, this invention proposes a region-adaptive resolution scaling mechanism. By analyzing the spatial distribution and task relevance of image content, it maintains higher resolution for key target regions while appropriately compressing redundant background regions, thereby achieving differentiated scaling processing. This mechanism can be implemented through a multi-scale feature pyramid network, ensuring a balance between accuracy and efficiency under limited overall computational resources.

[0104] Technical effects:

[0105] 1) Maintaining high-resolution details in key target areas significantly improves recognition and detection accuracy;

[0106] 2) By compressing low-value areas, the overall computational load and storage requirements are reduced;

[0107] 3) Achieve a balance between high precision and high efficiency, adapting to scenarios with limited computing power, such as edge computing.

[0108] 4. The method and system of this invention propose a dynamic pruning method based on union and intersection analysis.

[0109] This invention proposes a dynamic cropping strategy based on task region union and intersection analysis. By modeling the spatial distribution of target regions in multiple input images, it automatically generates the optimal cropping boundary. Unlike existing fixed-window cropping, this method can adaptively adjust the cropping range according to the position and shape of the task object in the image. This ensures the integrity of the task target while removing redundant background, thereby improving fusion efficiency.

[0110] Technical effects:

[0111] 1) Ensure that key target areas are fully preserved and avoid being cropped or truncated; 2) Remove invalid background to reduce the interference of redundant information on the fusion result; 3) Improve the signal-to-noise ratio and task effectiveness of the fused image and enhance the performance of downstream tasks.

[0112] 4. The method and system of this invention propose a robust alignment mechanism driven by multi-level feature fusion.

[0113] Existing technologies are prone to alignment errors when there are significant differences in viewpoint or local distortions. This invention proposes a multi-level feature fusion alignment method that refines local features layer by layer based on global alignment. By fusing global structural features and local detail features through multi-scale convolution and attention mechanisms, robust end-to-end alignment is achieved.

[0114] Technical effects:

[0115] 1) Significantly reduces misalignment and artifacts in complex scenes, improving alignment stability; 2) Balances global structure and local details, ensuring that the fused image is both macroscopically accurate and rich in detail; 3) Enhances adaptability to multi-source images of different sizes, enabling the method to be extended to large-scale application scenarios. Attached Figure Description

[0116] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0117] Figure 1 This is a schematic diagram of the heterogeneous visible light image fusion method of the present invention;

[0118] Figure 2 This is a schematic diagram of the overall process of an embodiment of the present invention;

[0119] Figure 3 This is a schematic diagram of the union / intersection builder according to an embodiment of the present invention;

[0120] Figure 4 This is a schematic diagram of the multi-level pyramid registration process according to an embodiment of the present invention;

[0121] Figure 5 This is a schematic diagram of union / intersection driven clipping according to an embodiment of the present invention;

[0122] Figure 6 This is a schematic diagram of the heterogeneous visible light image fusion system of the present invention;

[0123] Figure 7 This is a schematic diagram of the computer hardware of the present invention. Detailed Implementation

[0124] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0125] It should also be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0126] It should also be understood that, in various embodiments of the present invention, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0127] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0128] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0129] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0130] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0131] To make the above-mentioned features and effects of the present invention clearer and easier to understand, specific embodiments are described below in conjunction with the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are merely illustrative. The scope of protection of the present invention is not limited to the disclosed embodiments, but is defined by the appended claims.

[0132] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0133] While conducting research on heterogeneous image fusion, the inventors discovered some key problems that cannot be solved in existing technologies, especially in terms of task-driven detail fidelity, structural consistency, efficient processing, and accurate cropping in heterogeneous image fusion using only visible light.

[0134] Technical challenge 1: Lack of task-related fusion strategies

[0135] Existing technologies for multi-source image fusion mostly employ a uniform global processing flow, neglecting the priority preservation requirements of key information regions based on specific task needs (such as object detection, classification, or semantic segmentation). Through analysis, the inventors confirmed that the root cause lies in the lack of a task-aware fusion decision mechanism in existing technologies; the fusion process fails to dynamically adjust weights based on the task importance of the target region. The technical challenge lies in ensuring that information in task-relevant regions is not weakened or lost while maintaining the overall fusion effect. The inventors discovered that by constructing a task-relevance decision function, the union and intersection operations of the task target regions of the input image can be performed, and the fusion scheme with the least information loss can be selected from the intersection, thus achieving priority preservation of key information. This solution introduces a task-driven region-priority mechanism on the basis of traditional feature-level fusion methods, effectively improving the fidelity of the target region and the accuracy of subsequent tasks.

[0136] Technical challenge 2: Insufficient deformation control leads to geometric distortion of the target.

[0137] In their experiments aligning visible light images of varying sizes, the inventors discovered that existing methods commonly employ global scaling to unify the size when input images have significant resolution differences. This leads to the stretching or compression of the target, resulting in geometric deformation and impacting detection and recognition accuracy. This is because existing technologies lack a quantitative evaluation mechanism for deformation costs during size normalization, considering scaling effects only at the pixel level without regard to the integrity of the target's geometric structure. The technical challenge lies in simultaneously satisfying the dual constraints of spatial alignment requirements and geometric proportion preservation. Through multiple rounds of experimental research, the inventors discovered that by establishing a minimum deformation decision function, the impact of different scaling schemes on the target shape can be quantified, and the image with the smallest deformation can be automatically selected as the benchmark for scaling. This method is an improvement on existing geometric registration techniques, incorporating task-aware deformation constraints to achieve a balance between alignment accuracy and geometric fidelity.

[0138] Technical challenge 3: Single resolution scaling method

[0139] While researching multi-resolution visible light image fusion, the inventors discovered that existing methods often employ a globally consistent scaling ratio, leading to wasted computational resources in non-critical areas while losing details in critical areas due to insufficient resolution. This is because existing technologies lack adaptive analysis capabilities regarding the importance of regions during image resolution scaling, failing to optimize resolution allocation strategies based on the distribution of task objectives. The technical challenge lies in how to simultaneously ensure compression efficiency in non-critical areas and preserve detail in critical areas under limited computational resources. Through in-depth research, the inventors discovered that this deficiency can be addressed by introducing a region-specific adaptive scaling mechanism. This mechanism dynamically allocates pixel density during resolution adjustment, assigning higher resolution to task-critical areas while moderately compressing background or redundant areas. This solution represents an improvement on the traditional uniform resolution scaling strategy, achieving a balance between computational efficiency and information fidelity through differentiated scaling guided by regional importance.

[0140] Technical challenge 4: The cutting method lacks intelligence.

[0141] While conducting experiments on the fusion of visible light images of different sizes, the inventors discovered that existing methods generally employ fixed windows or simple boundary cropping, failing to intelligently determine the cropping range based on the spatial distribution of the task region. This often results in the truncation of key targets or the introduction of a large amount of irrelevant background. The reason lies in the lack of a dynamic cropping strategy that combines the position and shape of the task region, failing to fully utilize the union and intersection characteristics of the task region. The technical challenge lies in minimizing the introduction of invalid background while ensuring the integrity of the target. Through research, the inventors discovered that a dynamic cropping algorithm based on the union and intersection of task regions can be designed to dynamically generate the optimal cropping range according to the spatial distribution relationship of target regions in multiple images, thereby maximizing the preservation of key information and reducing background redundancy. This solution is a task-aware optimization based on traditional cropping strategies, effectively improving the task effectiveness of the fusion results.

[0142] Technical challenge 5: Insufficient alignment accuracy and robustness

[0143] In experiments on visible light image registration at multiple views and resolutions, the inventors discovered that even with pyramid layer-by-layer search or affine transformation, existing methods struggle to achieve high-precision global and local alignment under conditions of significant scale differences, viewpoint variations, or geometric distortions. This often results in local misalignment, texture mismatch, and fusion artifacts. The reason for this lies in the lack of multi-level alignment strategies that combine global and local features, and insufficient adaptability to scale and distortion during feature extraction and matching. The technical challenge lies in balancing global structural consistency with local detail accuracy during alignment. The inventors discovered that this problem can be solved through a multi-level feature fusion registration method: first, preliminary geometric transformation is performed using global features, then the alignment is refined by combining local features, and finally, optimization is performed layer by layer within a multi-scale pyramid. This approach extends traditional registration methods, significantly improving alignment robustness and accuracy under conditions of large scale differences and distortions.

[0144] This invention proposes a task-driven visible light image fusion and alignment method for edge / cloud applications, which effectively overcomes the shortcomings of existing technologies in the following aspects:

[0145] 1. Image Alignment Problem of Different Sizes: Existing technologies often rely solely on global scaling or simple affine transformations when aligning visible light images at different resolutions or viewpoints. This leads to stretching, compression, and other deformations in the target area, affecting the geometric consistency and recognition accuracy of the object. This invention proposes a geometric correction algorithm based on minimum deformation constraints. During the alignment process, a deformation cost function is constructed to automatically select the image with the smallest deformation as the benchmark, achieving high-precision, low-distortion image alignment.

[0146] 2. Insufficient Task Relevance: Existing technologies lack a task-aware mechanism during the fusion process, often treating different regions equally, which can easily lead to the loss of information in key target areas. This invention introduces a task-driven decision function to perform union and intersection analysis on the target regions of the input image, and selects the solution with the least information loss in the intersection, ensuring that task-related information is preferentially preserved.

[0147] 3. Problem of a single resolution scaling strategy: Traditional methods typically use a uniform scaling ratio when processing images of varying sizes, failing to accommodate the different resolution requirements of different regions. This invention addresses this issue by employing a region-adaptive resolution scaling mechanism, allocating higher resolution to critical task areas while appropriately compressing background areas, thereby improving overall processing efficiency while ensuring the fidelity of key information.

[0148] 4. Lack of intelligent cropping: Existing methods often use fixed windows or simple boundary cropping in the cropping step before image fusion, which often leads to the truncation of key targets or the introduction of a large amount of irrelevant background. This invention proposes a dynamic cropping method based on the intersection of task regions, which automatically generates the optimal cropping range according to the spatial distribution relationship of the target region, ensuring the integrity of the target while reducing redundant background.

[0149] 5. Insufficient alignment robustness: In complex scenes, such as those with significant viewpoint differences or local distortions, existing alignment techniques often result in misalignment and artifacts. This invention addresses this issue by employing a multi-level feature fusion alignment method, combining global and local features to optimize alignment accuracy layer by layer, significantly improving robustness and fusion quality in complex scenes.

[0150] The system of this application embodiment will be described in detail below with reference to specific embodiments:

[0151] Example 1

[0152] like Figure 1 and Figure 2 As shown in the figure, this invention proposes a fusion method for visible light images of different sizes, the method comprising:

[0153] Task Region Extraction and Importance Mapping Step 101: Acquire multiple visible light images, determine candidate regions based on task relevance, generate a task weight map, and use a union / intersection builder to obtain the union and intersection of task regions; for any two images, perform bidirectional candidate geometric transformation and multi-level pyramid refinement and sub-pixel optimization to generate a candidate baseline map;

[0154] Dual Decision Function Evaluation and Benchmark Map Selection Step 102: Calculate the decision function that minimizes task information loss and the decision function that minimizes stretching / non-distortion completion for the candidate benchmark maps, and select the benchmark map;

[0155] Resolution scaling and dynamic cropping step 103: Based on the pixel density and task weight map of the reference map, determine the global scale map and local scale map on the reference coordinate system, use high-precision interpolation in the task area and lightweight interpolation in the non-task area, and dynamically crop the global scale map and local scale map to generate the optimal cropping window.

[0156] Multi-scale weighted fusion and reconstruction step 104: Construct the Laplacian pyramid, calculate the content weight of each layer of each dynamically cropped image, introduce the task weight, perform layer-by-layer fusion and reconstruction to obtain the fused image, and output the fusion result of the heterogeneous visible light image.

[0157] Specifically, in a specific embodiment of this invention, a fusion method for visible light images of varying sizes is proposed. Its core objective is to address the problems of inaccurate alignment, fusion distortion, and lack of task relevance in decision-making in existing technologies. To achieve this objective, this invention introduces task-constrained decision function design, multi-level geometric and affine transformations, and adaptive resolution scaling and cropping mechanisms. Through these innovative designs, this invention not only ensures that the image retains structural and detail integrity during the fusion process but also minimizes the risk of object deformation and decreased recognition accuracy due to stretching, scaling, or cropping. The overall process is as follows: Figure 2 As shown:

[0158] I. Overall Process

[0159] S1 Input Acquisition: Acquire two or more visible light images. , of which The image size is Pixel pitch / equivalent resolution is The corresponding pixel domain is Pixel positions are unified based on This indicates, and if necessary, specifies the coordinate system to which it belongs.

[0160] S2 Task Region Extraction and Importance Mapping: This involves processing multiple input visible light images. Through task models or algorithms (Including but not limited to object detection, semantic segmentation, saliency detection, or combinations thereof) Obtain task region masks on each image. ,in For each image Importance weights are calculated based on statistical measures such as task confidence, gradient strength, and local entropy. It can be obtained by combining statistical measures such as task confidence, gradient strength, and local entropy, and is used to measure the contribution of a pixel to subsequent fusion and task objectives. A unified reference coordinate system is established. When selecting a benchmark map in subsequent operations, any candidate benchmark map can be selected. And apply the current estimated geometric transformation to all masks and weight maps. Mapping to the reference coordinate system, we get And in a unified reference coordinate system Next, perform set operations on the mapped task mask to obtain the union and intersection of the task regions: ,in, Used for dynamic cropping to ensure complete coverage of the target. The weighted summation field is used for subsequent alignment accuracy evaluation and benchmark selection. For example... Figure 3 The "Union / Intersection Builder" shown in the image. The output of this step includes... , , , .

[0161] S3 Candidate Geometric Alignment (Bidirectional / Multidirectional): For any two images Constructing bidirectional candidate geometric transformations: (Will Mapped to coordinate system) and (Reverse). In multi-graph scenarios, candidate baseline graphs are used. Established for reference This transformation model maps reference coordinates to an image. Geometric transformations (similar / affine / homography / TPS or cascaded). Initial values ​​for candidate transformations are obtained through local feature extraction and matching (e.g., ORB / SIFT / AKAZE) combined with outlier removal (e.g., RANSAC). It can be performed at coarse pyramid layers or the original resolution to enhance robustness. Consistency and reversibility regularization are introduced into the bidirectional transformation to reduce initial value bias and improve registration stability. For example... Figure 4 As shown, this is the multi-level pyramid registration process.

[0162] S4 Multi-level Pyramid Refinement and Sub-pixel Optimization: Constructing a Gaussian / Laplacian multi-level pyramid and performing top-down geometric transformations on candidate pixels. The process involves coarse-to-fine parameter optimization, updating the transformation parameters to minimize the alignment error objective function. At the finest layer, high-precision interpolation is used, along with a small-range parameter search (rotation, scaling, translation, shearing, etc.) to achieve sub-pixel-level fine-tuning until a preset convergence threshold is met. This step outputs the converged result. .

[0163] S5 Dual Decision Function Evaluation and Benchmark Map Selection: In the Intersection Region Calculate separately:

[0164] Decision function that minimizes task information loss ;

[0165] Minimum stretching / non-distortion completion decision function .

[0166] The baseline diagram is determined based on the combined objective (Equation (11)). (See the relevant chapter for specific expressions and weightings). The non-reference image is then processed... Register to the reference coordinate system.

[0167] S6 Resolution Scalability (Global + Local): Based on Baseline Map pixel density Using the task weight graph, determine the global scale on the reference coordinate system. With local scale map (Equation (12)–(13)), high-precision interpolation (such as Lanczos / bicubic) is used in the task region, and lightweight interpolation (such as bilinear / regional averaging) is used in the non-task region to achieve regional adaptive resolution configuration and computational overhead control.

[0168] S7 Dynamic Clipping: Based on , And the registration error safety boundary, to generate the optimal clipping window. (can be the minimum bounding rectangle, rotated rectangle or convex hull) (Equation (14)), to achieve complete coverage of the task target and remove redundant background as much as possible, thereby improving fusion efficiency and signal-to-noise ratio.

[0169] S8 Multi-scale Weighted Fusion and Reconstruction: A Laplacian pyramid is constructed, and the content weights of each layer are calculated and task weights are introduced (Equations (15)–(16)). Under the combined effect of these weights, layer-by-layer weighted fusion is performed, and reconstruction is completed from top to bottom to obtain the fused image. .

[0170] S9 Output and Optional Post-processing: Output Fusion Results .

[0171] In this embodiment of the invention, the above-mentioned task region extraction and importance mapping step 101 includes:

[0172] Input acquisition steps: Acquire two or more visible light images;

[0173] Task region extraction and importance mapping steps: Obtain task region masks on each image through task model or algorithm, calculate importance weight map for each image, and use union / intersection builder to obtain the union and intersection of task regions in a unified reference coordinate system;

[0174] Candidate geometric alignment step: For any two images, generate bidirectional candidate geometric transformations;

[0175] Multi-level pyramid refinement and sub-pixel optimization steps: The calculation results of bidirectional candidate geometric transformations are coarse-to-fine optimized from top to bottom in each layer of the Gaussian / Laplacian pyramid to minimize alignment error, and sub-pixel interpolation and small-range parameter search are used for fine-tuning in the last layer.

[0176] In this embodiment of the invention, the above-mentioned dual decision function evaluation and benchmark graph selection step 102 includes:

[0177] Union / Intersection builder calculation steps: On a unified coordinate system of all images, perform union and AND logical operations on the input task mask to form the union and intersection of task regions; the union of task regions is the coverage area of ​​the task regions of all images, used for dynamic cropping to ensure that all task targets are covered; the intersection of task regions is the common area of ​​the task regions of all images, used to evaluate reliability / alignment accuracy and assist the decision function in selecting the reference image.

[0178] The calculation steps for the decision function to minimize task information loss are as follows: When any image is selected as the reference image, the evaluation focuses on maximizing the task information that can be retained before fusion within the intersection of task regions and minimizing information loss.

[0179] The calculation steps for the minimum stretch / non-distortion completion decision function are as follows: When any image is selected as the reference image, the deformation cost imposed on the non-reference image for alignment is evaluated, and isotropic, area-conserving, and low-deformation completion is encouraged.

[0180] Reference image selection steps: Determine the reference image by combining targets, and transform and register the non-reference images to the reference coordinate system.

[0181] In this embodiment of the invention, the resolution scaling and dynamic cropping step 103 includes:

[0182] Resolution scaling steps: Based on the pixel density of the baseline map and the task weight map, determine the global scale and local scale maps on the baseline coordinate system. High-precision interpolation is used for the task area, and lightweight interpolation is used for the non-task area.

[0183] Dynamic cropping steps: Based on the union and intersection of task regions and error boundaries, an optimal cropping window is generated to ensure the integrity of the target and reduce redundant background.

[0184] In this embodiment of the invention, the above-mentioned multi-level pyramid refinement steps include:

[0185] The steps for constructing a multi-layer pyramid are as follows: Each image involved in the registration is sampled separately to form a pyramid from coarse to fine. The Gaussian / Laplacian pyramid is used to obtain the image pairs and their pixel domains for each layer, which serve as input for subsequent layer-by-layer optimization.

[0186] Coarse-layer initial estimation steps: Feature point extraction and matching (ORB / SIFT / AKAZE, etc.) are performed in the coarsest layer, combined with outlier removal (RANSAC) to obtain the initial geometric transformation. The aforementioned It can be one of similarity, affine, homography (perspective), or thin plate spline (TPS) or a cascade thereof, depending on the geometric complexity of the scene. For the coarse layer of the multi-layer pyramid, feature point extraction and feature matching are performed, outliers are removed, and the initial geometric transformation matrix is ​​obtained;

[0187] Optimization steps from coarse to fine layers: In the first... layer Using the converged transformation parameters of the previous layer as initial values ​​for this layer, perform a geometric transformation on the current layer image to align it with the result of the previous layer; in the intersection domain... Internal calculation of weighted alignment error objective function

[0188]

[0189] in For pixel weights, For gradient operators; : respectively represent the first Aspect (candidate benchmark) and the first Input visible light image; : vector Norms are used to improve robustness to changes in illumination and local occlusion. To be Mapped to The geometric transformation of a coordinate system is denoted by its parameter vector. At each pyramid level, at least one numerical optimization method among gradient descent, Gauss-Newton, or Levenberg-Marquardt is used to minimize... To update It is generated and continuously updated by the "candidate geometry alignment (S4 step) - multi-level pyramid refinement (S5 step)" process until the convergence threshold is met. Or a maximum number of iterations limit.

[0190] The finest layer subpixel optimization steps are as follows: receive the converged transformation results, use high-precision interpolation in the highest resolution layer, and search for each alignment parameter until the fine alignment reaches the preset threshold.

[0191] Local refinement steps: If non-rigid differences exist, constraints are applied to avoid excessive deformation;

[0192] Output the final transformation step: obtain the globally optimal geometric transformation matrix.

[0193] In this embodiment of the invention, the above-mentioned task information loss minimization decision function calculation step includes:

[0194] (1) Intersection region selection: After unifying the symbols of the two images to be registered, the intersection task region is obtained based on the task mask, and the intersection is mapped to the reference coordinate system, denoted as . ;

[0195] (2) Definition of detail intensity: For the potential amount of detail or structure that can be preserved in the image of the intersection task region, in... For each pixel Define details / structural strength

[0196]

[0197] in, For the first Image; For gradient operators; For the Laplace operator; The weights satisfy the local entropy. .

[0198] (3) Resampling attenuation modeling: Using a resampling attenuation factor, a differentiable attenuation function for the resampling scale is constructed, preserving the predictable attenuation:

[0199]

[0200] in, For local equivalent scale, The local equivalent rotation angle (both are derived from the transformation) (obtained by Jacobi polar decomposition). , .

[0201] (4) Calculation of task importance weights: The task weights are determined based on the task model output and image statistics.

[0202]

[0203] in For task confidence (detection / segmentation / saliency probability); To normalize to The operator; the weights satisfy In the absence of a task model, it can be made .

[0204] (5) Upper bound of detail to be retained and actual amount to be retained: for each pixel definition

[0205]

[0206]

[0207] in, Indicates the reference coordinates Mapping to Image Geometric transformations, .

[0208] (6) Decision function construction: Based on the principle of minimizing the loss of intersection information, the decision function that minimizes the loss of task information is constructed as follows:

[0209]

[0210] in, To prevent zero constant, Index for candidate benchmark maps, For another graph index; The reference coordinates are pixels; For the task intersection on the baseline map The set of pixels on; Defined as above. For each pixel in the intersection region, calculate the non-negative difference between "ideal reachable detail" and "details preserved after resampling," and then average this difference according to task weights. A smaller value indicates a higher selection rate. The smaller the loss of task-related information when using it as a baseline.

[0211] In this embodiment of the invention, the calculation step of the minimum stretching / non-distortion completion decision function includes:

[0212] (1) Calculation of local deformation measurement: Let the image be Registration to reference map The geometric transformation is Its pixel The Jacobian matrix at point is

[0213]

[0214] Singular value decomposition yields Define the cost of local deformation.

[0215]

[0216] in The first term measures anisotropic stretching, and the second term measures local area change. (This is a local area scale factor), and the third term measures the smoothness of the deformation field. The local deformation metric is calculated based on the Jacobian and singular values ​​of the image transform.

[0217] (2) Equivalent scale and rotation extraction: Local equivalent scale and rotation are obtained from polar decomposition.

[0218]

[0219] in This represents the rotated portion of Jacobi; and It can also be used for resampling-fidelity modeling.

[0220] (3) Decision function based on the principle of minimizing deformation cost: in the task intersection region Above, construct the minimum stretch / non-distortion completion decision function.

[0221]

[0222] in, For the cost of deformation The weighted normal average on the chart, the smaller the value, the better when the chart is selected. When using a reference and registering other images to its coordinate system, the completion process is closer to an isotropic, area-conserving, and smooth "non-distortion" transformation. Deformation-sensitive weights are used to emphasize regions sensitive to geometric distortion.

[0223]

[0224] As the weight of task importance, To normalize the input to Operators (preferably in the region) (Normalization is performed based on minimum / maximum values ​​or quantiles). For models without a task, the following can be used: , To prevent zero constant. It is "to put the first Transform the image to the first When the image is in pixels The cost of local deformation at the point of origin. It uses the singular values ​​of the transformed Jacobian. To measure three things: whether it is isotropic, whether the area is conserved, and whether the deformation is smooth:

[0225]

[0226] in: For the tradeoff coefficients; the first term is the anisotropy penalty: when This term is zero when scaling proportionally / purely rotating; the greater the deviation, the greater the penalty. The second term is the area preservation penalty: when... The value is zero when the local area remains constant, used to suppress density distortion caused by excessive magnification / compression; the third term is a smoothing penalty. for The spatial gradient (obtained at the current pyramid level) constrains the spatial variation of the deformation field to be smooth, avoiding folding, tearing, and jagged edges. Represents a local approximate rigid body (without stretching, scaling, and smooth deformation); The larger the value, the stronger the required stretching / compression / torsion, and the greater the geometric distortion.

[0227] Specifically, in a specific embodiment of the present invention, the above two decision functions include:

[0228] II. Two Major Decision Functions

[0229] 1. Decision function that minimizes task information loss

[0230] Objective: When selecting the first When using a map as a baseline, the evaluation focuses on the intersection area. Maximize the amount of task information that can be retained before internal fusion (equivalent to minimizing information loss).

[0231] Firstly, when fusing multiple images of varying sizes, the task region (such as detected objects or segmented defects) is often distributed across different locations in different images, and each image has different resolutions and viewpoints. This leads to two core problems:

[0232] Union (U)

[0233] Meaning: "Coverage area" of all image task regions.

[0234] Function: Used for dynamic cropping to ensure all target areas are covered. It guarantees that no important regions detected in any image are missed.

[0235] Example scenario: One image detects "crack A" and another image detects "crack B". Only by taking the union can we ensure that both A and B are retained in the final fusion result.

[0236] Intersection (C)

[0237] Meaning: The "common area" of all image task areas.

[0238] Function: Used to evaluate reliability / alignment accuracy, and assists in the decision function (selection of a baseline image). Provides an area consistently covered by multiple images, typically with low noise and high reliability.

[0239] Example scenario: If all perspectives consistently detect "critical defect X", the intersection area is often the most reliable reference area.

[0240] Therefore, the union guarantees no omissions, and the intersection guarantees reliability.

[0241] like Figure 3 As shown, in a specific embodiment of the present invention, a "union / intersection builder" module is proposed:

[0242] First, unify the coordinate system of all images;

[0243] Then the task mask Perform logical operations ;

[0244] This results in two mask images: and This is used for subsequent pruning (S7) and decision functions (S5).

[0245] Ensuring both the integrity of the target and the robustness of the fusion is a key step in avoiding omissions and misjudgments in multi-source image fusion.

[0246] (1) Unified Symbols

[0247] Let the input set of visible light images be . , No. The pixel domain of the image is The selected candidate benchmark number is... Then, in the reference coordinate system The calculations are performed within the reference frame. Two images defined on the reference frame are expressed as follows:

[0248]

[0249] in The reference pixel position; : To map the reference pixel x to the first pixel... Differentiable geometric transformations of the image coordinates (one of similarity / affine / homography / TPS or a cascade thereof), whose parameter vectors are denoted as... The pixel values ​​used for mapping employ sub-pixel interpolation (bicubic or Lanczos). The task intersection region is denoted as... The task mask of all images is obtained through their respective... Mapped to I later obtained it.

[0250] (2) Details / Structure "Potential Retainable Amount"

[0251] Define detail intensity:

[0252]

[0253] in For gradient norm, For Laplace, Local entropy; This metric measures local texture / edge / complexity; a higher value indicates a greater potential for "preservable" detail.

[0254] (3) Resampling attenuation factor

[0255] The resampling scale exhibits a predictable decrease in detail retention; a differentiable decay function can be constructed as follows:

[0256]

[0257] in It is a local equivalent scale (given by the Jacobi singular value, see equation (8)). For local equivalent rotation quantization; These are the weighting coefficients. Range of values . and The calculation is shown in equation (8).

[0258] (4) Task importance weight

[0259] Define the k-th task weight graph on the baseline.

[0260]

[0261] in For task model Pixel-level confidence scores for (object detection / semantic segmentation / saliency, etc.); This indicates that linear normalization is performed within the current image / layer. ; When no task model exists, take... Pixels that are "more important and clearer to the task" should be given higher weight. .

[0262] (5) The "upper bound that can retain details" in the intersection is in the intersection Within this framework, the estimation for each pixel can preserve details:

[0263]

[0264] Ideal Upper Limit:

[0265]

[0266] in and Calculated using equation (2) respectively; This is given by equation (3). If the data from the previous sample can still be preserved after resampling... The high level of detail, near . From equation (2) in the first Detail intensity calculated on the image; : From equation (2) in the first Image on, position The level of detail calculated at the location (using subpixel interpolation); The resampling attenuation factor given by equation (3) is used to estimate the resampling attenuation factor of the first... width Fidelity loss when mapped to a reference system.

[0267] (6) Decision function (the smaller the better)

[0268]

[0269] in, To prevent zero constant, Index for candidate benchmark maps, For another graph index; The reference coordinates are pixels; For the task intersection on the baseline map The set of pixels on; This represents the task importance weight. For each pixel in the intersection region, the non-negative difference between "ideal reachable detail" and "details preserved after resampling" is calculated and then averaged according to the task weight. A smaller value indicates a higher priority image. The smaller the loss of task-related information when using it as a baseline.

[0270] Intuitively, if we take Resampling to Even in the intersection, it can still retain high detail. So small; therefore, choose To minimize the loss of intersection information.

[0271] 2. Minimum stretching / non-distortion completion decision function

[0272] Objective: When selecting the first When using the map as a reference, assess the deformation cost imposed on non-reference maps for alignment, and encourage isotropic, area-conserving, and low-deformation completion.

[0273] (7) Local deformation measurement setting transformation Jacobi is singular value Define the cost of local deformation:

[0274]

[0275] in The weighting factors are as follows: the first term penalizes anisotropic stretching (deviating from proportionality), the second term penalizes area change (deviating from area conservation), and the third term penalizes uneven deformation field (to prevent folding / tearing).

[0276] (8) Local equivalent scale extraction is obtained from polar decomposition or SVD.

[0277]

[0278] in for Rotational components (polar decomposition) (or obtained by one of the left and right orthogonal factors of SVD). Take the principal value (radians). This is used for the detailed attenuation assessment of equation (3).

[0279] (9) Decision function (the smaller the better)

[0280]

[0281] in, For the cost of deformation The weighted normal average on the chart, the smaller the value, the better when the chart is selected. When using a reference and registering other images to its coordinate system, the completion process is closer to an isotropic, area-conserving, and smooth "non-distortion" transformation. Deformation-sensitive weights are used to emphasize regions sensitive to geometric distortion.

[0282]

[0283] As the weight of task importance, Can be with Share the source, or emphasize the edges of slender targets (e.g., increase the skeletal weight of the Xichang structure). To normalize the input to The operator. Suitable for task-free models. , To prevent zero constant. It is "to put the first Transform the image to the first When the image is in pixels The cost of local deformation at the point of origin. It uses the singular values ​​of the transformed Jacobian. To measure three things: whether it is isotropic, whether the area is conserved, and whether the deformation is smooth:

[0284]

[0285] in: For the tradeoff coefficients; the first term is the anisotropy penalty: when This term is zero when scaling proportionally / purely rotating; the greater the deviation, the greater the penalty. The second term is the area preservation penalty: when... The value is zero when the local area remains constant, used to suppress density distortion caused by excessive magnification / compression; the third term is a smoothing penalty. for The spatial gradient (obtained at the current pyramid level) constrains the spatial variation of the deformation field to be smooth, avoiding folding, tearing, and jagged edges. Represents a local approximate rigid body (without stretching, scaling, and smooth deformation); The larger the value, the stronger the required stretching / compression / torsion, and the greater the geometric distortion.

[0286] 3. Combination selection criteria and alignment error

[0287] (10) Alignment error (used for S4 thinning)

[0288] The present invention in candidate reference map Within the coordinate domain, construct a weighted alignment error objective function for multi-level pyramid refinement:

[0289]

[0290] in, Reference coordinate system Pixel position within; This is the area where tasks intersect. The pixel value of the reference image at this coordinate; For the first Geometric transformation of the image Mapped and resampled pixel values ​​obtained by subpixel interpolation; ∇( ) is a first-order gradient operator (Sobel / Scharr or equivalent implementation); The L1 norm of the vector; ∈[0,1] represents the pixel weight.

[0291] (11) Combined objectives of benchmark selection

[0292] To simultaneously ensure fidelity in task details and low geometric deformation, the present invention constructs a combination of objectives based on the chosen criteria:

[0293]

[0294] in, As a task information loss index, the difference between the retainable details and the ideal upper bound is measured in the intersection region according to the aforementioned formula (6); The minimum stretch / non-distortion completion index is used to measure the local deformation cost in the intersection region according to the aforementioned equation (9). Wherein, This is a weighting factor used to adjust the relative importance of the two types of indicators: The larger the scale, the more emphasis is placed on ensuring the fidelity of task details; The smaller the value, the greater the emphasis on geometric distortion-free operation. The appropriate value can be selected based on the scenario in the implementation example. The range of values ​​for which the value is taken does not constitute a limitation on the claims. When different candidates... of Difference less than threshold In this case, the lexicographical order rule is preferred for discrimination: first compare... If they are still ranked equally, then compare them. Alternatively, in scenarios where geometric accuracy is prioritized, the reverse order can be used.

[0295] III. Resolution scaling and dynamic cropping

[0296] (12) Global scale

[0297] To unify the overall pixel density of images from different sources in a reference coordinate system, this invention sets a global scale factor. Let the equivalent pixel density of the reference image be... Align to uniform working density :

[0298]

[0299] Within the intersection median(⋅) is the median operator to improve robustness to outliers; For the first The equivalent pixel density of an image in its original coordinate domain; Intersection area The equivalent scale mean within the range, Transformation Definition; obtained by taking the median It can suppress the interference of extreme densities on a uniform scale. The value is a single real number, and the entire image is scaled at the same scale to ensure the overall geometric consistency of multi-source fusion.

[0300] (13) Local scale map

[0301] To enhance detail in critical mission regions while suppressing redundancy in non-critical regions, this invention constructs a local scale map in the reference coordinate domain.

[0302]

[0303] in Reference coordinate system Pixel position within; The task importance weighting graph is obtained by combining task confidence, gradient strength, and local entropy. For hierarchical linear normalization, the input is mapped to... ; For the baseline diagram The detailed intensity at a given point is obtained by weighting the gradient norm, Laplace magnitude, and local entropy. This is the ideal upper bound for the source details at corresponding positions within the intersection; To maintain stability and avoid zero denominator; Used to amplify differences in detail at low and medium intensity levels and suppress maxima; This is a truncation function. To maximize local gain and constrain amplification; The intensity coefficient is the local magnification factor; when Larger and When relatively insufficient Triggering limited micro-amplification, otherwise maintaining Or slightly compressed; the final pixel-level scaling factor is This ensures both the resolution of key areas and the control of the overall computational load.

[0304] (14) Dynamic clipping window

[0305] like Figure 5 As shown, to reduce invalid background while ensuring complete coverage of the task objective, this invention generates a dynamically cropped window based on the task union. Let... Union The region mapped to the reference coordinates is then subjected to shape dilation. For safety boundaries (including positioning error / PSF margin), the minimum circumscribed rectangle / polygon is required. ( ):

[0306]

[0307] To broaden the safety boundary scale to cover registration residuals, positioning errors, and point spread function (PSF); the resulting It directly serves as the working field for subsequent interpolation, fusion, and reconstruction, achieving a balance between preventing the target from being truncated and maintaining a compact computational domain. A rotating rectangle (minimum area) or a convex hull can be selected as the clipping boundary.

[0308] IV. Multi-scale weight fusion and reconstruction

[0309] (15) The content weight of the hierarchy in the Laplace / Gaussian pyramid is... Layer, for the first The input image defines a pixel-level content metric in the first... Layers define the content metrics for each image:

[0310]

[0311] in, For this layer of coordinates Pixel position within; For the first The image in the first A Gaussian pyramid diagram with multiple layers; It is a first-order gradient operator (Sobel / Scharr or equivalent implementation). For the gradient vector Norm; The magnitude of the Laplace operator; For local entropy (in) (Implemented using histograms / integral graphs over the neighborhood). For non-negative weights of content metrics, the preferred option is to satisfy... Its value is used to make a trade-off among the three types of information: texture, edge, and complexity, and does not constitute a limitation on the claims.

[0312] On the same floor With the same pixel At the location, a temperature-controlled device is used. By integrating content metrics with task importance, a normalized weight for cross-source competition is obtained:

[0313]

[0314] in, In the source index Dimensional normalization ensures ; To compute the task importance weight map in the reference coordinate domain (see the implementation of Equation (4) above), each layer is aligned to the target coordinates using bilinear / bicubic interpolation. ; This is the task enhancement factor, used to adjust the proportion of task weight in this layer; for Temperature coefficient The larger the value, the "sharper" the weight distribution (more biased towards the optimal source). The smaller the value, the more "smooth" (multi-source equilibrium). For stable values, calculations can be performed... Subtract time To avoid index overflow.

[0315] (16) Integration and Reconstruction The layer linearly fuses the Laplace components of each source using normalized weights:

[0316]

[0317] in For the first The image in the first The Laplace subband of the layer; the corresponding Gaussian components of the bottom layer (or near-DC layer) are fused into

[0318]

[0319] in For each source, the lowest-level Gaussian component is represented. The top-down reconstruction uses a standard Laplace pyramid recursive reconstruction: let the highest-level index be... ,make If the layer is a fused Gaussian, then

[0320]

[0321] Final fused image

[0322]

[0323] in Upsampling is used in conjunction with low-pass filtering; mirror / copy extension is employed at the boundaries; to avoid cross-layer energy imbalance, energy can be applied to each layer. Apply a small gain calibration and then normalize the reconstructed data to [0,255] or [0,1].

[0324] The above weights are simultaneously normalized at the pixel, layer, and source dimensions to ensure energy conservation and unbiased fusion; through and The hierarchical adaptive setting can enhance structural consistency at higher levels and strengthen detail fidelity at lower levels; task enhancement items This allows key target regions to receive higher weights at each layer, thereby improving the robustness of downstream detection / recognition; the linear fusion and standard reconstruction are easily implemented in parallel and can be efficiently deployed on CPU / GPU / edge NPU.

[0325] V. Specific Implementation Examples of this Application:

[0326] Example A (Classic + Lightweight)

[0327] In edge device scenarios with limited computing power and storage, task importance weighting Generated jointly by Canny edge detection and local entropy, where Task item weight coefficients are taken (No external task model involved) Normalization; Geometric Transformation Similarity / affine model employed; deformation cost function The coefficient of the smoothing term is taken as follows (Reduce computation) Merge pyramid levels Bicubic interpolation is used for upsampling and downsampling. This implementation achieves preservation of critical edges and textures with low computational overhead, and is suitable for real-time or near-real-time fusion in embedded SoCs / NPUs.

[0328] Expected results: Under typical 720p input and edge NPU / CPU 2–3W conditions, the P95 inference latency is maintained at approximately 30–50ms / frame, and the target edge PSNR / SSIM is improved compared to single source.

[0329] Example B (Enhanced Robustness)

[0330] In scenes with significant differences in perspective and marked local distortion Task confidence is provided by a lightweight detection / segmentation model. It is formed together with gradient / entropy; geometric alignment is achieved through affine + local TPS refinement; deformation cost. The three coefficients of the middle term are taken To simultaneously constrain anisotropy, area conservation, and smoothness; pyramid layer number , temperature A larger value is chosen to improve weight selectivity. This implementation reduces misalignment and artifacts in regions with high parallax / weak texture, improving fusion stability under complex conditions.

[0331] Expected results: Compared with Example A, the proportion of misaligned pixels in the intersection area is reduced, the edge ghosting is significantly reduced, and the downstream detection mAP is improved by 1-3 percentage points.

[0332] Example C (Multiple Figures) )

[0333] When input source number At that time, in the candidate benchmark set The combined target of formula (11) Search for the optimal benchmark ; the rest Each image is registered to Parallel computation of resampling attenuation With deformation cost At each scale and at each pixel and use Normalized weights This approach employs multi-source competition and complementarity. It enhances information coverage and robustness under multi-resolution / multi-field-of-view conditions.

[0334] Expected result: Union region ( The target integrity retention rate has improved, and the intersection area ( Information loss decline.

[0335] VI. Complexity and Key Implementation Points

[0336] Number of pyramid levels Take 3–5; number of key points Approximately 1k–3k; RANSAC iterations < 1e4.

[0337] and Pixel-wise summation can be block-based / vectorized; It can be fused into the same kernel on the GPU.

[0338] Edge / cloud devices can use fixed-point / half-precision computing. , The exponential / squared operation can be approximated by a LUT.

[0339] Formulas (1)–(16) given in the specific embodiments of this invention belong to the core decision-making and measurement model of this invention, in particular: The task-driven approach is based on the principle of "upper bound of intersection information loss - difference in retainable quantity". Anisotropy / area / smoothness triple deformation penalty based on Jacobi singular values. , and The construction achieves an integrated closed loop of resampling-fidelity prediction, partition scaling, and union / intersection cropping. These models, when combined with conventional Laplacian multi-scale fusion / affine registration, form a task-aware and deformation-controllable fusion scheme specifically for visible light images of different sizes.

[0340] As described above, the method of the present invention can be implemented well.

[0341] Compared with existing technologies, the present invention has the following significant advantages in the process of processing visible light images of different sizes for fusion:

[0342] 1) Improved information fidelity

[0343] This invention significantly reduces geometric distortion and loss of key information during image fusion by introducing an alignment mechanism with minimum deformation constraints and a task-driven decision function. Compared with traditional global interpolation and single scaling methods, this invention can better maintain the true proportions and detailed features of the target object, ensuring the geometric consistency of the fused image, thereby improving the accuracy and stability of downstream detection, recognition, and other tasks.

[0344] 2) Enhanced task effectiveness

[0345] This invention introduces task relevance evaluation metrics into the union and intersection regions, prioritizing the retention of information most valuable to the task and appropriately cropping and compressing redundant areas. This mechanism enables the fused image to not only possess high-quality visual performance but also greater semantic adaptability to specific tasks. Compared to existing "average treatment" information fusion methods, this invention better meets the needs of applications such as intelligent recognition and intelligent monitoring.

[0346] 3) Optimization of fusion efficiency and computational overhead

[0347] By leveraging a region-adaptive resolution scaling mechanism and a dynamic pruning strategy, this invention significantly reduces the invalid processing of low-value regions, thereby lowering overall computational overhead and storage requirements. Under the same computing power conditions, this invention can achieve higher-precision fusion results; even in edge device environments with limited computing power, this invention can operate stably, demonstrating better resource utilization and energy efficiency.

[0348] 4) Enhanced robustness and adaptability

[0349] By employing multi-level feature fusion and attention mechanisms, this invention achieves high-precision image alignment and fusion even under conditions of viewpoint differences, local distortion, or complex ambient lighting. In practical applications, this method demonstrates stronger environmental adaptability and robustness, overcoming the shortcomings of existing technologies that are prone to misalignment, artifacts, and information loss under complex conditions.

[0350] In summary, compared with existing technologies, this invention has achieved systematic improvements in image alignment accuracy, information fidelity, fusion efficiency, and environmental robustness, and can more comprehensively and efficiently meet the needs of visible light heterogeneous image fusion in intelligent sensing and task-driven applications.

[0351] Example 2

[0352] like Figure 6 As shown, this application provides a heterogeneous visible light image fusion system, employing the heterogeneous visible light image fusion method described above. The system includes:

[0353] Task Region Extraction and Importance Mapping Module 201: Acquire multiple visible light images, determine candidate regions based on task relevance, generate a task weight map, and use a union / intersection builder to obtain the union and intersection of task regions; perform bidirectional candidate geometric transformation and multi-level pyramid refinement and sub-pixel optimization on any two images to generate a candidate baseline map.

[0354] Dual Decision Function Evaluation and Benchmark Map Selection Module 202: Calculates the decision function that minimizes task information loss and the decision function that minimizes stretching / non-distortion completion for candidate benchmark maps, and selects the benchmark map;

[0355] Resolution scaling and dynamic cropping module 203: Based on the pixel density and task weight map of the reference map, it determines the global scale map and local scale map on the reference coordinate system, uses high-precision interpolation in the task area and lightweight interpolation in the non-task area, and dynamically crops the global scale map and local scale map to generate the optimal cropping window.

[0356] Multi-scale weighted fusion and reconstruction module 204: Constructs a Laplacian pyramid, calculates the content weights for each layer of each dynamically cropped image, introduces task weights, performs layer-by-layer fusion and reconstruction to obtain a fused image, and outputs the fusion results of visible light images of different sizes.

[0357] Example 3

[0358] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the heterogeneous visible light image fusion method.

[0359] Example 4

[0360] This application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the heterogeneous visible light image fusion method as described above.

[0361] In addition, combined Figure 1 The heterogeneous visible light image fusion method described in this application embodiment can be implemented by an electronic device, such as a computer device. Figure 7 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of this application.

[0362] In some embodiments, the computer device may further include a communication interface 83 and a bus 80. For example, Figure 7 As shown, the processor 81, memory 82, and communication interface 83 are connected through bus 80 and complete communication with each other.

[0363] Specifically, the processor 81 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0364] The memory 82 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 81.

[0365] The processor 81 reads and executes computer program instructions stored in the memory 82 to implement any of the heterogeneous visible light image fusion methods in the above embodiments.

[0366] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0367] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for fusing visible light images of different sizes, characterized in that, The method includes: Task region extraction and importance mapping steps: acquire multiple visible light images, determine candidate regions based on task relevance, generate a task weight map, and use a union / intersection builder to obtain the union and intersection of task regions; for any two of the images, perform bidirectional candidate geometric transformation and multi-level pyramid refinement and sub-pixel optimization to generate a candidate baseline map; Dual decision function evaluation and benchmark selection steps: Calculate the decision function that minimizes task information loss and the decision function that minimizes stretching / non-distortion completion for the candidate benchmark graphs, and select the benchmark graph; Resolution scaling and dynamic cropping steps: Based on the pixel density and task weight map of the reference map, determine the global scale map and local scale map on the reference coordinate system, use high-precision interpolation in the task area and lightweight interpolation in the non-task area, and dynamically crop the global scale map and local scale map to generate the optimal cropping window. Multi-scale weighted fusion and reconstruction steps: Construct a Laplacian pyramid, calculate the content weight of each layer of each dynamically cropped image and introduce the task weight, perform layer-by-layer fusion and reconstruction to obtain a fused image, and output the fusion result of the heterogeneous visible light image.

2. The heterogeneous visible light image fusion method according to claim 1, characterized in that, The task region extraction and importance mapping steps include: Input acquisition steps: Acquire two or more visible light images; Task region extraction and importance mapping steps: Obtain task region masks on each image through a task model or algorithm, calculate importance weight maps for each image, and use the union / intersection builder to obtain the union and intersection of task regions in a unified reference coordinate system; Candidate geometric alignment step: For any two images, generate bidirectional candidate geometric transformations; Multi-level pyramid refinement and sub-pixel optimization steps: The calculation results of bidirectional candidate geometric transformations are coarse-to-fine optimized from top to bottom in each layer of the Gaussian / Laplacian pyramid to minimize alignment error, and sub-pixel interpolation and small-range parameter search are used for fine-tuning in the last layer.

3. The heterogeneous visible light image fusion method according to claim 1, characterized in that, The steps for evaluating the dual decision function and selecting the benchmark graph include: Union / Intersection builder calculation steps: On a unified coordinate system of all images, perform union and AND logical operations on the input task mask to form the union and intersection of task regions; the union of task regions is the coverage area of ​​all image task regions, used for dynamic cropping to ensure that all task targets are covered; the intersection of task regions is the common area of ​​all image task regions, used to evaluate reliability / alignment accuracy and assist the decision function in selecting the reference image; The calculation steps for the decision function to minimize task information loss are as follows: When any image is selected as the reference image, the evaluation focuses on maximizing the task information that can be retained before fusion within the intersection of task regions and minimizing information loss. The calculation steps for the minimum stretch / non-distortion completion decision function are as follows: When any image is selected as the reference image, the deformation cost imposed on the non-reference image for alignment is evaluated, and isotropic, area-conserving, and low-deformation completion is encouraged. Reference image selection steps: Determine the reference image by combining targets, and transform and register the non-reference images to the reference coordinate system.

4. The heterogeneous visible light image fusion method according to claim 1, characterized in that, The resolution scaling and dynamic cropping steps include: Resolution scaling steps: Based on the pixel density of the baseline map and the task weight map, determine the global scale and local scale maps on the baseline coordinate system. High-precision interpolation is used for the task area, and lightweight interpolation is used for the non-task area. Dynamic cropping step: Based on the union of the task regions, the intersection of the task regions, and the error boundary, an optimal cropping window is generated to ensure the integrity of the target and reduce redundant background.

5. The heterogeneous visible light image fusion method according to claim 2, characterized in that, The multi-level pyramid refinement steps include: The steps for constructing a multi-layer pyramid are as follows: Each image involved in the registration is sampled separately to form a pyramid from coarse to fine. The Gaussian / Laplacian pyramid is used to obtain the image pairs and their pixel domains for each layer, which serve as input for subsequent layer-by-layer optimization. Coarse initial estimation steps: For the coarse layer of the multi-layer pyramid, feature point extraction and feature matching are performed. Robust model estimation is used to remove outliers, resulting in an initial geometric transformation from the image to be registered to the reference candidate image. The parameter vector of this transformation is used as the starting value for subsequent optimization. Optimization steps from coarse to fine layers: In the first... layer Using the converged transformation parameters of the previous layer as initial values ​​for this layer, the image to be registered is mapped to the reference coordinate system through this transformation; the weighted alignment error objective function is calculated in the intersection domain; the candidate geometric alignment step is repeatedly executed to generate and continuously update until the preset convergence threshold or the maximum number of iterations is met; The finest layer subpixel optimization steps are as follows: receive the converged transformation results, use high-precision interpolation in the highest resolution layer, and search for each alignment parameter until the fine alignment reaches the preset threshold. Local refinement steps: If non-rigid differences exist, constraints are applied to avoid excessive deformation; Output the final transformation step: obtain the globally optimal geometric transformation matrix.

6. The heterogeneous visible light image fusion method according to claim 3, characterized in that, The steps for calculating the decision function that minimizes task information loss include: Intersection region selection steps: After unifying the symbols of the two images to be registered, the intersection task region is obtained based on the task mask, and the intersection is mapped to the reference coordinate system; Detail intensity definition step: For the potential amount of detail or structure that can be preserved in the intersection task region image, define the detail / structure intensity for each pixel in the reference coordinate system; Resampling attenuation modeling steps: Use a resampling attenuation factor to construct a differentiable attenuation function for the resampling scale, retaining the predictable attenuation; Task importance weight calculation steps: Determine task weights based on task model output and image statistics; Steps for defining the upper bound of reservable detail and the actual amount that can be retained: Define the upper bound of reservable detail and the actual amount that can be retained for each pixel; Decision function construction steps: Based on the principle of minimizing the loss of intersection information, construct a decision function that minimizes the loss of task information. for: in, To prevent zero constant, Index for candidate benchmark maps, The reference coordinates are pixels. For the task intersection on the baseline map The set of pixels on; In order to be in For each pixel Define the maximum value of the detail / structural strength. For each pixel The upper limit of the details that can be preserved. For each pixel The task weight value is calculated pixel-by-pixel for the difference between the ideal reachable detail and the detail that can be retained after resampling in the intersection region, and then averaged normally by task weight. The smaller the value, the more likely the image is to be selected. The smaller the loss of task-related information when using it as a baseline.

7. The heterogeneous visible light image fusion method according to claim 3, characterized in that, The calculation steps for the minimum stretching / non-distortion completion decision function include: Local deformation metric calculation steps: Calculate the local deformation metric based on the Jacobian and singular values ​​of the image transformation; Equivalent scale and rotation extraction steps: Local equivalent scale and rotation are obtained from polar decomposition; The steps for constructing a decision function based on the principle of minimizing deformation cost are as follows: In the task intersection region... Above, construct the minimum stretch / non-distortion completion decision function. : in, For the cost of deformation The weighted normal average on the chart, the smaller the value, the better when the chart is selected. When using a reference image and registering other images to its coordinate system, the closer the completion process is to an isotropic, area-conserving, and smooth non-distortion transformation, the better. Index for candidate benchmark maps, The reference coordinates are pixels; For the task intersection on the baseline map The set of pixels on; Deformation-sensitive weights are used to emphasize regions sensitive to geometric distortion. As the weight of task importance, For the reference image in The first-order gradient vector at that point, Its amplitude; The normalization operator is used to map the input scalar to... Interval; can be taken when there is no task model. , To prevent zero constant; In order to make the first Transform the image to the first When the image is in pixels The cost of localized deformation at the site; The singular values ​​of the transformed Jacobian are used to measure whether it is isotropic, whether the area is conserved, and whether the deformation is smooth.

8. A heterogeneous visible light image fusion system, employing the heterogeneous visible light image fusion method as described in any one of claims 1-7, characterized in that, The system includes: Task Region Extraction and Importance Mapping Module: Acquire multiple visible light images, determine candidate regions based on task relevance, generate a task weight map, and use a union / intersection builder to obtain the union and intersection of task regions; perform bidirectional candidate geometric transformation and multi-level pyramid refinement and sub-pixel optimization on any two of the images to generate a candidate baseline map; Dual Decision Function Evaluation and Benchmark Map Selection Module: Calculates the decision function that minimizes task information loss and the decision function that minimizes stretching / non-distortion completion for the candidate benchmark maps, and selects the benchmark map; Resolution scaling and dynamic cropping module: Based on the pixel density and task weight map of the reference map, determine the global scale map and local scale map on the reference coordinate system, use high-precision interpolation in the task area and lightweight interpolation in the non-task area, and dynamically crop the global scale map and local scale map to generate the optimal cropping window. Multi-scale weighted fusion and reconstruction module: Constructs a Laplacian pyramid, calculates the content weights of each layer of each dynamically cropped image, introduces task weights, performs layer-by-layer fusion and reconstruction to obtain a fused image, and outputs the fusion result of visible light images of different sizes.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the heterogeneous visible light image fusion method according to any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the heterogeneous visible light image fusion method as described in any one of claims 1-7.

Images