Multi-modal flight visual image fusion method and system under complex weather

Abstract

The application relates to the technical field of flight vision image processing, and discloses a multi-modal flight vision image fusion method and system under complex weather. The method comprises the following steps: synchronously collecting visible light, infrared, micro-light images and weather parameters to generate a space modal reliability field; implementing gating and compensation according to the reliability field to reconstruct a fusion image; identifying a deviation source through no-reference quality and flight task perception results, selectively updating collection, registration or gating configuration until the quality is stable. The system comprises a multi-modal image collection component, a weather parameter collection component and a synchronous registration, reliability evaluation, hierarchical fusion and feedback control processor. The application can inhibit the degradation caused by fog, rain and snow, and keep runway edge, horizon and obstacle information continuous.

Description

Technical Field

[0001] This invention relates to the field of flight visual image processing technology, specifically to a method and system for multimodal flight visual image fusion under complex weather conditions. Background Technology

[0002] Flight visual imagery is a crucial medium for pilots and trainees to acquire information about their external environment. Visible light sensors provide color and texture, infrared sensors provide differences in target thermal radiation, and low-light sensors provide contours and weak textures under low-light conditions. Different sensors have varying spectral responses, spatial resolutions, noise types, and dynamic ranges; therefore, fusion processing requires determining the information each mode can provide in different regions under a unified spatiotemporal reference. Fog, haze, rain, snow, and low light not only alter the average quality of the entire image but also cause uneven degradation in near-ground layers, sky regions, runway light regions, hot target regions, and distant ground features. When fusion is controlled by a single quality score or a fixed fusion weight for the entire image, locally visible information is easily diluted by low-quality modes, and local error edges may also be enhanced.

[0003] Existing infrared and visible light fusion schemes can assign weights based on image entropy, no-reference quality score, low-frequency scene information, or high-frequency detail intensity. Aviation enhanced vision schemes also utilize infrared, millimeter-wave, or other sensor information to supplement runway and obstacle information in low-visibility environments. Existing schemes provide a foundation for multi-sensor visual displays, but weather conditions are typically treated as scene conditions, display conditions, or global parameters, and image quality indicators are often based on the entire image. Degradation caused by complex weather conditions exhibits spatial and temporal variations, making it difficult for a single global weight to simultaneously handle sky scattering, runway reflections, raindrop occlusion, high snow reflectivity, weak contrast of thermal targets, and low-light noise.

[0004] Multimodal images also suffer from the problem of coupled registration and degradation errors. Rain and snow streaks, fog layers, and motion blur can alter the correspondence between cross-modal edges. If the fusion algorithm misjudges the registration residual as detail differences, the fused image may exhibit double edges, ghosting, or texture misalignment. Conversely, if the fusion algorithm generally reduces detail weights to suppress ghosting, runway edges, horizon lines, and obstacle outlines will be weakened. Therefore, it is necessary to transform meteorological conditions, single-modal degradation, cross-modal consistency, and registration reliability into a reliable basis for spatial distribution.

[0005] Existing feedback control typically recalculates the fusion parameters as a whole when the fusion quality falls below a threshold. Meteorological measurement errors, image registration errors, single-modal degradation estimation errors, and fusion gating errors have different correction targets. A complete update can easily lead to simultaneous changes in multiple parameter sets, causing fusion weights to oscillate between adjacent frames and potentially increasing the sampling burden of irrelevant meteorological parameters. The quality of flight visuals cannot be judged solely by general image metrics. Runway edges, runway centerlines, horizons, and obstacles are mission-related information during approach, landing, and low-altitude flight; even with a high average sharpness in the fused image, a decrease in the continuity of key targets can still affect visual perception.

[0006] Therefore, there is a need to provide a multimodal flight visual image fusion method and system under complex weather conditions, establish a spatial correlation between weather conditions and local image degradation, use spatial reliability fields to constrain brightness, chromaticity and detail fusion, attribute fusion deviations to no-reference quality results and flight mission perception results, update only the acquisition, registration, degradation assessment or gating parameters corresponding to the source of deviation, and end the closed-loop update under continuous quality stability conditions. Summary of the Invention

[0007] The purpose of this invention is to solve the problems of different degradation mechanisms, inconsistent distribution of local available information, and easy oscillation of the overall update of closed-loop parameters under complex weather conditions for visible light images, infrared images, and low-light images. It provides a method and system for multimodal flight visual image fusion under complex weather conditions.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A multimodal flight visual image fusion method under complex weather conditions includes: Visible light images, infrared images, and low-light images, along with meteorological parameters corresponding to the acquisition time, are collected from the same flight field of view. The visible light images, infrared images, and low-light images are then time-aligned and spatially registered to obtain an aligned image group. Structural features, radiation features, noise features, and cross-modal consistency features of each mode are extracted from the aligned image group, and combined with the meteorological parameters to generate a modal reliability field characterizing the degree of degradation and reliability of different spatial regions. Based on the modal reliability field, reliability gating and cross-modal compensation are applied to each modal feature to form luminance fusion features, chroma transfer features and detail preservation features, respectively, and to reconstruct the flight visual fusion image; Extract no-reference quality results and flight mission perception results from the flight visual fusion image, and determine the source of fusion deviation based on the no-reference quality results, the flight mission perception results, and the modal reliability field; According to the sources of the fusion deviation, the meteorological parameter acquisition configuration used for control acquisition, the spatial registration configuration used for control registration, the degradation assessment configuration used for control degradation assessment, or the reliability gating parameters are selectively updated, and the updating stops when the continuous quality results meet the stability conditions, and the flight visual fusion image is output.

[0009] As a preferred embodiment of the present invention, the time alignment and spatial registration include: establishing a synchronized image group based on hardware synchronization signals, image timestamps, and meteorological parameter timestamps; projecting the infrared image and the low-light image onto the visible light image coordinate system using sensor extrinsic parameters; correcting the projection relationship based on cross-modal edges, stable ground feature contours, and local mutual information; outputting the registration image, registration residual map, and registration reliability map in the aligned image group, and inputting the registration residual map and the registration reliability map into the generation process of the modal reliability field.

[0010] As a preferred embodiment of the present invention, the meteorological parameters include at least three of the following: visibility, relative humidity, ambient temperature, wind speed, wind direction, precipitation type, and precipitation intensity; the continuous meteorological parameters are subjected to time filtering and trend estimation to form a meteorological state vector and a meteorological uncertainty. The meteorological state vector is used to determine the meteorological degradation prior, and the meteorological uncertainty is used to limit the update amplitude of the modal reliability field.

[0011] As a preferred embodiment of the present invention, the generation process of the modal reliability field includes: determining the atmospheric scattering degradation features of the visible light image, the thermal contrast degradation features of the infrared image, and the low-light noise degradation features of the low-light image; inputting each modal degradation feature, meteorological degradation prior, the registration residual map, and the cross-modal consistency features into the degradation evaluation network; outputting the modal reliability values ​​corresponding to each spatial region by the degradation evaluation network, and performing normalization on the modal reliability values ​​of different modes, so that the modal reliability values ​​at the same spatial location are used to constrain the subsequent fusion contribution.

[0012] As a preferred embodiment of the present invention, the generation process of the modal reliability field includes: determining the atmospheric scattering degradation characteristics of the visible light image, the thermal contrast degradation characteristics of the infrared image, and the low-light noise degradation characteristics of the low-light image; inputting each modal degradation characteristic, the meteorological degradation prior, the meteorological uncertainty, and the cross-modal consistency characteristics into a degradation evaluation network; outputting the modal reliability values ​​corresponding to each spatial region by the degradation evaluation network, and performing normalization on the modal reliability values ​​of different modes, so that the modal reliability values ​​at the same spatial location are used to constrain the subsequent fusion contribution.

[0013] As a preferred technical solution of the present invention, the reliability gating and cross-modal compensation include: performing regional adaptive weighting on multi-scale brightness features according to the modal reliability field; determining the chroma transfer region according to visible light chroma confidence and scene semantic boundary; determining the detail preservation direction according to the edge response of each modality and the modal reliability field; using the brightness fusion feature as the brightness constraint for chroma transfer, and inputting the chroma transfer feature and the detail preservation feature together into the neural network for reconstruction, so that the output of the previous processing constitutes the input or constraint of the next processing.

[0014] As a preferred embodiment of the present invention, the referenceless quality results include local contrast, sharpness, noise residue, color stability, and temporal flicker; the flight mission perception results include at least two of the following: runway edge continuity, runway centerline visibility, horizon stability, obstacle detection confidence, and ground feature segmentation consistency; based on the abnormal distribution of the referenceless quality results and the flight mission perception results in spatial and temporal locations, meteorological parameter measurement bias, spatial registration bias, single-modal degradation estimation bias, and fusion gating bias are distinguished.

[0015] As a preferred embodiment of the present invention, the selective update includes: when the source of the fusion deviation is a meteorological parameter measurement deviation, increasing the sampling frequency of the relevant meteorological parameters or switching the meteorological data source; when the source of the fusion deviation is a spatial registration deviation, updating the local registration control points and image preprocessing intensity; when the source of the fusion deviation is a single-mode degradation estimation deviation, updating the degradation feature weights of the corresponding mode; when the source of the fusion deviation is a fusion gating deviation, updating the reliability gating parameters of the corresponding spatial region, while the parameter groups not identified as sources of deviation remain unchanged.

[0016] As a preferred embodiment of the present invention, the stability conditions include the changes in the no-reference quality result, the changes in the flight mission perception result, and the changes in the reliability gating parameter all being lower than the corresponding preset stability thresholds within multiple consecutive fusion cycles; when the reliability values ​​of all modalities are lower than the preset safety threshold, the closed-loop update is stopped and the single-modal image with the highest reliability or the single-modal image with the highest reliability after conservative enhancement is output, while a visual reliability prompt is generated.

[0017] As a preferred embodiment of the present invention, the degradation evaluation network includes a convolutional encoder for extracting local degradation features, an attention encoder for establishing the association between meteorological conditions and spatial degradation regions, and a decoder for outputting the modal reliability field. During the training phase, synchronous multimodal samples with meteorological ground truth labels, visibility ground truth labels, and flight mission ground truth labels are used, and reconstruction quality, mission preservation, modal reliability consistency, and temporal smoothness constitute the joint training objectives. During the application phase, the degradation evaluation network runs on an airborne edge computing node or a flight simulation graphics processing node.

[0018] As a preferred technical solution of the present invention, the flight mission is an approach or landing mission, and the flight mission perception results include at least the runway threshold identification results, runway edge continuity and runway centerline visibility. The reliability gating process sets mission maintenance constraints on the runway area so that the thermal profile information provided by the infrared image and the structural information provided by the visible light image or the low-light image remain spatially consistent within the runway area.

[0019] As a preferred technical solution of the present invention, the flight mission is a flight training simulation mission. The perception results of the flight mission are compared with the key ground object categories, obstacle locations and visibility levels set by the training subjects. The deviation attribution results are used to update the reliability gating parameters in the training simulation graphics processing node, and the pre-fusion image, the flight visual fusion image, the meteorological state vector and the feedback update record are saved as training review data.

[0020] As a preferred technical solution of the present invention, under the meteorological conditions of dense fog, heavy precipitation or snowfall, the consistency of airborne meteorological sensor data, ground meteorological station data and meteorological data link data is checked to form a multi-source meteorological state vector and meteorological change trend; the reliability gating parameters to be used in subsequent fusion cycles are pre-adjusted using the meteorological change trend, and after the image quality result of the new moment arrives, the reliability gating parameters to be used in subsequent fusion cycles are confirmed or rolled back according to the source of the fusion deviation.

[0021] In addition, the present invention also discloses a multimodal flight visual image fusion system under complex weather conditions, used to execute the multimodal flight visual image fusion method under complex weather conditions as described above, characterized in that: it includes a multimodal image acquisition component, a meteorological parameter acquisition component, a synchronous registration processor, a reliability assessment processor, a hierarchical fusion processor, a feedback control processor, a memory, and a data bus; The multimodal image acquisition component includes a visible light sensor, an infrared sensor, and a low-light sensor, and outputs images of the same flight field of view to the synchronization registration processor via a data bus; The meteorological parameter acquisition component outputs timestamped meteorological parameters to the reliability assessment processor and the feedback control processor; The synchronous registration processor outputs the aligned image group, the registration residual map, and the registration confidence map to the reliability evaluation processor; The reliability assessment processor generates a modal reliability field based on the image features extracted from the aligned image group and the received meteorological parameters, and outputs it to the hierarchical fusion processor. The hierarchical fusion processor forms a flight visual fusion image based on the modal reliability field, and outputs the flight visual fusion image to the display interface and the feedback control processor; The feedback control processor sends selective update instructions to the meteorological parameter acquisition component, the synchronization registration processor, the reliability assessment processor, or the hierarchical fusion processor based on the no-reference quality results extracted from the flight visual fusion image, the flight mission perception results, and the determined fusion deviation sources. The memory stores the processing program, model parameters, and feedback update records.

[0022] As a preferred embodiment of the present invention, the step of determining the atmospheric scattering degradation characteristics of the visible light image, the thermal contrast degradation characteristics of the infrared image, and the low-light noise degradation characteristics of the low-light image includes: for the active approach lighting system activated during the approach and landing phase, identifying local brightness peaks in the visible light image and the low-light image where the brightness value is greater than a preset sensor saturation threshold, and calculating the radial gradient decay rate centered on the local brightness peak; when the radial gradient decay rate is lower than a preset standard gradient threshold and the brightness value of the local brightness peak is greater than the preset sensor saturation threshold, marking the connected region around the local brightness peak as an active approach lighting system. A halo region scattering light from a light source is identified. For this region, a halo spatial expansion model is constructed, dynamically mapped from the visibility and relative humidity parameters. This model generates a spatially continuous dynamic suppression coefficient. This dynamic suppression coefficient is then superimposed on the atmospheric scattering degradation features of the visible light image and the low-light noise degradation features of the low-light image. This forces an increase in the response value of the atmospheric scattering degradation features input to the degradation evaluation network within the halo region scattering light from a light source, thereby reducing the modal reliability values ​​of the visible light image and the low-light image output by the degradation evaluation network within the halo region scattering light from a light source.

[0023] As a preferred embodiment of the present invention, the step of determining the detail-preserving direction based on the edge response of each modality and the modal reliability field includes: constructing a multimodal joint structure tensor with the modal reliability field as the weight; performing eigenvalue decomposition on the multimodal joint structure tensor to obtain the principal eigenvalue, secondary eigenvalue, and corresponding eigenvector representing the saliency of the structure; using the ratio of the principal eigenvalue to the secondary eigenvalue to distinguish flat regions, edge regions, and corner regions in the image space; in the edge regions, extracting the direction of the principal eigenvector determined by the multimodal joint structure tensor as the cross-modal common edge direction; performing one-dimensional smoothing filtering along the cross-modal common edge direction to suppress false edges caused by cross-modal registration residuals, and performing one-dimensional high-pass filtering along the normal direction perpendicular to the cross-modal common edge direction to extract edge detail features, and using the joint output of the bidirectional filtering as the detail-preserving feature.

[0024] Compared with the prior art, the present invention has the following beneficial effects: This invention's modal reliability field unifies meteorological conditions, single-modal degradation, cross-modal consistency, and registration reliability to spatial location, allowing the fusion contribution to vary with local degradation. Compared to using a single weight for the entire image, different modal contributions can be used for sky scattering areas, runway light areas, snow-covered high-reflectivity areas, and hot target areas, reducing the dilution of locally usable information by low-quality modes. Reliability gating forms a pre- and post-dependent relationship with brightness fusion, chromaticity transfer, and detail preservation. Brightness results constrain chromaticity transfer, with chromaticity and detail jointly entering the reconstruction, and registration reliability participating in detail preservation, which can reduce double edges, ghosting, color drift, and over-enhancing of details. The no-reference quality branch and the flight mission perception branch jointly evaluate the fused image. The evaluation objects include not only contrast, sharpness, and noise, but also runway edges, centerline, horizon, and obstacles, ensuring that the feedback targets fall within the technical applications of the flight vision. Selective feedback after bias attribution only updates the parameter set related to the source of the anomaly, reducing temporal oscillations caused by synchronous changes in irrelevant parameters and avoiding erroneous increases in meteorological sampling frequency when image registration problems occur. Continuous quality stability conditions and low-reliability safety fallback provide a termination boundary for the closed loop. This invention can be deployed on airborne edge computing nodes or flight simulation graphics processing nodes. The core claims do not depend on specific sensor models or fixed meteorological thresholds. The system implementation layer can meet real-time processing requirements through synchronous data acquisition, multi-source meteorological access, and hardware acceleration. The scenario layer can cover approach and landing simulations as well as training simulations. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating the overall process of the method of the present invention.

[0027] Figure 2 This is a sub-flowchart of the present invention for evaluating and optimizing the degradation of the halo region scattered by an active light source.

[0028] Figure 3 This is a sub-flowchart of the present invention for detail preservation based on multimodal joint structure tensors. Detailed Implementation

[0029] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the embodiments of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0030] The following is in conjunction with the appendix Figures 1-3 The embodiments of the present invention will be described in detail below.

[0031] In this invention, a multimodal image acquisition component arranges visible light sensors, infrared sensors, and low-light sensors facing the same flight field of view. These three types of sensors can be installed on the aircraft's nose, fuselage photoelectric windows, or the visual acquisition position of a flight simulator. The visible light sensor outputs a digital image containing brightness and chromaticity, the infrared sensor outputs a radiation intensity image or a temperature response image, and the low-light sensor outputs a low-illuminance grayscale image or a low-illuminance image with narrow-band color. When the sensors have different resolutions, they are first projected onto a unified coordinate system according to calibration parameters, and then resampling is performed according to the processing resolution.

[0032] The meteorological parameter acquisition component can include a visibility meter, relative humidity sensor, temperature sensor, wind speed and direction sensor, and precipitation sensor, and can also receive meteorological information from ground weather stations or meteorological data links. Data from different sources are accompanied by the acquisition time and data source identifier. The feedback control processor can adjust the sampling period of specified parameters based on the deviation attribution results. The adjusted meteorological parameters are still time-aligned before being included in the meteorological state vector, and do not directly replace the meteorological data at the corresponding image time.

[0033] The synchronous registration processor first selects temporally adjacent image frames based on hardware trigger signals, and then corrects the buffer delays of different sensors based on timestamps. Spatial registration employs a combination of offline extrinsic parameters and online correction. Offline extrinsic parameters are obtained from calibration boards, distant stable ground features, or the known field of view of a flight simulator. Online correction extracts transspectral stable structures, including long straight edges, building outlines, runway edges, and mountain outlines, from different modes, and optimizes local control points using local mutual information and edge orientation consistency. Online correction outputs a registration residual map, which records local correspondence errors; the registration confidence map is jointly determined by the number of control points, control point distribution, mutual information gain, and edge orientation consistency.

[0034] Furthermore, image feature extraction uses the registered image as input. Degradation features of visible light images include dark channel response, local contrast, color shift, gradient decay, and the proportion of bright scattering regions. Degradation features of infrared images include local thermal contrast, non-uniform response, saturation region proportion, noise variance, and edge stability. Degradation features of low-light images include photon noise estimation, readout noise estimation, local signal-to-noise ratio, brightness compression, and stripe noise. Cross-modal consistency features include edge location consistency, structure tensor orientation consistency, local mutual information, and stable target response consistency.

[0035] The meteorological state vector consists of time-filtered meteorological parameters, the rate of change of meteorological parameters, and the data source consistency result. Decreased visibility enhances the prior knowledge of visible light scattering degradation; increased precipitation intensity enhances the prior knowledge of stripe noise and local occlusion; increased relative humidity, along with visibility changes, constrains fog degradation; and ambient temperature and infrared detector temperature can be used to distinguish changes in scene thermal contrast and detector drift. Meteorological uncertainty is jointly determined by sensor accuracy, data time difference, data source differences, and short-term fluctuations. When the meteorological uncertainty is high, the single update amplitude of the modal reliability field is limited.

[0036] The degradation assessment network employs a combination of a convolutional encoder and an attention encoder. The convolutional encoder receives image features from each modality and extracts multi-scale local degradation representations. The attention encoder receives the meteorological state vector and projects the meteorological degradation prior onto the image feature scale. Registration residual maps and registration confidence maps are input to the decoder via additional channels. The decoder outputs visible light reliability maps, infrared reliability maps, and low-light reliability maps corresponding to the processed image resolution; these spatially distributed reliability maps constitute the modal reliability field. Reliability values ​​from different modalities are normalized at the same location to form the modal reliability field.

[0037] Furthermore, the functional relationship within the degradation assessment network used to generate normalized reliability values ​​can be expressed as:

[0038] In the formula, Indicates time In spatial location Mode The normalized reliability value; Indicates visible light, infrared, or low-light modes; Represents a set of modes; This represents the modal index in normalized summation; Representing modes At any moment and location The degree of degradation at the site; Representing modes At any moment and location The degree of degradation at the site; This indicates the transformation from the aforementioned meteorological uncertainty, and is related to the mode. At any moment The relevant modal-related prior uncertainty is obtained by allocating the meteorological uncertainty based on the spectral response characteristics of each modal sensor. This indicates the transformation from the aforementioned meteorological uncertainty, and is related to the mode. At any moment Related modal-related prior uncertainties; Representing modes At any moment and location The combination of the degree of cross-modal inconsistency and the registration residual at the location; Representing modes At any moment and location The combination of the degree of cross-modal inconsistency and the registration residual at the location; , and Representing modes The calibration coefficients for the degree of degradation, modal-related prior uncertainty, and cross-modal inconsistency; , and Representing modes The calibration coefficients for the degree of degradation, modal-related prior uncertainty, and cross-modal inconsistency; This indicates an exponential mapping.

[0039] Furthermore, the modal degradation degree, modal-related prior uncertainty, and cross-modal inconsistency degree are all converted to the same non-negative dimension. Calibration coefficients can be optimized using training samples or initialized based on sensor noise models and meteorological cabin calibration results. The exponential mapping in the formula converts increased degradation into decreased reliability, and normalization ensures that the sum of modal reliability values ​​at the same location is 1. The degradation assessment network output can be directly used as formula input or the formula can be used as a normalization constraint for the network output layer.

[0040] The hierarchical fusion processor feeds the registered image into a multi-scale feature encoder. The multi-scale feature encoder uses different channels for luminance, chrominance, and detail. The luminance channel preserves a wide range of light and dark relationships, the chrominance channel preserves visible light colors and suppresses low-confidence chrominance, and the detail channel preserves edge orientation and high-frequency texture. The modal reliability field performs average pooling or guided downsampling according to the multi-scale feature hierarchy to correspond to feature locations at different scales.

[0041] Furthermore, luminance fusion first selects local contributions based on the modal reliability field, and then improves the preservation of reliable structures in runway edges, horizons, or obstacle regions based on flight mission area constraints. Chromaticity transfer reads the luminance fusion result to avoid misalignment between chromaticity and luminance boundaries. When visible light chromaticity reliability is high, the chromaticity mainly comes from the visible light image; when visible light chromaticity reliability is low, the system reduces chromaticity saturation and uses infrared or low-light structures to limit chromaticity diffusion, not directly interpreting infrared grayscale as true color. Detail preservation reads the registration reliability map; regions with low registration reliability do not perform cross-modal high-frequency superposition, but instead retain single-source details from the mode with the highest reliability.

[0042] Furthermore, the compensation features are not fixed enhancements. When cross-modal edge positions are consistent and reliability is complementary, the compensation features retain unique edges from the low-weight mode; when cross-modal edge conflicts or registration reliability is low, the compensation features suppress high-frequency conflict information. Flight mission region constraints are generated by the soft region map output by the mission-aware model. The soft region map only enhances structures already confirmed by the modal reliability field and does not bypass modal reliability judgments.

[0043] Among them, the hierarchical fusion processor at a certain multi-scale feature level The process of generating fusion features can be represented as:

[0044] In the formula, Representing multi-scale feature levels The fusion characteristics; Represents multi-scale feature levels; Representing modes At multi-scale feature levels The coding features; This represents the compensation feature formed by cross-modal complementarity, flight mission area constraints, and registration reliability. Indicates time In spatial location Mode The normalized reliability value; Indicates visible light, infrared, or low-light modes; Represents a set of modes; and Represents spatial location coordinates; Indicates the time.

[0045] The reconstruction network receives luminance fusion features, chroma transfer features, and detail preservation features at various scales. The reconstruction network can employ a convolutional decoder, a residual decoder, or a lightweight transformer decoder. The output image undergoes dynamic range mapping and display gamut conversion to obtain a fused flight view image. The parameters of the dynamic range mapping are determined by the histogram of the luminance fusion result and the display interface capabilities, avoiding the direct introduction of sensor bit depth differences into the displayed image.

[0046] The no-reference quality assessment branch calculates local contrast, sharpness, noise residue, color stability, and temporal flicker for the flight-view fusion image. Local contrast is calculated by measuring the difference in brightness across multiple scales; sharpness is obtained by combining gradient energy and edge width; noise residue is estimated using residuals in flat regions and temporal differences; color stability is determined by changes in chromaticity distribution and color differences on both sides of semantic boundaries; and temporal flicker is determined by the brightness difference and weight difference between adjacent frames after motion compensation.

[0047] The flight mission perception branch can load different models based on the application scenario. Approach or landing scenarios load models of the runway threshold, runway edges, runway centerline, horizon, and obstacles; low-altitude flight scenarios load models of terrain boundaries, power lines, buildings, and moving obstacles; training simulation scenarios load terrain models specified in the training subject configuration. The mission perception model outputs target location, target confidence, and temporal continuity. The mission perception results do not directly change the fusion weights but are fed into the bias attributor along with the no-reference quality results.

[0048] Furthermore, the bias attributor establishes a correspondence between anomalous modes and parameter sets. If multiple modes exhibit degradation inconsistent with visibility or precipitation data simultaneously, while the registration residuals remain stable, the bias tends to favor meteorological parameter measurements. If anomalies are concentrated at contour edges and are directionally consistent but geographically misaligned across mode edges, the bias tends to favor spatial registration. If the degradation prediction for a single mode is inconsistent with actual noise or contrast changes, the bias tends to favor single-mode degradation estimation. If the mode reliability assessment is reasonable, but the fused image exhibits chromaticity overflow, excessive detail enhancement, or temporal weight jumps, the bias tends to favor fusion gating. The bias attributor can employ a combination of rule-based and classification networks; rules are used to limit unreasonable attributions, while classification networks handle composite anomalies.

[0049] The closed-loop evaluation metric is used to determine whether the update improves the fusion result, and is not used as a mathematical optimization objective detached from the flight view scenario. The feedback control processor quantifies the overall performance of the feedback cycle using the following closed-loop evaluation function:

[0050] In the formula, Indicates time Closed-loop evaluation quantity; This indicates the normalized quality result without a reference. This represents the normalized flight mission perception results; This indicates the amount of parameter change between adjacent fusion cycles; Indicates the percentage of areas with low reliability; , , and These represent the evaluation weights for no-reference quality, flight mission awareness, parameter stability, and low reliability regions, respectively. Indicates the time.

[0051] After the update command is executed, the feedback control processor compares the closed-loop evaluation values ​​before and after the update. If the closed-loop evaluation value decreases but the task perception result does not decrease, the update is retained; if the closed-loop evaluation value increases or the task perception result decreases, the update is rolled back. If the changes in the closed-loop evaluation value, the task perception result, and the parameter changes are all below a preset stability threshold for multiple consecutive fusion cycles, the feedback control processor ends the update and maintains the current configuration.

[0052] The preset stability thresholds are derived from meteorological cabin calibration, flight simulation playback, and equipment noise statistics. Different sensor combinations can use different thresholds, but all thresholds correspond to observable measurements. The upper limit of the meteorological parameter sampling frequency is determined by the sensor response time and data link bandwidth; the upper limit of the number of registration control points is determined by the processing latency; and the upper limit of the reliability gating update magnitude is determined by the timing flicker constraint. The system does not update all parameter groups simultaneously in each fusion cycle.

[0053] The training data for the degradation assessment network consists of time-synchronized multimodal images, corresponding meteorological parameters, registration labels, and flight mission labels. Meteorological labels include weather type, parameter range, and trend. Visibility labels can be obtained through manual annotation, standard target measurements, or ground truth values ​​from simulated environments. Mission labels include runway edges, centerline, horizon, and obstacles. During training, fog, rain streaks, snowflakes, low-light noise, and sensor drift are sampled hierarchically to allow the model to learn different degradation sources. Training, validation, and test sets are divided according to meteorological processes or flight sorties to avoid information leakage caused by adjacent frames crossing sets.

[0054] The joint training objectives include reconstruction quality, mission retention, modal reliability consistency, and temporal smoothness. Reconstruction quality is used to limit structural and brightness deviations; mission retention is used to maintain critical flight targets; modal reliability consistency is used to ensure that reliability values ​​correspond to occlusion, noise, and registration residuals; and temporal smoothness is used to limit reliability jumps between adjacent frames. After training, the backbone parameters are frozen, and airborne applications can update only a small number of gating parameters or calibration coefficients. Incremental updates must be verified with replay data before being written into the running model; all model parameters are not directly modified using feedback from a single flight.

[0055] Furthermore, the system can run on a processing platform composed of a central processing unit, a graphics processing unit, a neural network processor, or a programmable logic device. The synchronous registration processor and the hierarchical fusion processor can employ pipelined parallelism, while meteorological parameter processing and feedback attribution can run in lower-frequency threads. Image frames are transmitted through a circular buffer, each carrying an image timestamp, sensor identifier, calibration version, meteorological state version, and gating parameter version. Version information is used to review the closed-loop update process.

[0056] When the reliability values ​​of all modalities fall below a preset safety threshold, the system will not continue to increase the fusion enhancement intensity. The feedback control processor selects the modality with the highest average reliability and the most stable task perception results, outputs the original single-modal image or a conservatively enhanced image, and displays a visual reliability prompt on the display interface. The safe rollback ends when the reliability of at least one modality recovers continuously and the registration confidence meets the recovery threshold. Safe rollback avoids generating fused images with false structures when all modalities are severely contaminated.

[0057] To facilitate a better understanding of the present invention by those skilled in the art, the present invention will be further described below with reference to specific embodiments.

[0058] Example 1: This example illustrates the basic implementation of the core algorithm layer. In this example, the system acquires visible light, infrared, and low-light images from the same near-field perspective. Each acquisition frame is accompanied by a hardware trigger number and a timestamp. Meteorological parameters include visibility, relative humidity, ambient temperature, and precipitation intensity. The synchronous registration processor completes the initial projection based on the extrinsic parameters and then performs local corrections using the runway edge and distant building outlines. The output includes three registered images, a registration residual map, and a registration confidence map.

[0059] In this embodiment, visible light images are used to extract local contrast, color shift, dark channel response, and gradient attenuation; infrared images are used to extract thermal contrast, non-uniform noise, and edge stability; and low-light images are used to extract local signal-to-noise ratio, photon noise, and brightness compression. The meteorological state vector and registration information are input into the degradation evaluation network, which outputs three spatially distributed modal reliability fields. The visible light reliability in the far-end fog region is lower than that in the near-end runway region, the infrared reliability in the hot target region is higher than that in the sky region, and the low-light reliability varies with local illuminance and noise.

[0060] The layered fusion processor performs reliability gating on the brightness features of the three modes. When the runway edge region has both infrared thermal differences and visible light structure, the compensation features retain two types of consistent edges. When the far-end visible light edge is weakened by the haze layer, infrared or low-light structure is used for compensation, but the compensation magnitude is still limited by the registration confidence level. Chroma transfer retains saturated colors only when the visible light chroma confidence level meets the conditions, and the chroma intensity is reduced in the haze layer region. The detail preservation process uses single-modal details in low registration confidence areas to avoid forming double edges.

[0061] The flight visual fusion imagery is fed into no-reference quality assessment and flight mission perception. No-reference quality assessment outputs local sharpness, noise persistence, and flicker levels, while flight mission perception outputs runway edge continuity, centerline visibility, and obstacle confidence. If edge continuity decreases and anomalies are concentrated in cross-modal contour misalignment areas, the bias attributor outputs spatial registration bias. The feedback control processor only updates local control points and registration preprocessing, without changing the meteorological sampling frequency or reliability gating parameters for other areas.

[0062] For example, the visible light degradation level at a certain runway edge location is set to 0.68, the infrared degradation level to 0.22, and the low-light degradation level to 0.47; the corresponding modal correlation prior uncertainties are set to 0.10, 0.06, and 0.08, respectively; and the cross-modal inconsistencies are set to 0.18, 0.08, and 0.14, respectively. When each calibration coefficient is set to 1, the visible light, infrared, and low-light reliability values, according to the aforementioned reliability value function relationship, are 0.242, 0.441, and 0.317, respectively. These three reliability values ​​are used for brightness feature fusion at the same location.

[0063] Assuming the encoding features of the three modalities at a certain scale are 0.35, 0.76, and 0.51, respectively, and the cross-modal compensation feature is 0.04, the fusion feature obtained according to the aforementioned fusion feature function relationship is 0.622. The reconstruction network combines the fusion feature with features from other scales to form the runway edge region.

[0064] The feedback control processor uses a closed-loop evaluation function. Evaluate the fusion effect.

[0065] Normalized quality results before update The score is 0.76, indicating a good task perception result. The parameter change is 0.75. The reliability ratio is 0.15, indicating a low reliability percentage. The value is 0.20; take , , , Calculations yielded The value is 0.2265. After local registration update, Increased to 0.82, Increased to 0.86, It dropped to 0.08. Reduced to 0.14, recalculated to obtain The value was 0.145. The closed-loop evaluation value decreased from 0.2265 to 0.145, a decrease of 0.0815, while the task perception result did not decrease, and the feedback control processor retained the update. Subsequently, for five consecutive fusion cycles, the change in the closed-loop evaluation value was less than 0.005, the change in the task perception result was less than 0.004, and the change in the gating parameter was less than 0.003, and the system entered a stable holding state.

[0066] Example 2: This example is a further optimized system implementation based on Example 1. In this example, the airborne edge computing node includes a central processing unit, a graphics processing unit, a neural network processor, and a high-speed image interface. Visible light sensors, infrared sensors, and low-light sensors receive the same trigger signal via a synchronization trigger line, and the image enters a circular buffer via a high-speed serial interface. The meteorological parameter acquisition component reads visibility, humidity, temperature, and precipitation data through the airborne bus and receives information from ground meteorological stations via a data link.

[0067] The central processing unit (CPU) handles time synchronization, meteorological state vector construction, and feedback control; the graphics processing unit (GPU) performs image resampling, multi-scale feature encoding, and reconstruction; and the neural network processor (NN) develops the degradation assessment network and the flight mission perception model. These three processing units exchange aligned image groups, modal reliability fields, and fused flight visual images via shared memory. Each data object contains a version number, which enables the feedback control processor to determine the meteorological state, calibration parameters, and gating parameters used in the fused image.

[0068] When discrepancies arise between multi-source meteorological data, the system first performs a source consistency check. If the difference between airborne visibility data and ground-based meteorological station visibility data exceeds the source tolerance, and the image degradation trend is more consistent with the ground data, the bias attributor lists the airborne visibility sensor as a source to be verified. The feedback control processor increases the airborne visibility sampling frequency and temporarily improves the reliability of ground-based meteorological station data in the meteorological state vector. Adjustments to data sources do not directly change the image gating parameters; the new meteorological state affects the modal reliability field only after passing through the degradation evaluation network.

[0069] The system operates at two frequencies. Image synchronization, reliability assessment, and hierarchical fusion run at the image frame rate; meteorological trend estimation and bias attribution run at a lower frequency. If the task perception results change abruptly, the bias attribution process can be triggered by the image thread. Frame-level processing maintains a fixed processing budget, and feedback updates are written to the new version parameters at safe points between frames. The old version parameters are retained until the new version passes the closed-loop evaluation before being released.

[0070] In this embodiment, the degradation evaluation network employs a convolutional encoder and a lightweight attention encoder. The input image is first scaled to the processing resolution, and the output modal reliability field is then upsampled to the fusion resolution via edge guidance. Model training uses synchronized data of fog, rain, snow, and low illumination, and uses runway regions to label the task preservation degree. The model is run with quantized parameters, and the modal reliability ranking remains consistent before and after quantization. The system records quantization error and changes in closed-loop evaluation; when quantization error causes a decline in task perception results, it switches to a high-precision model.

[0071] This embodiment also includes a safety watchdog. The watchdog monitors image frame time difference, registration reliability, processing latency, and overall modal reliability. When processing latency continuously exceeds limits, the system reduces the processing resolution of non-task areas, while maintaining the original processing resolution for runway and obstacle areas. When the overall modal reliability falls below a preset safety threshold, the system outputs the single modal image with the highest reliability and freezes the feedback update.

[0072] Example 3: This example is based on the same principles as Example 1, except that it is applied to flight training simulation. Visible light images, infrared images, and low-light images can be generated by the simulation rendering engine according to the same scene conditions, or they can be generated by replaying real-world data. Meteorological parameters are configured by the training subjects and output by the simulated meteorological model. The simulation environment can provide true visibility values, precipitation particle distribution, runway location, and obstacle location for model validation and training evaluation.

[0073] The training scenario configuration includes approach heading, glide phase, runway visual range (DVR) rating, and key features to be identified. The flight mission perception branch selects runway threshold, centerline, edge lights, horizon, and obstacle models based on the training scenario configuration. The mission perception results in the fused image are compared with the simulation ground truth to obtain the positional deviation, continuity, and occurrence time deviation of key features. The deviation attributor analyzes both mission deviation and no-reference quality anomalies, avoiding direct adjustment of the fused results based solely on the simulation ground truth.

[0074] The feedback update log includes the meteorological state vector, modal reliability field summary, fusion bias sources, updated parameter set, closed-loop evaluation metrics before and after the update, and flight mission perception results. After training, the loop recording module saves the pre-fusion image, the fused image, pilot operation data, and the feedback update log. The debriefing program can freeze a specified parameter set on the same input and compare the impact of different feedback paths on the visibility of the mission target.

[0075] When the training scenario is set to low-visibility approach, the system first loads the pending gating parameters based on weather change trends. Upon arrival of a new batch of images, the degradation assessment network generates the actual modal reliability field. The system is confirmed when the pending gating parameters match the actual reliability; it is regressed when the pending gating parameters cause a decrease in runway edge continuity. Predictive adjustments are only used to shorten the transition time after sudden weather changes and do not replace feedback confirmation based on image quality and task awareness.

[0076] The training evaluation module can statistically analyze pilots' control response times after the appearance of key visual information, but control response time is not directly used as a technical effect of the fusion algorithm. The system correlates control response time with runway edge continuity, obstacle confidence, and visual reliability cues to determine whether visual information is presented in the set time sequence during training exercises. The resulting replay data is used for offline calibration of stability thresholds and mission area weights.

[0077] Example 4: This example details the degradation assessment extension and structural tensor detail preservation mechanism of the method of the present invention under complex meteorological conditions for strong light source forward scattering effects. This example builds upon the system architecture of Example 2, further refining the physical model mapping process in the degradation assessment network to address severe local degradation, as follows: When an aircraft performs a Category II or Category III instrument landing approach, the ground approach lighting system and runway centerline lights emit high-intensity directional beams. The visible light sensor and low-light sensor in the multimodal image acquisition assembly receive photons formed by the intense Mie scattering of suspended water droplets, resulting in a large-area bright halo on the image surface. This halo from the active light source obscures the physical edges of the runway, causing conventional global sharpness evaluation metrics to fail. Infrared sensors, operating in the long-wave or mid-wave band, can penetrate water droplets of this size and are unaffected by this type of scattering halo.

[0078] To address the extremely uneven local physical degradation phenomenon mentioned above, the reliability assessment processor simultaneously initiates an active light source scattering halo region identification mechanism while extracting atmospheric scattering degradation features from visible light images and low-light noise degradation features from low-light images. The system traverses the pixel matrices of both visible and low-light images, filters the set of pixels whose brightness values ​​exceed the sensor saturation threshold, and connects adjacent pixels to form peak connectivity regions. The system then establishes a local brightness peak point at the geometric center of each peak connectivity region.

[0079] The reliability assessment processor extracts pixel brightness sequences along eight discrete spatial directions, centered on the local brightness peak point. It calculates the decreasing derivative of pixel brightness with increasing radial distance and takes the average value as the radial gradient decay rate. Under clear skies, light edges are sharp, resulting in a high radial gradient decay rate; under dense fog, light edges exhibit Gaussian dispersion, leading to a low radial gradient decay rate. When the radial gradient decay rate falls below a preset standard gradient threshold, the system confirms that the bright area is not the actual outline of a physical light source, but rather a halo region caused by active light source scattering due to meteorological conditions.

[0080] To accurately eliminate interference from the active light source's scattered halo in the modal reliability field, the reliability assessment processor uses visibility and relative humidity data provided by the meteorological parameter acquisition component to construct a halo spatial expansion model and calculate the dynamic suppression coefficient field. The specific mathematical relationships of the halo spatial expansion model are defined as follows:

[0081] In the formula, Indicates spatial location The dynamic inhibition coefficient at the location; This represents the system's preset halo suppression gain constant; Indicated based on visibility and relative humidity The meteorological scattering intensity mapping function increases exponentially with lower visibility and higher relative humidity. This represents the total number of local brightness peaks identified in the field of view; This represents the cumulative index of the local brightness peak points; Indicates the first The coordinates of a local brightness peak point; The scattering radius, derived from the meteorological state vector, is positively correlated with the concentration of water vapor condensation nuclei in the environment. This indicates the operation of natural exponents.

[0082] The aforementioned dynamic suppression coefficient, acting as a penalty multiplier, is directly multiplied and superimposed onto the atmospheric scattering degradation feature matrix of the visible light image and the low-light noise degradation feature matrix of the low-light image. In the central region of the active light source scattering halo, the dynamic suppression coefficient is significantly larger, forcing the degradation assessment network to identify this area as an extremely high degradation region. The visible light reliability value and low-light reliability value output by the degradation assessment network experience a precipitous drop within the active light source scattering halo region, and the normalization mechanism subsequently pushes the infrared reliability value to an extremely high level close to 1. This localized, physically-based forced intervention blocks the penetration of the strong scattering light curtain into the fused image.

[0083] After receiving the modal reliability field with deep halo suppression, the hierarchical fusion processor performs luminance fusion and chromaticity transfer, followed by detail preservation processing. Because the physical boundaries of visible light within the halo region scattered by the active light source are disrupted, while the infrared image retains the true outline of the runway lighting fixture's heating element or the runway boundary, cross-modal registration residuals can easily lead to edge direction determination conflicts at this point. To resolve this conflict, the hierarchical fusion processor calculates the multimodal joint structure tensor.

[0084] The calculation process of the multimodal joint structure tensor is executed in the shared memory of the graphics processor via the data bus, and the matrix representation of the multimodal joint structure tensor is as follows:

[0085] In the formula, Indicates time In position The 2×2 multimodal joint structure tensor matrix at the location; It represents the set of modes for visible light, infrared light, and low-light. Indicates visible light, infrared, or low-light modes; This represents the modal reliability field output by the degradation assessment network at time [time]. and location Mode The normalized reliability value; and Representing modes Local spatial gradients in the horizontal and vertical directions of the image.

[0086] The layered fusion processor performs a layered fusion process on each pixel location. Perform eigenvalue decomposition on the matrix to find the principal eigenvalues. Secondary eigenvalues (and and the corresponding principal feature vectors and secondary eigenvectors The principal eigenvalue represents the energy magnitude in the direction of the most drastic local gray-level change in the image, while the secondary eigenvalue represents the energy of gray-level changes along the edge direction. The system calculates the eigenvalue ratio. ,in It is a very small division-by-zero constant. When When the value is much greater than 1, the system determines that the location belongs to the edge region; when and When both values ​​are large and close, it is determined to be a corner region; when both values ​​are small, it is determined to be a flat region.

[0087] At the coordinates of the region determined to be an edge area, the multimodal joint structure tensor is generated using the modal reliability field as weights. Therefore, in the active light source scattering halo coverage area, since the infrared reliability value is absolutely dominant, and This determines the directional distribution of the tensor of the multimodal joint structure. The extracted secondary eigenvectors... The direction is the actual runway edge direction in the infrared image that is not affected by the light curtain. The system defines it as the cross-modal common edge direction.

[0088] The hierarchical fusion processor then deploys anisotropic filtering modules. These anisotropic filtering modules are positioned along the sub-feature vector. The direction, i.e., the cross-modal common edge direction, is configured with a one-dimensional Gaussian smoothing kernel to perform smoothing filtering, eliminating false spikes and band breaks caused by cross-modal registration residuals or micro-light photon noise; simultaneously, along the principal feature vector... The direction of the normal, which is perpendicular to the direction of the shared cross-modal edge, is configured with a one-dimensional Laplacian kernel to perform high-pass filtering, extracting the true sharp edges of the cross-modal structures. This bidirectional spatial filtering operation, while ensuring that no double edge artifacts are generated, imposes the structural constraints of the infrared into the reconstruction network where luminance and chrominance have intertwined, as a detail-preserving feature output.

[0089] After acquiring the final flight visual fusion image, the feedback control processor extracts the no-reference quality result and the flight mission perception result. If the humidity data from the meteorological parameter acquisition component is delayed, resulting in insufficient penalty from the halo spatial extension model, local blurring and breaks will appear at the runway edges of the fused image. The runway edge continuity measured by the mission perception model exhibits regional drops. The bias attributor matches this continuity drop at specific spatial locations with the brightness peak distribution characteristics of the halo region scattered by the active light source, and outputs the fusion gating bias conclusion. Based on this, the feedback control processor issues a selective update command to increase the halo suppression gain constant. The dynamic suppression coefficient is strengthened in the next image frame cycle until the change in runway edge continuity is less than 0.005 over three consecutive cycles, at which point the system re-enters a stable condition.

[0090] For example, the system is in a dense fog landing scenario with low visibility of 750 m and relative humidity of 96%. A runway centerline light forms a halo with a radius of 25 pixels in the visible light image, with a local brightness peak of 255 and a radial gradient decay rate of 6 / pixel (the standard gradient threshold for clear environments is 45 / pixel). The system identifies this as an active light source scattering halo region. Visibility and humidity are jointly driving the function. Output coefficient 4.5. The dynamic suppression coefficient of the halo center region, calculated using the halo spatial expansion model. It reached 18.5.

[0091] After superimposing the dynamic suppression coefficient into the visible light degradation feature, the atmospheric scattering degradation feature value of the active light source scattering halo region input degradation evaluation network surged from 0.56 to 10.36. The normalized reliability values ​​changed accordingly: the infrared reliability value jumped from 0.38 before halo suppression was enabled to 0.91, while the visible light reliability value dropped from 0.45 to 0.03. Multimodal joint structure tensor matrix. The principal feature vectors are perfectly aligned with the boundaries of the real light structure captured by the infrared sensor. Before the update, due to light curtain crosstalk, the runway edge continuity output by the task perception model was only 0.41. After implementing the above dynamic suppression and tensor detail preservation mechanism, the closed-loop evaluation value dropped significantly, and the runway edge continuity jumped to 0.89, achieving strong anchoring of key visual guidance elements in complex optical degradation scenarios.

[0092] A set of continuous data is used to illustrate the complete processing chain of this invention. For example: at time 1, the visibility is 0.9 km, the relative humidity is 88%, and the precipitation intensity is 3 mm / h; at time 2, the visibility is 0.75 km, the relative humidity is 90%, and the precipitation intensity is 5 mm / h; at time 3, the visibility is 0.62 km, the relative humidity is 91%, and the precipitation intensity is 6 mm / h. After time filtering, the meteorological parameters show an increasing trend of fog and rain degradation, and the meteorological uncertainty increases from 0.06 to 0.09.

[0093] The synchronous registration processor acquired 36 local control points at time 1, with an average registration residual of 0.42 pixels; 32 control points at time 2, with an average registration residual of 0.58 pixels; and 24 control points at time 3, affected by rain streaks, with an average registration residual of 0.91 pixels. The registration reliability map decreased at the far end of the runway and in areas with dense rain streaks, indicating that the registration information entered the modal reliability field and was not treated as ordinary image details.

[0094] At the far end of the runway at time 3, the degradation degree for visible light is 0.72, for infrared light 0.29, and for low-light light 0.55; the corresponding modal correlation prior uncertainties are 0.09, 0.07, and 0.08; and the cross-modal inconsistencies are 0.24, 0.12, and 0.19. According to the aforementioned reliability value function relationship, the normalized reliability for visible light, infrared light, and low-light light is 0.216, 0.475, and 0.309, respectively. The reliability ranking is consistent with the infrared thermal profile preservation capability under fog and rain conditions.

[0095] The brightness encoding features of the three modes at the mesoscale are set to 0.31, 0.81, and 0.49, respectively, while the cross-modal compensation feature is set to 0.03. The mesoscale fusion feature obtained according to the aforementioned fusion feature function relationship is 0.633. For fine-scale locations with low registration confidence, high-frequency superposition of infrared and low-light signals is not performed; instead, the infrared single-source edge is preserved, and chromaticity diffusion is limited by the visible light chromaticity boundary.

[0096] The normalized no-reference quality of the reconstructed image is 0.79, the flight mission perception quality is 0.83, the parameter variation is 0.04, and the proportion of low-reliability regions is 0.12%. Evaluation weights were set to 0.35, 0.40, 0.15, and 0.10 respectively, and the evaluation score obtained using the closed-loop evaluation formula was 0.1595. Anomalies were concentrated at the far edge of the runway, the registration residuals coincided with the edge break space, and the deviation attributor output spatial registration deviation.

[0097] The feedback control processor increases the search range of local control points at the far end of the runway and improves the intensity of rain streak suppression preprocessing, while the meteorological sampling frequency, infrared degradation feature weights, and gating parameters for other areas remain unchanged. The updated registration residual decreases to 0.63 pixels, the no-reference quality improves to 0.82, the flight mission perception improves to 0.87, and the evaluation score decreases to 0.139. The update is retained.

[0098] In the subsequent five fusion cycles, the evaluation values ​​were 0.139, 0.137, 0.136, 0.136, and 0.135, respectively; the task perception results were 0.87, 0.872, 0.873, 0.873, and 0.874, respectively; and the changes in the gating parameters were all below 0.003. Continuous results met the stability condition, and the feedback control processor stopped updating and maintained the current configuration. When the meteorological trend continued to deteriorate, new quality anomalies could trigger bias attribution again.

[0099] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0100] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for fusing multimodal flight visual images under complex weather conditions, characterized in that, include: Visible light images, infrared images, and low-light images, along with meteorological parameters corresponding to the acquisition time, are collected from the same flight field of view. The visible light images, infrared images, and low-light images are then time-aligned and spatially registered to obtain an aligned image group. Structural features, radiation features, noise features, and cross-modal consistency features of each mode are extracted from the aligned image group, and combined with the meteorological parameters to generate a modal reliability field characterizing the degree of degradation and reliability of different spatial regions. Based on the modal reliability field, reliability gating and cross-modal compensation are applied to each modal feature to form luminance fusion features, chroma transfer features and detail preservation features, respectively, and to reconstruct the flight visual fusion image; Extract no-reference quality results and flight mission perception results from the flight visual fusion image, and determine the source of fusion deviation based on the no-reference quality results, the flight mission perception results, and the modal reliability field; According to the sources of the fusion deviation, the meteorological parameter acquisition configuration used for control acquisition, the spatial registration configuration used for control registration, the degradation assessment configuration used for control degradation assessment, or the reliability gating parameters are selectively updated, and the updating stops when the continuous quality results meet the stability conditions, and the flight visual fusion image is output.

2. The method for multimodal flight visual image fusion under complex weather conditions according to claim 1, characterized in that: The time alignment and spatial registration include: establishing a synchronized image group based on hardware synchronization signals, image timestamps, and meteorological parameter timestamps; projecting the infrared image and the low-light image onto the visible light image coordinate system using sensor extrinsic parameters; correcting the projection relationship based on cross-modal edges, stable ground object contours, and local mutual information; outputting the registration image, registration residual map, and registration confidence map in the aligned image group, and inputting the registration residual map and the registration confidence map into the generation process of the modal reliability field.

3. The method for multimodal flight visual image fusion under complex weather conditions according to claim 1, characterized in that: The meteorological parameters include at least three of the following: visibility, relative humidity, ambient temperature, wind speed, wind direction, precipitation type, and precipitation intensity. Continuous meteorological parameters are subjected to time filtering and trend estimation to form a meteorological state vector and a meteorological uncertainty. The meteorological state vector is used to determine the meteorological degradation prior, and the meteorological uncertainty is used to limit the update amplitude of the modal reliability field.

4. The method for multimodal flight visual image fusion under complex weather conditions according to claim 2, characterized in that: The generation process of the modal reliability field includes: determining the atmospheric scattering degradation features of the visible light image, the thermal contrast degradation features of the infrared image, and the low-light noise degradation features of the low-light image; inputting each modal degradation feature, meteorological degradation prior, the registration residual map, and the cross-modal consistency features into the degradation evaluation network; outputting the modal reliability values ​​corresponding to each spatial region by the degradation evaluation network, and performing normalization on the modal reliability values ​​of different modes, so that the modal reliability values ​​at the same spatial location are used to constrain the subsequent fusion contribution.

5. The method for multimodal flight visual image fusion under complex weather conditions according to claim 3, characterized in that: The generation process of the modal reliability field includes: determining the atmospheric scattering degradation features of the visible light image, the thermal contrast degradation features of the infrared image, and the low-light noise degradation features of the low-light image; inputting each modal degradation feature, the meteorological degradation prior, the meteorological uncertainty, and the cross-modal consistency features into the degradation evaluation network; outputting the modal reliability values ​​corresponding to each spatial region by the degradation evaluation network, and performing normalization on the modal reliability values ​​of different modes, so that the modal reliability values ​​at the same spatial location are used to constrain the subsequent fusion contribution.

6. The method for multimodal flight visual image fusion under complex weather conditions according to claim 4 or 5, characterized in that: The reliability gating and cross-modal compensation include: performing regional adaptive weighting on multi-scale brightness features based on the modal reliability field; determining the chroma transfer region based on visible light chroma confidence and scene semantic boundaries; determining the detail preservation direction based on the edge response of each modality and the modal reliability field; using the brightness fusion feature as the brightness constraint for chroma transfer; and inputting the chroma transfer feature and the detail preservation feature together into the neural network for reconstruction, so that the output of the previous processing constitutes the input or constraint of the next processing.

7. The method for multimodal flight visual image fusion under complex weather conditions according to claim 1, characterized in that: The no-reference quality results include local contrast, sharpness, noise persistence, color stability, and temporal flicker; the flight mission perception results include at least two of the following: runway edge continuity, runway centerline visibility, horizon stability, obstacle detection confidence, and ground feature segmentation consistency. Based on the abnormal distribution of the no-reference quality results and the flight mission perception results in spatial and temporal locations, meteorological parameter measurement bias, spatial registration bias, single-mode degradation estimation bias, and fusion gating bias are distinguished.

8. The method for multimodal flight visual image fusion under complex weather conditions according to claim 7, characterized in that: The selective updates include: when the source of the fusion bias is a meteorological parameter measurement bias, increasing the sampling frequency of the relevant meteorological parameters or switching the meteorological data source; when the source of the fusion bias is a spatial registration bias, updating the local registration control points and image preprocessing intensity; when the source of the fusion bias is a single-mode degradation estimation bias, updating the degradation feature weights of the corresponding mode; when the source of the fusion bias is a fusion gating bias, updating the reliability gating parameters of the corresponding spatial region, while parameter groups not identified as sources of bias remain unchanged.

9. The method for multimodal flight visual image fusion under complex weather conditions according to claim 8, characterized in that: The stability conditions include the changes in the no-reference quality result, the changes in the flight mission perception result, and the changes in the reliability gating parameter all being lower than the corresponding preset stability thresholds within multiple consecutive fusion cycles; when the reliability values ​​of all modes are lower than the preset safety threshold, the closed-loop update is stopped and the single-mode image with the highest reliability or the single-mode image with the highest reliability after conservative enhancement is output, while a visual reliability prompt is generated.

10. A multimodal flight visual image fusion system under complex weather conditions, used to execute the multimodal flight visual image fusion method under complex weather conditions as described in any one of claims 1 to 9, characterized in that: It includes a multimodal image acquisition component, a meteorological parameter acquisition component, a synchronous registration processor, a reliability assessment processor, a hierarchical fusion processor, a feedback control processor, a memory, and a data bus; The multimodal image acquisition component includes a visible light sensor, an infrared sensor, and a low-light sensor, and outputs images of the same flight field of view to the synchronization registration processor via a data bus; The meteorological parameter acquisition component outputs timestamped meteorological parameters to the reliability assessment processor and the feedback control processor; The synchronous registration processor outputs the aligned image group, the registration residual map, and the registration confidence map to the reliability evaluation processor; The reliability assessment processor generates a modal reliability field based on the image features extracted from the aligned image group and the received meteorological parameters, and outputs it to the hierarchical fusion processor. The hierarchical fusion processor forms a flight visual fusion image based on the modal reliability field, and outputs the flight visual fusion image to the display interface and the feedback control processor; The feedback control processor sends selective update instructions to the meteorological parameter acquisition component, the synchronization registration processor, the reliability assessment processor, or the hierarchical fusion processor based on the no-reference quality results extracted from the flight visual fusion image, the flight mission perception results, and the determined fusion deviation sources. The memory stores the processing program, model parameters, and feedback update records.

Images