A multispectral image lens foreign matter detection and repair method and system

Abstract

The present application relates to the field of image processing, and provides a multispectral image lens foreign matter detection and repair method and system, comprising: acquiring a plurality of multi-band synchronous image frames collected by cameras with at least partially overlapping fields of view; performing target-level cross-band matching on the multi-band synchronous image frames, and predicting the motion trajectory of each real scene target based on the matching result through a motion trajectory prediction model; identifying a region in the multi-band synchronous image frames that does not have a cross-band corresponding relationship and whose motion characteristics deviate from the predicted behavior of the real scene target, determining it as a lens foreign matter region and generating a foreign matter mask; based on the foreign matter mask, using the guide information of the unobstructed band image to reconstruct the content of the lens foreign matter region, and constraining the reconstruction result to be consistent with the reference characteristics of the unobstructed band and the spectral response of the current band, and generating a repaired image. The present application realizes accurate discrimination and spectral fidelity repair of lens foreign matter, and guarantees the continuous execution of multispectral high-precision tasks.

Description

Technical Field

[0001] This invention relates to the field of image processing, and specifically to a method and system for detecting and repairing foreign objects in multispectral image lenses. Background Technology

[0002] Currently, in the field of image acquisition, such as airborne multispectral optoelectronic pods, multiple cameras with a common field of view are typically used to simultaneously acquire multi-band images to achieve all-weather, all-time scene perception. However, in actual flight, camera lenses are easily contaminated with raindrops, dust, insects, and other foreign objects, causing partial image occlusion. In existing technologies, single-camera pure vision solutions rely on the assumption of "static foreign objects and moving scene," resulting in extremely high false positive and false negative rates in hovering and weak-texture scenes. Multi-camera solutions mostly use pixel-level fusion, failing to utilize cross-band spectral characteristics and thus unable to fundamentally distinguish lens foreign objects from real scene targets. Furthermore, most restoration solutions rely only on single-channel temporal information, leading to distorted spectral characteristics after restoration, which cannot meet the requirements of high-precision multispectral tasks. Summary of the Invention

[0003] To address the problems of existing technologies that cannot accurately distinguish between foreign objects in the lens and real-world targets, and that restoration can easily lead to spectral distortion, this invention provides a multispectral image lens foreign object detection and restoration method and system. This method utilizes the characteristic of at least partial overlap of multi-band fields of view, combined with trajectory prediction and image plane stillness, to identify foreign objects. It is further supplemented by cross-band spectral fidelity restoration to achieve accurate identification of foreign objects and spectral fidelity image restoration.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: A method for detecting and repairing foreign objects in a multispectral image lens includes: acquiring multi-band synchronous image frames from cameras with at least partial overlap of multiple fields of view; performing target-level cross-band matching on the multi-band synchronous image frames, and predicting the motion trajectory of each real-scene target based on the matching results using a motion trajectory prediction model; identifying regions in the multi-band synchronous image frames that do not have cross-band correspondence and whose motion characteristics deviate from the predicted behavior of real-scene targets, determining them as lens foreign object regions, and generating a foreign object mask; based on the foreign object mask, using guiding information from the unobstructed band image to reconstruct the content of the lens foreign object region, and constraining the spectral response of the reconstruction result in the current band to be consistent with the reference features of the unobstructed band, thereby generating a repaired image.

[0005] Optionally, the motion trajectory prediction model is a Kalman filter model; the state vector of the Kalman filter model includes the pixel coordinates of the target, the image plane velocity, and the image plane acceleration; the observation vector of the Kalman filter model includes the actual observation coordinates of the same target by cameras in each band; the process noise of the Kalman filter model is dynamically set based on the body angular velocity, linear acceleration, and gimbal angular velocity, and the observation noise is dynamically set based on at least one of image sharpness, target detection confidence, exposure status, and environmental visibility.

[0006] Optionally, the determination of a foreign object region in the lens satisfies the following joint criteria: a candidate target exists in the current band image and no valid matching target is found in other band images, wherein a valid matching target must satisfy the following conditions: the similarity of the target descriptor is greater than a preset similarity threshold, the spatial registration error is less than a preset error threshold, and the time synchronization tolerance is less than a preset time threshold; the center position drift of the candidate target between adjacent frames is less than a preset drift threshold and the contour intersection-union ratio is greater than a preset intersection-union ratio threshold; the deviation between the position of the candidate target and the predicted position of the motion trajectory is greater than a dynamic threshold, wherein the dynamic threshold is determined based on a weighted sum of a base threshold, the uncertainty corresponding to the filter prediction covariance, and the gimbal angular velocity; in the detection of M consecutive frames, each frame satisfies the above conditions, where M is an integer greater than 1.

[0007] Optionally, the step of reconstructing the content of the foreign object region of the lens using the guiding information of the unobstructed band image includes: acquiring the obstructed band image, the foreign object mask, the unobstructed band guiding image, temporal frame data, and camera calibration parameters, inputting them into an image restoration model, and outputting the restored image; the image restoration model is configured to generate the restored image based on the constraints of a total loss function, the total loss function including at least a spectral fidelity loss term, and optionally one or more of a reconstruction loss term, an edge consistency loss term, a temporal consistency loss term, and a task preservation loss term; the spectral fidelity loss term is determined by: calculating the first norm distance between the spectral response features of the restored image in the current band and the spectral response reference features; calculating the second norm distance between the radiometric normalization result of the restored image and the mapping result from the responses of other unobstructed bands to the response of the current band; and determining the spectral fidelity loss term based on the weighted sum of the first norm distance and the second norm distance.

[0008] Optionally, the method further includes: performing intrinsic parameter calibration and extrinsic parameter joint calibration on cameras with at least partial overlap of the multiple fields of view, establishing a mapping relationship between the pixel coordinate system of each camera and a unified coordinate system, and achieving pixel-level spatial registration; using a globally unified clock as a reference, aligning the timestamps of the multiple image frames through interpolation resampling, and setting a time synchronization tolerance; when the time difference between the multiple image frames exceeds the time synchronization tolerance, initiating interpolation compensation or increasing the similarity threshold of cross-band matching.

[0009] Optionally, the method further includes: calculating an alarm score based on at least one of the following: the occlusion area of ​​the foreign object region on the lens, the overlap ratio with the task center area, the task weight of the camera, the number of continuous frames, and the decrease in downstream recognition confidence; classifying alarm levels according to the alarm score, with different alarm levels corresponding to different handling strategies, the handling strategies including at least one of ground station recording, audible and visual alarms and gimbal adjustment suggestions, and emergency handling strategies.

[0010] Optionally, the method further includes: saving foreign object sample data during device operation and updating only at least one of the threshold parameter, matching weight, and normalized statistics; uploading the foreign object sample data to the cloud after the device stops operating, and completing model retraining and threshold recalibration through the cloud; sending the updated model and parameters to the device after version verification, and retaining the previous version as a rollback version; keeping the detection and repair model on the device unchanged when consistency check and version signature verification fail.

[0011] This invention also provides a multispectral image lens foreign object detection and repair system, comprising: an acquisition module configured to acquire multi-band synchronous image frames acquired by cameras with at least partial overlap of multiple fields of view; a matching and prediction module configured to perform target-level cross-band matching on the multi-band synchronous image frames, and predict the motion trajectory of each real scene target based on the matching result using a motion trajectory prediction model; a foreign object identification module configured to identify regions in the multi-band synchronous image frames that do not have cross-band correspondence and whose motion characteristics deviate from the predicted behavior of real scene targets, determine them as lens foreign object regions, and generate a foreign object mask; and a repair module configured to reconstruct the content of the lens foreign object region based on the foreign object mask, using guiding information from the unobstructed band image, and constrain the spectral response of the reconstruction result in the current band to be consistent with the reference features of the unobstructed band, thereby generating a repaired image.

[0012] The present invention also provides an electronic device, including a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program to implement the method described above.

[0013] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method described above.

[0014] Beneficial effects: This invention utilizes target-level cross-band matching and motion trajectory prediction. Real-world targets can achieve cross-band matching imaging and conform to motion trajectory prediction in cameras with at least partial overlap of multiple fields of view. In contrast, foreign objects in the lens exist only on a single imaging plane and are stationary relative to the image plane, deviating from the predicted trajectory—a fundamental physical difference that enables accurate identification of lens foreign objects. This fundamentally solves the problem of high false positive and false negative rates caused by the difficulty in distinguishing foreign objects from real targets in traditional solutions. Simultaneously, during the restoration phase, guidance information from unobstructed band images is used for content reconstruction. A spectral fidelity loss term constrains the reconstruction result to maintain consistency between the spectral response of the current band and the reference features of the unobstructed band, avoiding spectral distortion caused by simple replacement of the restoration area with textures from other bands. This ensures the continuous execution of high-precision multispectral tasks. Furthermore, by dynamically adjusting process noise and observation noise, the robustness of trajectory prediction under complex flight conditions is improved. A graded alarm and edge-cloud collaborative closed-loop mechanism enables graded risk handling and continuous iterative optimization of the model, significantly improving the mission reliability and flight safety of the airborne multispectral optoelectronic system. Attached Figure Description

[0015] Figure 1 This is a flowchart of a multispectral image lens foreign object detection and repair method according to a specific embodiment of the present invention; Figure 2 This is a structural block diagram of the multispectral image lens foreign object detection and repair system according to a specific embodiment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the specific embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described specific embodiments are only a part of the specific embodiments of this invention, not all of them. All other specific embodiments obtained by those skilled in the art based on the specific embodiments of this invention without creative effort are within the scope of protection of this invention.

[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0018] like Figure 1As shown in the illustration, this specific embodiment provides a method for detecting and repairing foreign objects in multispectral image lenses. This method is primarily executed by the image processing platform of a multispectral optoelectronic pod, covering a closed-loop process from multi-band image matching and foreign object identification to spectral fidelity restoration. Specifically, it includes the following steps: Step S100: Acquire multi-band synchronized image frames acquired by cameras with at least partial overlap of multiple fields of view.

[0019] Specifically, cameras with at least partial overlap in their fields of view can be visible light cameras, near-infrared cameras, mid-wave infrared cameras, etc. Before this step, a pre-processing spatiotemporal synchronization and joint calibration is performed: intrinsic parameter calibration and extrinsic parameter joint calibration are performed on each camera with at least partial overlap in their fields of view to establish a mapping relationship between the pixel coordinate systems of each camera and a unified coordinate system, achieving pixel-level spatial registration; using a globally unified clock as a reference, the timestamps of the multiple image frames are aligned through interpolation resampling, and a time synchronization tolerance is set; when the time difference between multiple image frames exceeds the time synchronization tolerance, interpolation compensation is initiated or the similarity threshold for cross-band matching is increased. For example, intrinsic parameter calibration can obtain the camera's focal length, principal point coordinates, and distortion coefficients; extrinsic parameter joint calibration can obtain the camera's rotation and translation matrices relative to the fuselage coordinate system. The time synchronization tolerance can be dynamically set according to flight speed and attitude change rate, such as 10 milliseconds for high-speed flight and 20 milliseconds for hovering. When the time difference exceeds the tolerance, historical frames are resampled and compensated using a linear interpolation algorithm, or the similarity threshold of that frame in subsequent cross-band matching is directly increased to tighten the matching conditions and prevent mismatches caused by spatiotemporal asynchrony. By establishing a unified coordinate system and time synchronization tolerance, a precise spatiotemporal reference is provided for subsequent cross-band feature matching, avoiding registration errors caused by camera asynchrony or viewpoint differences.

[0020] Step S200: Target-level cross-band matching is performed on the multi-band synchronized image frames, and the motion trajectory of each real-world target is predicted based on the matching results using a motion trajectory prediction model.

[0021] Specifically, target-level cross-band matching overcomes the noise-introducing shortcomings of traditional pixel-level fusion. It uses independent, strongly feature-rich targets within a scene as basic units, extracting core information such as target pixel coordinates, contours, feature descriptors, and spectral response characteristics. Cross-band matching employs a bidirectional nearest neighbor plus threshold plus geometric consistency rule: when target A's best match in another band is B, and B matches back to A, and the matching similarity is greater than a preset similarity threshold, the spatial registration error is less than a preset error threshold, and the time synchronization tolerance is less than a preset time threshold, it is considered a valid cross-band match. For example, the preset similarity threshold can adaptively range from 0.60 to 0.85, the preset error threshold can range from 2 to 5 pixels, and the preset time threshold can range from 10 to 30 milliseconds. If a candidate target exists only in one path and no matching target meeting the above conditions is found in the other two paths, it enters the suspected foreign object queue. Through bidirectional matching and multi-dimensional threshold constraints, mismatched targets are effectively eliminated, improving the accuracy of cross-band matching.

[0022] In this specific embodiment, the motion trajectory prediction model is a Kalman filter model; the state vector of the Kalman filter model includes the target's pixel coordinates, image plane velocity, and image plane acceleration; the observation vector of the Kalman filter model includes the actual observation coordinates of the same target by cameras in each band; the process noise of the Kalman filter model is dynamically set based on the body angular velocity, linear acceleration, and gimbal angular velocity, and the observation noise is dynamically set based on at least one of image sharpness, target detection confidence, exposure status, and environmental visibility. For example, the state vector can be represented as... ,in , For reference image coordinates, , For the image plane velocity, , The image plane acceleration is represented as the observation vector can be expressed as... These correspond to the observation coordinates in visible light, near-infrared, and mid-infrared, respectively. The covariance matrix Q of the process noise can be dynamically amplified based on the modulus of the aircraft's angular velocity, linear acceleration, and gimbal angular velocity. When the aircraft is in high-maneuver flight, the process noise is increased to adapt to the uncertainty of the target's motion state. The covariance matrix R of the observation noise can be dynamically set based on image sharpness, target detection confidence, exposure status, and environmental visibility. When environmental visibility is low or the image is blurry, the observation noise is increased to reduce the weight of the observation data. By dynamically adjusting the process noise and observation noise, the Kalman filter model can adapt to complex flight conditions, improving the robustness of trajectory prediction.

[0023] Step S300: Identify regions in multi-band synchronous image frames that do not have cross-band correspondence and whose motion features deviate from the predicted behavior of the target in the real scene, determine them as foreign object regions of the lens, and generate foreign object masks.

[0024] Specifically, the following joint criteria are used to determine if a region is a foreign object in the lens: a candidate target exists in the current band image and no valid matching target is found in other band images. A valid matching target must satisfy the following conditions: target descriptor similarity greater than a preset similarity threshold, spatial registration error less than a preset error threshold, and time synchronization tolerance less than a preset time threshold; the center position drift of the candidate target between adjacent frames is less than a preset drift threshold, and the contour intersection-over-union (IoU) is greater than a preset IoU threshold; the deviation between the candidate target's position and the predicted position of its motion trajectory is greater than a dynamic threshold, which is determined based on a weighted sum of the base threshold, the uncertainty corresponding to the filter prediction covariance, and the gimbal angular velocity. In the detection of M consecutive frames, each frame satisfies the above conditions, where M is an integer greater than 1. For example, M can be 3 to 10 frames, the preset drift threshold can be 1 to 3 pixels, and the preset IoU threshold can be 0.6 to 0.9. Dynamic threshold The calculation formula can be ,in Based on the threshold, The uncertainty corresponding to the filter prediction covariance. The angular velocity of the gimbal. and These are weighting coefficients. The uncertainty corresponding to the Kalman filter prediction covariance in hovering or high-maneuver scenarios. and gimbal angular velocity It will increase, thereby dynamically raising the dynamic threshold. This prevents misjudgments caused by inaccurate predictions or rapid gimbal rotation. Furthermore, it distinguishes between translucent and opaque foreign objects based on the grayscale attenuation rate of the occluded area, local contrast changes, edge blurring, and cross-frame texture preservation: a translucent foreign object is identified when the area retains background texture but exhibits low-frequency brightness or contrast attenuation; an opaque foreign object is identified when the area lacks texture, has strong boundaries, and a significantly reduced signal-to-noise ratio. Different types of foreign objects generate different confidence levels and different restoration weights, avoiding misjudging real low-texture areas as lens contamination. Through these joint criteria, real-scene targets can achieve cross-band matching imaging and conform to motion trajectory prediction in cameras with at least partial overlap of multiple fields of view, while lens foreign objects exist only on a single imaging plane and are stationary relative to the image plane, deviating from the predicted trajectory—a fundamental physical difference that enables accurate identification of lens foreign objects. This fundamentally solves the problem of high false positive and false negative rates caused by the difficulty in distinguishing foreign objects from real targets in traditional solutions.

[0025] After identifying a foreign object area in the lens, an alarm score can be calculated based on at least one of the following: the obstruction area of ​​the foreign object area, the overlap ratio with the task center area, the task weight of the camera, the number of consecutive frames, and the decrease in downstream recognition confidence. Alarm levels are then classified according to the alarm score, with different handling strategies corresponding to different alarm levels. These strategies include at least one of ground station recording, audible and visual alarms and gimbal adjustment suggestions, and emergency response strategies. For example, the alarm score R can be calculated from... The calculation is performed, where A is the proportion of the occluded area to the total image area, and C is the overlap ratio between the occluded area and the task center area or target area. The weight of the camera in the current task is T, where T is the normalized value of duration or consecutive frame count. To help downstream identify or track the decrease in confidence, to This is a weighting coefficient. Minor alarms can be addressed accordingly. If the obstruction area is less than 3% and does not cover the mission center area, only the camera number, object type, mask area, confidence level, and timestamp are recorded at the ground station; a moderate alarm can be addressed accordingly. If the obstruction area is 3% to 15% or affects a single task target, an audible and visual alarm will be output simultaneously, and a pan-tilt adjustment suggestion will be generated based on the location of the foreign object mask, such as moving the target area out of the obstruction area or switching the task camera weight; a severe alarm can be correspondingly... If the obstruction area exceeds 15%, covers the central mission area, or causes a decrease in the confidence level of downstream key target identification exceeding 30%, a high-priority alarm message is sent to the flight control and ground stations, triggering mission degradation, return-to-home recommendation, hovering check, backup payload switching, or manual takeover prompts. Through tiered alarms and closed-loop handling, risk tiered handling and mission safety closed-loop are achieved, significantly improving the mission reliability and flight safety of the airborne multispectral optoelectronic system.

[0026] Step S400: Based on the foreign object mask, the content of the foreign object region of the lens is reconstructed using the guiding information of the unobstructed band image, and the spectral response of the reconstruction result in the current band is constrained to be consistent with the reference features of the unobstructed band, thereby generating a repaired image.

[0027] Specifically, the foreign object region of the lens is reconstructed using guiding information from unobstructed band images. This includes: acquiring obstructed band images, foreign object masks, unobstructed band guiding images, temporal frame data, and camera calibration parameters; inputting these into an image inpainting model; and outputting a repaired image. The image inpainting model is configured to generate the repaired image based on constraints of a total loss function. The total loss function includes at least a spectral fidelity loss term, and optionally one or more of a reconstruction loss term, an edge consistency loss term, a temporal consistency loss term, and a task preservation loss term. The spectral fidelity loss term is determined by: calculating the first norm distance between the spectral response features of the repaired image in the current band and the spectral response reference features; calculating the second norm distance between the radiometric normalization result of the repaired image and the mapping result from the responses of other unobstructed bands to the response of the current band; and determining the spectral fidelity loss term based on the weighted sum of the first and second norm distances. For example, image inpainting models can employ lightweight encoder-decoder, gated convolution, or partially convolutional structures, and incorporate cross-band attention modules to use the contours and textures of the unoccluded target as strong guiding information. The total loss function can be expressed as... ,in To conceal the reconstruction losses in the area, For edge gradient consistency loss, For time-series consistency loss, To preserve losses for downstream target detection or tracking tasks, To preserve spectral fidelity, , , , , These are the weighting coefficients for each loss term. Spectral fidelity loss can be calculated using... ,in For the repair result, For unobstructed true or false true values, The spectral or radiative response feature extraction function for the i-th band. Let be the radiation normalization function. This is a learning or calibration mapping that maps the responses of the other two bands to the response of the i-th band. The first norm distance constrains the absolute difference between the repaired result and the ground truth in spectral features, while the second norm distance constrains the consistency between the repaired result and the cross-band mapping result after radiometric normalization. This constraint ensures that the repaired area is not a simple RGB, NIR, or MWIR texture patchwork, but rather satisfies the consistency with the cross-band mapping result after radiometric normalization within the target band, and that the spectral response remains consistent with the reference features of the unoccluded band, avoiding spectral distortion caused by the simple replacement of the repaired area with textures from other bands. The training strategy can adopt a combination of pre-training with synthetic occlusion and fine-tuning with real flight samples: first, supervised training is performed by superimposing synthetic masks such as raindrops, dust, insect spots, and ice crystals onto unoccluded three-band synchronous data; then, weak supervision or self-supervised fine-tuning is performed using real contamination samples manually confirmed after flight; for real occluded areas where the ground truth cannot be obtained, temporally adjacent unoccluded frames, cross-band consistency, and downstream task performance can be used as pseudo-label constraints. Through the above repair mechanism, image repair with foreign object removal and spectral fidelity is achieved, ensuring the continuous execution of multispectral high-precision tasks.

[0028] Furthermore, this method also includes an incremental learning closed loop of edge-cloud collaboration: During device operation, foreign object sample data is saved, and only at least one of the threshold parameters, matching weights, and normalized statistics is updated; after the device stops operating, the foreign object sample data is uploaded to the cloud, and model retraining and threshold recalibration are completed through the cloud; after version verification, the updated model and parameters are sent back to the device, and the previous version is retained as a fallback version; when consistency checks and version signature verification fail, the detection and repair model on the device remains unchanged. For example, the device saves high-confidence foreign object samples, low-confidence problematic samples, and false positive or false negative candidate segments in real time; during operation, only a small number of threshold parameters, matching weights, or normalized statistics are updated, without directly updating the backbone network on a large scale, to ensure operational safety and model stability; after the device stops operating, samples, masks, filtering status, and alarm handling results are uploaded to the ground station or cloud; incremental datasets are generated through manual review or semi-automatic annotation; model retraining, threshold recalibration, and regression testing are completed in the cloud; after version verification, the data is sent back to the device, and the previous version is retained as a fallback version. When a new threshold causes an abnormal increase in alarm frequency or a higher risk of missed detections, the system automatically reverts to the default threshold. All models, thresholds, and training samples are linked and recorded with the number of flights, environmental conditions, and camera status. When consistency checks and version signature verification fail, the detection and repair models on the device side remain unchanged to prevent system failures caused by malicious tampering or transmission errors. Through edge-cloud collaborative incremental learning, continuous iterative optimization of the model and security boundary control are achieved, perfecting the system's self-evolutionary closed loop.

[0029] like Figure 2As shown in the illustration, this specific embodiment also provides a multispectral image lens foreign object detection and repair system. This device employs a virtual module division corresponding to the steps of the above method, mainly including an acquisition module, a matching and prediction module, a foreign object identification module, and a repair module. The modules interact at high speed via a data bus or shared memory to collaboratively implement the above method flow.

[0030] The acquisition module is configured to acquire multi-band synchronized image frames from multiple cameras with at least partial overlap in their fields of view. Specifically, the acquisition module serves as the data entry point for the entire device, responsible for receiving raw video streams from image acquisition hardware such as visible light cameras, near-infrared cameras, and mid-wave infrared cameras. The acquisition module integrates a timestamp alignment unit and a spatial registration unit. While acquiring image frames, it interpolates and resamples multiple image frames using a globally unified clock as a reference, and maps the pixel coordinates of each stream to a unified coordinate system using pre-calibrated intrinsic and extrinsic parameters. For example, the acquisition module can connect to the camera hardware via an onboard Ethernet or high-speed serial bus, packaging and sending the processed spatiotemporally synchronized multi-band image frames to subsequent modules. It should be understood that the acquisition module can be either a standalone data preprocessing chip or a hardware logic thread integrated into the main processor, as long as it fulfills the functions of multi-band image synchronization acquisition and spatiotemporal alignment.

[0031] The matching and prediction module is configured to perform target-level cross-band matching on multi-band synchronized image frames, and predict the motion trajectory of each real-world target based on the matching results using a motion trajectory prediction model. Specifically, after receiving the synchronized image frames output by the acquisition module, the matching and prediction module first extracts information such as pixel coordinates, contours, and feature descriptors of independent, strongly characteristic targets within the scene, and performs cross-band matching using a bidirectional nearest neighbor plus geometric consistency rule. For successfully matched real-world targets, the matching and prediction module internally constructs a Kalman filter model, using the target's historical state vector (including pixel coordinates, image plane velocity, and image plane acceleration) and the current observation vector to predict the trajectory. For example, the matching and prediction module can perform feature extraction and matrix operations in parallel using a field-programmable gate array (FPGA) or a graphics processing unit (GPU) to meet the real-time requirements of the airborne platform. By dynamically adjusting process noise and observation noise, the matching and prediction module can adapt to complex flight conditions and output highly robust real-world target motion trajectory prediction results.

[0032] The foreign object detection module is configured to identify regions in multi-band synchronized image frames that lack cross-band correspondence and whose motion characteristics deviate from the predicted behavior of the target in the real scene, thus determining them as foreign object regions and generating a foreign object mask. Specifically, the foreign object detection module is the core discrimination center of the entire device, and its internal logic revolves around the joint criteria of "cross-band mismatch, static image plane, trajectory deviation, and continuous frame continuity." When the matching and prediction module determines that a candidate target exists in the current band but no effective cross-band match is found in other bands, the foreign object detection module continuously tracks the center position drift and contour intersection-over-union ratio of the target between adjacent frames and calculates the deviation between its center position and the predicted motion trajectory position. If this deviation is greater than a dynamic threshold, and the above conditions are met in every frame of M consecutive frames, it is determined to be a foreign object region in the lens. For example, the foreign object detection module can execute the above logical judgment through a digital signal processor (DSP) and generate a corresponding pixel-level binary mask image. In addition, the foreign object recognition module can further distinguish between translucent and opaque foreign objects, providing different confidence levels and repair weights for subsequent repairs, and avoiding misjudging real low-texture areas as lens contamination.

[0033] The repair module is configured to reconstruct the content of the foreign object region in the lens based on the foreign object mask and using the guiding information from the unobstructed band image. It also constrains the spectral response of the reconstructed result in the current band to maintain consistency with the reference features of the unobstructed band, generating a repaired image. Specifically, after receiving the foreign object mask output from the foreign object recognition module and the multi-band image output from the acquisition module, the repair module reconstructs the content of the obstructed region using the guiding information from the unobstructed band image. The repair module internally deploys a lightweight image repair model, which constrains the generation of the repaired image through a total loss function, particularly by using a spectral fidelity loss term to ensure that the spectral response features of the reconstructed result in the current band maintain consistency with the spectral response reference features. For example, the repair module can employ a lightweight encoder-decoder structure, combined with a cross-band attention module, using the texture of the unobstructed band as strong guiding information input to the network. By calculating the weighted sum of the first-norm distance and the second-norm distance, the repair module ensures that the repaired region is not a simple patchwork of ordinary textures, but rather satisfies the consistency between the radiometrically normalized result and the cross-band mapping result within the target band, and that the spectral response remains consistent with the reference features of the unobstructed band. The repaired image can be directly output to downstream task modules, ensuring the continuous execution of high-precision multispectral tasks. Through the coordinated work of the above four modules, the device in this specific embodiment achieves a closed-loop process from foreign object detection to spectral fidelity repair, significantly improving the mission reliability and flight safety of the airborne multispectral optoelectronic system.

[0034] This specific embodiment also provides an electronic device, including a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program to implement the method as described above.

[0035] Specifically, the processor and memory are connected via a system bus. The memory stores the operating system, database, and computer programs that can run on the operating system. When the processor calls the computer programs stored in the memory, it can execute the various steps of the aforementioned multispectral image lens foreign object detection and repair method. For example, when executing the computer program, the processor can acquire multi-band synchronous image frames from cameras with at least partial overlap of multiple fields of view, perform target-level cross-band matching on the multi-band synchronous image frames, and predict the motion trajectory of each real-scene target based on the matching results using a motion trajectory prediction model. This allows it to identify the lens foreign object region and generate a foreign object mask. Finally, it uses the guiding information from the unobstructed band image to reconstruct the content of the lens foreign object region and constrains the spectral response of the reconstruction result in the current band to be consistent with the reference features of the unobstructed band, generating a repaired image. It should be understood that this electronic device is not limited to a specific hardware platform structure; it can be an onboard computing platform for UAVs, an embedded processing unit built into an optoelectronic pod, a ground station control computer, or a cloud-based model training server, or any other hardware entity with data processing capabilities. By embedding the above method into a computer program and having it executed by a processor, the hardware device can be directly applied to real-time image processing tasks in multispectral optoelectronic systems, significantly improving the system's task reliability and operational security.

[0036] This specific embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method described above.

[0037] Specifically, the computer-readable storage medium can be any entity or device capable of carrying computer program code and being read by a processor. For example, a computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any combination thereof. More specific examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any combination thereof. When a computer program is deployed on any of the aforementioned computer-readable storage media and distributed to different hardware carriers, whether it is updated online through a software distribution network or deployed offline through physical media, as long as the processor executes the computer program, the complete foreign object detection and spectral fidelity repair process described in the aforementioned specific embodiment can be realized. By using the computer-readable storage medium as a carrier, both software distribution and hardware deployment scenarios are covered, facilitating evidence collection and rights protection in infringement litigation.

[0038] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting and repairing foreign objects in multispectral imaging lenses, characterized in that, include: Acquire multi-band synchronized image frames from cameras with at least partial overlap of multiple fields of view; Target-level cross-band matching is performed on the multi-band synchronized image frames, and the motion trajectory of each real-scene target is predicted based on the matching results using a motion trajectory prediction model. Identify regions in the multi-band synchronized image frames that do not have cross-band correspondence and whose motion characteristics deviate from the predicted behavior of the target in the real scene, determine the regions as lens foreign object regions and generate foreign object masks; Based on the foreign object mask, the content of the foreign object region of the lens is reconstructed using the guiding information of the unobstructed band image, and the spectral response of the reconstruction result in the current band is constrained to be consistent with the reference features of the unobstructed band, thereby generating a repaired image.

2. The method for detecting and repairing foreign objects in multispectral imaging lenses according to claim 1, characterized in that, The motion trajectory prediction model is a Kalman filter model; The state vector of the Kalman filter model includes the target's pixel coordinates, image plane velocity, and image plane acceleration; The observation vector of the Kalman filter model includes the actual observation coordinates of the same target by cameras in each band; The process noise of the Kalman filter model is dynamically set based on the body angular velocity, linear acceleration and gimbal angular velocity, and the observation noise is dynamically set based on at least one of image sharpness, target detection confidence, exposure status and environmental visibility.

3. The method for detecting and repairing foreign objects in multispectral imaging lenses according to claim 2, characterized in that, The criteria for determining a foreign object area in the lens are the following combined criteria: If a candidate target exists in the current band image and no valid matching target is found in other band images, the valid matching target must satisfy the following conditions: the similarity of the target descriptor is greater than a preset similarity threshold, the spatial registration error is less than a preset error threshold, and the time synchronization tolerance is less than a preset time threshold. The center position drift of the candidate target between adjacent frames is less than a preset drift threshold and the contour intersection-union ratio is greater than a preset intersection-union ratio threshold; The deviation between the position of the candidate target and the predicted position of the motion trajectory is greater than a dynamic threshold, which is determined based on a weighted sum of a base threshold, the uncertainty corresponding to the filtered prediction covariance, and the gimbal angular velocity. In the detection of M consecutive frames, each frame satisfies the criteria that there is a candidate target in the current band image and no valid matching target is found in other band images, and the criteria that the deviation between the position of the candidate target and the predicted position of the motion trajectory is greater than a dynamic threshold; and any two adjacent frames satisfy the criteria that the center position drift of the candidate target between the adjacent frames is less than a preset drift threshold and the contour intersection-union ratio is greater than a preset intersection-union ratio threshold, where M is an integer greater than 1.

4. The method for detecting and repairing foreign objects in multispectral imaging lenses according to claim 1, characterized in that, The method of reconstructing the content of the foreign object region of the lens using guiding information from the unobstructed band image includes: The image of the obscured band, the foreign object mask, the unobscured band guiding image, the time frame data, and the camera calibration parameters are acquired, input into the image restoration model, and the restored image is output. The image inpainting model employs a lightweight encoder-decoder, gated convolution or partial convolution structure, and incorporates a cross-band attention module to use the contours and textures of other unoccluded targets as strong guiding information; the total loss function is expressed as follows: ,in To conceal the reconstruction losses in the area, For edge gradient consistency loss, For timing consistency loss, To preserve losses for downstream target detection or tracking tasks, To preserve spectral fidelity, , , , , These are the weighting coefficients for each loss term; The spectral fidelity loss is adopted ,in For the repair result, For unobstructed true or false true values, Let i be the spectral or radiative response feature extraction function for the i-th band. Let be the radiation normalization function. This is a learning or calibration mapping that maps responses from other unobstructed bands to the response of the i-th band. These are the weighting coefficients.

5. The method for detecting and repairing foreign objects in a multispectral image lens according to claim 1, characterized in that, The method further includes: For cameras with at least partial overlap of multiple fields of view, perform intrinsic parameter calibration and extrinsic parameter joint calibration respectively, establish the mapping relationship between the pixel coordinate system of each camera and the unified coordinate system, and achieve pixel-level spatial registration; Based on a globally unified clock, the timestamps of multiple image frames are aligned through interpolation resampling, and a time synchronization tolerance is set. When the time difference between multiple image frames exceeds the time synchronization tolerance, interpolation compensation is initiated or the similarity threshold for cross-band matching is increased.

6. The method for detecting and repairing foreign objects in a multispectral image lens according to claim 3, characterized in that, The method further includes: The alarm score is calculated based on at least one of the following: the occlusion area of ​​the lens foreign object region, the overlap ratio with the task center area, the task weight of the camera, the number of continuous frames, and the decrease in downstream identification confidence. Alarm levels are classified according to the alarm score, and different alarm levels correspond to different handling strategies. The handling strategies include at least one of the following: ground station recording, audible and visual alarms and PTZ adjustment suggestions, and emergency handling strategies.

7. The method for detecting and repairing foreign objects in a multispectral image lens according to claim 1, characterized in that, The method further includes: During equipment operation, foreign object sample data is saved, and only one of the threshold parameter, matching weight, and normalized statistic is updated; After the equipment stops operating, the foreign object sample data is uploaded to the cloud, and the model is retrained and the threshold is recalibrated through the cloud. After version verification, the updated model and parameters are sent to the device, and the previous version is retained as a rollback version; When consistency checks and version signature verification fail, the detection and repair model on the device side remains unchanged.

8. A multispectral imaging lens foreign object detection and repair system, characterized in that, include: The acquisition module is configured to acquire multi-band synchronous image frames captured by cameras with at least partial overlap of multiple fields of view; The matching and prediction module is configured to perform target-level cross-band matching on the multi-band synchronized image frames, and predict the motion trajectory of each real-scene target based on the matching results using a motion trajectory prediction model. The foreign object recognition module is configured to identify areas in the multi-band synchronous image frame that do not have cross-band correspondence and whose motion characteristics deviate from the predicted behavior of the target in the real scene, determine them as foreign object areas of the lens, and generate foreign object masks. The repair module is configured to reconstruct the content of the foreign object region of the lens based on the foreign object mask and using the guiding information of the unobstructed band image, and to constrain the spectral response of the reconstruction result in the current band to be consistent with the reference features of the unobstructed band, thereby generating a repaired image.

9. An electronic device, characterized in that, Including processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program to implement the multispectral image lens foreign object detection and repair method as described in claim 1.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the multispectral image lens foreign object detection and repair method as described in claim 1.

Images