Mirror contamination identification method and system combining visible light and infrared images

Abstract

The application discloses a mirror surface pollution recognition method and system combining visible light and infrared images. The method comprises the following steps: collecting a dual-mode image of a mirror surface area; using a fine-tuned visual-language model to recognize degradation types such as strong reflection and low contrast, and matching an adaptive enhancement strategy; constructing a global and local feature collaborative fusion network, realizing multi-scale deep interaction of dual-mode features through a semantic driving module and a cross-modal Transformer; inputting the fused features into an anti-interference recognition network, suppressing mirror reflection through a mask generated by dual-mode residual, and enhancing the feature response of small and semi-transparent pollution using cross-modal attention; constructing a joint loss function oriented to recognition accuracy for end-to-end training, and deploying it to an edge platform to realize dynamic power consumption adjustment. The application effectively solves the problem of pollution feature extraction under strong reflection interference of the mirror surface, and improves the recognition accuracy and the endurance of the edge device.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, specifically to a method and system for identifying mirror contamination by combining visible light and infrared images. Background Technology

[0002] With the rapid development of industrial automation and intelligent inspection technologies, the cleanliness detection of mirror surfaces (such as optical lenses, displays, solar panels, and high-precision instrument panels) has become a critical step in ensuring product quality and equipment lifespan. Traditional mirror contamination identification mainly relies on manual visual inspection or single visible light image analysis.

[0003] However, existing technologies have significant limitations:

[0004] Strong mirror reflection interference: The surface of a mirror easily reflects ambient light or the light source of the imaging device, resulting in a large area of ​​high brightness (overexposure) in the visible light image, which masks the real pollution texture and causes missed detection or misjudgment.

[0005] Low-contrast contamination is difficult to identify: For contaminants such as transparent water stains, oil stains, or translucent dust, the visual difference between them and the mirror background in visible light images is minimal, making it difficult for traditional image processing methods to effectively extract features.

[0006] Insufficient information in a single modality: visible light images alone cannot obtain the physical properties of pollutants (such as heat capacity), while infrared images alone lack fine texture details, resulting in a low recognition rate for pollutants in small or special materials.

[0007] Poor adaptability to complex environments: Under different lighting conditions (such as strong light and low illumination), image processing algorithms with fixed parameters have poor robustness and are difficult to meet the needs of all-weather online monitoring.

[0008] Limited computing power for edge deployment: Existing deep learning fusion algorithms are computationally intensive and difficult to deploy directly on resource-constrained edge computing platforms (such as mobile inspection robots), thus failing to achieve real-time, low-power online detection.

[0009] Therefore, there is an urgent need for a method and system for identifying mirror contamination that can suppress mirror reflection, integrate multimodal physical information, adapt to complex environments, and have edge computing capabilities. Summary of the Invention

[0010] This invention aims to address the problems of insufficient single-modal image information, poor adaptability to degradation scenes, insufficient multimodal feature fusion, insufficient anti-interference ability, and difficulty in edge computing deployment in existing mirror contamination recognition technologies. By fusing visible light and infrared images, adaptive enhancement of degradation perception, semantically guided cross-modal feature fusion, and anti-interference recognition mechanisms, it achieves high-precision mirror contamination recognition with a low false negative rate and supports real-time deployment and low-power operation of edge computing platforms.

[0011] In a first aspect, embodiments of the present invention provide a method for identifying specular contamination by combining visible light and infrared images, the method comprising:

[0012] S1: Acquire visible light and infrared thermal images of the same mirror area to reflect the differences in mirror surface texture, reflection information, and thermal radiation of pollutants, respectively.

[0013] S2: Input the acquired bimodal images into the fine-tuned pre-trained vision-language model to identify the degradation type of the mirror scene and match the enhancement strategy to obtain the enhanced visible light image and infrared image;

[0014] S3: Construct a global and local feature collaborative fusion network. Using the enhanced bimodal image as input, the semantic-driven fusion module enhances the features of the contaminated area, and the cross-modal Transformer fusion module realizes multi-scale deep interaction of bimodal features to generate a fused feature map.

[0015] S4: Input the fused feature map into the anti-interference recognition network, generate an adaptive mask through the clutter suppression submodule to suppress specular reflection and specular noise, enhance the feature response of micro-contamination and semi-transparent contamination through the cross-modal attention submodule, and output the contamination area mask and category information through the detection head;

[0016] S5: Construct a joint optimization loss function, guided by optimizing the accuracy of pollution identification, and perform end-to-end training on the fusion network and the identification network;

[0017] S6: Deploy the trained fusion recognition model to the edge computing platform, preprocess, fuse and recognize the real-time acquired dual-modal images in sequence, and output a fused image with pollution area annotations.

[0018] In a second aspect, embodiments of the present invention provide a specular contamination identification system combining visible light and infrared images, applied to the specular contamination identification method combining visible light and infrared images as described in the first aspect, the system comprising:

[0019] The image acquisition module is used to acquire visible light images and infrared thermal images of the same mirror area. The visible light images reflect the surface texture and reflection information of the mirror, and the infrared thermal images reflect the differences in thermal radiation of pollutants.

[0020] The degradation perception preprocessing module is used to input the acquired bimodal images into the fine-tuned pre-trained vision-language model, identify the degradation type of the mirror scene and match the enhancement strategy to obtain the enhanced visible light image and the enhanced infrared image.

[0021] The feature fusion module is used to construct a global and local feature collaborative fusion network. Taking the enhanced bimodal image as input, the semantic-driven fusion module enhances the features of the polluted area, and the cross-modal Transformer fusion module realizes multi-scale deep interaction of bimodal features to generate a fused feature map.

[0022] The contamination identification module is used to input the fused feature map into the anti-interference identification network, generate an adaptive mask through the clutter suppression submodule to suppress specular reflection and specular noise, enhance the feature response of micro-contamination and semi-transparent contamination through the cross-modal attention submodule, and output the contamination area mask and category information through the detection head.

[0023] The joint training module is used to construct a joint optimization loss function, with the aim of optimizing the accuracy of contamination identification, and to perform end-to-end training on the fusion network in the feature fusion module and the identification network in the contamination identification module;

[0024] The deployment inference module is used to deploy the trained fusion recognition model to the edge computing platform, and to perform preprocessing, fusion and recognition processing on the real-time acquired dual-modal images in sequence, and output a fused image with pollution area annotations.

[0025] Thirdly, embodiments of the present invention provide an electronic device, comprising:

[0026] processor;

[0027] Memory used to store processor-executable instructions;

[0028] The processor is configured to implement the mirror contamination identification method combining visible light and infrared images as described in the first aspect when executing the instructions.

[0029] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a program that instructs a device to perform the mirror contamination identification method combining visible light and infrared images as described in the first aspect.

[0030] Compared with existing technologies, this invention has the following advantages: It identifies strong reflective degradation types through a visual-language model and generates adaptive masks to specifically attenuate reflective areas, solving the problem of high-brightness interference masking contamination features. By leveraging the complementary thermal radiation differences in infrared images and the texture information of visible light, combined with cross-modal Transformer deep interaction, it significantly improves the detection rate of difficult-to-identify contaminants such as transparent water stains and oil stains. The adaptive enhancement strategy based on degradation perception ensures stable performance under different lighting and contamination conditions. Through a dynamic power consumption adjustment mechanism, it significantly reduces computational load while maintaining recall, adapting to the long-term battery life requirements of mobile inspection robots and online monitoring systems. The loss function design oriented towards recognition accuracy allows the fusion network to directly serve the final detection task, avoiding error accumulation in traditional step-by-step processing. Attached Figure Description

[0031] Figure 1 This is a schematic flowchart of a mirror contamination identification method combining visible light and infrared images, provided in an embodiment of the present invention.

[0032] Figure 2 This is an architecture diagram of a mirror contamination identification system combining visible light and infrared images, provided as an embodiment of the present invention.

[0033] Figure 3 This is a schematic diagram of an electronic device provided according to an embodiment of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0035] It should be noted that in the embodiments of the present invention, "at least one" refers to one or more, and "more than one" refers to two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0036] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0037] Example 1

[0038] Figure 1 This is a schematic flowchart illustrating a method for identifying mirror contamination by combining visible light and infrared images, provided in an embodiment of the present invention. Figure 1As shown, a method for identifying specular contamination by combining visible light and infrared images includes:

[0039] S1: Acquire visible light and infrared thermal images of the same mirror area to reflect the surface texture, reflection information, and thermal radiation differences of contaminants, respectively. This step uses a co-source imaging device to simultaneously acquire visible light color images and infrared thermal grayscale images of the mirror area, capturing the texture details of the mirror surface and the thermal radiation differences between contaminants and the mirror substrate, providing a basic data source for subsequent processing.

[0040] Specifically, in this embodiment, step S1 includes:

[0041] S1.1: The visible light image and the infrared thermal imaging image are acquired synchronously using a dual-spectrum imaging device to ensure strict alignment in spatial geometry. In this embodiment, a dual-spectrum imaging device equipped with both visible light and infrared lenses is used to image the mirror area. The two lenses of this device undergo hardware-level optical axis calibration, ensuring that the output visible light image and the infrared thermal imaging image are strictly aligned in spatial geometry (i.e., pixel-level registration), eliminating the need for subsequent affine transformation correction. This guarantees a one-to-one correspondence between the two modal data in spatial location, providing a precise geometric basis for subsequent feature fusion.

[0042] S1.2: The visible light image contains RGB three-channel color information, used to characterize the appearance texture of the mirror surface, the visual morphology of reflected objects, and contaminants. The acquired visible light image contains standard RGB three-channel color information, which can clearly reflect the physical properties of the mirror surface. Specifically, it records the appearance texture of the mirror, the reflected images of environmental objects (such as the reflection of lights or windows), and the visual morphological characteristics of contaminants (such as fingerprints, dust, or oil stains). These high-resolution texture details are the key basis for distinguishing the mirror substrate from surface deposits.

[0043] S1.3: The infrared thermal imaging image is a single-channel grayscale data. By recording the difference in thermal radiation intensity between the mirror surface and the pollutants, it characterizes the outline information and temperature distribution characteristics of the pollutants, and can still maintain stable imaging capability under the strong reflective environment of the mirror.

[0044] Simultaneously acquired infrared thermal images are single-channel grayscale data, and their imaging principle is based on the object's own thermal radiation rather than ambient light reflection. Because contaminants (such as those containing water stains or oil) and clean mirror substrates have physical differences in specific heat capacity and thermal conductivity, they exhibit different grayscale values ​​in the infrared image. This difference in thermal radiation intensity allows the infrared image to stably delineate the outline and temperature distribution characteristics of contaminants even under conditions of strong mirror reflection or extremely low ambient light.

[0045] S2: The acquired bimodal images are input into a finely tuned pre-trained visual-language model to identify the degradation type of the mirror scene and match enhancement strategies, resulting in enhanced visible light and infrared images. This step utilizes the finely tuned visual-language model to intelligently identify the degradation type of the current scene (such as strong reflection or low contrast) and automatically matches the corresponding image processing algorithm based on the prior knowledge base to perform targeted denoising and contrast enhancement on the original bimodal images.

[0046] In this embodiment, the visual-language model is constructed and initialized based on the CLIP (Contrastive Language-ImagePre-training) architecture. First, an image-text pair dataset containing tens of thousands of mirror surface samples is collected, with text descriptions covering various degradation types (such as strong reflections, oil stains, fingerprints, etc.). Then, fine-tuning of downstream tasks is performed on the pre-trained base model. Contrastive learning is used to adjust the model parameters, enabling it to establish strong associations between image features and text labels in a high-dimensional space. After training, the model can not only recognize regular objects but also accurately distinguish the unique physical degradation phenomena of mirror surfaces, thus possessing the ability to perform semantic understanding of scene degradation types in input visible light images.

[0047] Specifically, in this embodiment, identifying the degradation type of the mirror scene in step S2 includes:

[0048] S2.1. Use a pre-trained visual-language model finely tuned on a mirror contamination dataset to perform semantic understanding and analysis on the input visible light images.

[0049] In this embodiment, the visible light image acquired in step S1 is first input into a pre-trained visual-language model (VLM). This model has been fine-tuned on a dataset containing various types of mirror contamination samples, enabling it to understand the semantics of mirror scenes. The model can analyze the light and shadow distribution, texture structure, and abnormal regions in the image to determine whether there are imaging quality problems in the current image.

[0050] S2.2 The visual-language model outputs a degradation type label for the current scene. The degradation type label includes at least one or more combinations of: strong reflective interference, low contrast, transparent / semi-transparent contamination, minor stains, and background clutter occlusion.

[0051] After semantic analysis, the model outputs specific degradation type labels. These labels are used to quantify the imaging state of the current mirror scene, mainly including: strong reflective interference (such as overexposure caused by direct light), low contrast (small color difference between background and pollution), transparent / semi-transparent pollution (such as water stains or oil film), small stains (such as dust particles), and background clutter occlusion (such as reflection of non-pollution objects). The labels can be a single type or a combination of multiple types.

[0052] Specifically, in this embodiment, the matching enhancement strategy in step S2 includes:

[0053] S2.3 Construct a prior knowledge base containing mapping relationships between different degradation types and image processing algorithms. The system has a built-in prior knowledge base, which is a pre-configured mapping table. This database establishes logical connections between the different degradation type labels identified in step S2.2 and specific image processing algorithms, providing a decision-making basis for subsequently selecting the optimal image restoration strategy.

[0054] S2.4. Based on the identified degradation type labels, the system retrieves and invokes the corresponding enhancement strategy from the prior knowledge base. The system automatically performs a retrieval operation in the prior knowledge base based on the degradation type labels output by the model. Once a matching label is found, the system invokes the associated image processing algorithm combination to achieve customized enhancement schemes for different degradation scenarios, rather than using a general, single processing method.

[0055] S2.5 When strong reflective interference is detected, a highlight suppression and reflection removal algorithm is executed on the visible light image. When the system determines that strong reflective interference exists in the current scene, the highlight suppression and reflection removal algorithm is automatically triggered. This algorithm analyzes the pixel brightness distribution, identifies and darkens overexposed areas (such as the bright areas without thermal radiation response mentioned in S1.3), thereby restoring the mirror substrate or contamination details that were obscured by reflections and reducing the false alarm rate.

[0056] S2.6 When the image is determined to be low-contrast or has small stains, adaptive histogram equalization or sharpening filtering is performed on the visible light image. For low-contrast or small stain scenes, the system performs adaptive histogram equalization or sharpening filtering on the visible light image. The former is used to stretch the grayscale range of the image, increasing the brightness difference between the background and the contaminant; the latter is used to enhance the edge gradient of the small stains, making them more visually prominent.

[0057] S2.7 Simultaneously, contrast-limited adaptive histogram equalization (CLAHE) is uniformly performed on the infrared thermal imaging images to improve the clarity of the thermal radiation contours in contaminated areas. Regardless of the degradation type of the visible light image, the system uniformly performs contrast-limited adaptive histogram equalization (CLAHE) on the infrared thermal imaging images. Since infrared images mainly rely on thermal radiation difference imaging, CLAHE processing can effectively improve the clarity of the thermal radiation contours between contaminated areas and clean mirror surfaces, ensuring that the infrared mode can still provide effective detection clues when visible light fails.

[0058] To achieve dynamic adaptive adjustment of enhancement parameters under different degradation types, this embodiment further introduces a degradation-aware adaptive enhancement function, which drives the nonlinear mapping of enhancement intensity through the degradation type label output by the visual-language model, as shown in formulas (1)-(4).

[0059] (1)

[0060] in, This is a degradation type label vector, identifying degradation categories such as strong reflection, low contrast, and minor stains. The total number of degradation categories, For the fine-tuned pre-trained vision-language model, The input is a visible light image.

[0061] (2)

[0062] in, To adaptively enhance the strength coefficient, the intensity of the enhancement algorithm is controlled. For the first The learnable weight parameters corresponding to class degradation are obtained through network training. For indicator functions, It is the Sigmoid activation function. This is a bias term used to adjust the baseline value for the enhancement strength.

[0063] In this embodiment, K=5 corresponds to five degradation types and learnable parameters. and During training, the network parameters are optimized together; all parameters are initialized at this time. =0.5、 =0.0; Degradation type output according to formula (1) during forward propagation. Constructing indicator vectors ,when = (Take 1 if the condition is met, otherwise take 0) Calculate the weighted sum. Through the Sigmoid function Mapping to the [0,1] interval yields On the NVIDIA Jetson edge computing platform, this computation is performed through a single matrix multiplication and an activation function. γ is updated via backpropagation, and the loss function is the subsequent joint optimization loss. This allows the network to automatically learn the optimal enhancement strength corresponding to different degradation types.

[0064] (3)

[0065] in, For pixel space coordinates, To enhance the visible light image at position pixel values, This is a contrast-limited adaptive histogram equalization operator.

[0066] First, the original visible light image... Perform CLAHE processing with parameters set to 8×8 pixels for image block size, contrast limiting threshold of 2.0, and 256 gray levels. The specific process involves dividing the image into M×N non-overlapping blocks, calculating the histogram for each block, cropping the histogram according to the limiting threshold, and evenly distributing any excess histogram across all gray levels. After histogram equalization of each block, bilinear interpolation is used to smooth the boundaries between blocks to avoid block artifacts. The output is denoted as... Then, the result is calculated according to formula (2). Perform linear interpolation ,when When =0, the original image is completely preserved; when... When the result is equal to 1, the CLAHE enhancement is used to achieve a smooth transition for intermediate values; the weighted summation is implemented pixel by pixel in the code.

[0067] (4)

[0068] in, To enhance the infrared image at the position pixel values, To enhance the fusion coefficient of infrared images and balance the ratio of original information to enhanced information, These are the pixel values ​​of the original infrared image.

[0069] First, the original infrared image... Perform CLAHE processing with the same parameters as formula (3), namely, block size 8×8 and limiting threshold 2.0. The output is denoted as... Then set the fusion coefficient. =0.7 (this value was determined through a grid search of the validation set, achieving the highest average recognition accuracy across the five degradation scenarios), and weighted fusion was performed. =0.7· +0.3· Experiments show At a signal-to-noise ratio of 0.7, the signal-to-noise ratio is improved most while maintaining the physical authenticity of the original image; infrared enhancement intensity The visible light enhancement intensity is fixed at 0.7. Dynamically adjustable because visible light is greatly affected by degradation and requires adaptive adjustment, while infrared images rely on thermal radiation imaging and are less affected by ambient light, so a stable effect can be obtained by using a fixed enhancement coefficient.

[0070] S3: Construct a global and local feature collaborative fusion network. Taking the enhanced bimodal image as input, a semantically driven fusion module enhances the features of contaminated areas, and a cross-modal Transformer fusion module achieves multi-scale deep interaction of bimodal features to generate a fused feature map. This step constructs a deep learning network that, through semantic guidance and a cross-modal Transformer architecture, deeply mines and fuses complementary information from visible light and infrared images to generate a fused feature map with rich texture details and significant thermal radiation features.

[0071] In this embodiment, the global and local feature collaborative fusion network adopts an encoder-decoder architecture, with its input receiving the enhanced bimodal image output from step S2. On the encoder side, the network constructs two branches in parallel: one is a global feature extraction branch, which uses a deep convolutional neural network (CNN) or a Transformer encoder to capture the broad-spectrum contextual information and macroscopic semantic layout of the image; the other is a local feature extraction branch, which focuses on extracting high-frequency texture details and edge gradient information through shallow convolutional kernels. The outputs of these two branches are then fed into the semantic-driven fusion module and the cross-modal Transformer fusion module described in step S3. Through high-level semantic guidance and multi-scale feature interaction, global semantic consistency and local detail sharpness are organically combined, and finally, the decoder reconstructs and generates a fused feature map that takes into account both thermal radiation contours and texture details.

[0072] Specifically, in this embodiment, the specific processing procedure of the semantic-driven fusion module in step S3 includes:

[0073] S3.1 Extracting feature maps from the intermediate layers of the pre-trained segmentation network as high-level semantic guidance maps. In this embodiment, the system loads an image segmentation network (such as DeepLabv3+) pre-trained on a large-scale dataset. This network is used as a feature extractor, and the output feature maps of its intermediate layers are extracted and defined as high-level semantic guidance maps. These feature maps have filtered out low-level noise, preserving high-level semantic information about object categories and general outlines, effectively indicating regions in the image where objects may exist.

[0074] S3.2 Calculate the similarity weights between the high-level semantic guidance map and the visible light feature map and infrared feature map. Using the semantic guidance map obtained in step S3.1, calculate the similarity weights between it and the current visible light feature map and infrared feature map, respectively. This process essentially measures the degree of matching between each position in the feature map and the known "contamination" semantic pattern. The higher the similarity, the more likely the region is to contain real contamination information, thus gaining higher attention in subsequent steps.

[0075] S3.3. The dual-modal features are weighted and fused according to the similarity weights, causing the network to focus on regions containing contaminating semantic information and suppress responses from mirrored backgrounds and reflective areas. Based on the similarity weights calculated in step S3.2, the visible light and infrared feature maps are weighted and fused. The network automatically enhances feature responses highly correlated with the high-level semantic guidance map (i.e., suspected contamination areas) while suppressing semantically irrelevant regions (such as clean mirrored backgrounds or reflective reflections), thereby achieving precise feature-level filtering.

[0076] Specifically, in this embodiment, the specific processing procedure of the cross-modal Transformer fusion module in step S3 includes:

[0077] S3.4. Divide the enhanced visible light image features and infrared image features into sequences of local image blocks of the same size. Before entering the Transformer architecture, the enhanced visible light and infrared image feature maps are first cut into several local image blocks, and these image blocks are linearly unfolded into a sequence of one-dimensional vectors. This serialization process breaks the spatial continuity constraint of the image, allowing the model to process image features like natural language sequences, facilitating the establishment of long-distance dependencies.

[0078] S3.5. Capture global dependencies within a single modality using a multi-head self-attention mechanism. Perform the multi-head self-attention mechanism on both the visible light and infrared sequences. This step aims to capture long-range dependencies within a single modality, i.e., to understand the texture coherence within the visible light image or the temperature distribution pattern within the infrared image, thereby extracting more representative single-modal feature representations.

[0079] S3.6. The cross-attention mechanism is used to calculate the correlation weights between the visible light feature sequence and the infrared feature sequence, achieving deep interaction and information complementarity of the dual-modal features in multi-scale space, and outputting a fused feature map. After completing self-attention encoding, the cross-attention mechanism is used to calculate the correlation weights between the visible light feature sequence and the infrared feature sequence. This allows the model to use texture details in the visible light image to supplement blurred contours in the infrared image, or to use thermal radiation differences in the infrared image to verify suspected contamination areas in the visible light, ultimately achieving deep information complementarity and outputting a fused feature map.

[0080] To quantify the degree of deep interaction between bimodal features in multi-scale space, this embodiment defines the cross-attention weight calculation method in the cross-modal Transformer fusion module, as shown in formulas (5)-(8).

[0081] , , , (5)

[0082] in, This is a query matrix for visible light modes. The bond matrix for infrared modes. This is the value matrix for the infrared modes; and These are feature maps of the enhanced visible light and infrared images, respectively. , , It is a learnable linear projection matrix.

[0083] Enhanced visible light feature map and infrared feature map The dimensions are all C=64, H=128, W=128, and three independent 1×1 convolutional layers are defined. , , (Input channel 64, Output channel 64) =32, convolution kernel 1×1), during forward propagation through , , Generate query, key, and value matrices, all with an output size of [B, 32, 128, 128]; then flatten the spatial dimensions into a sequence form using Q= The `.view(B,C,H×W).permute(0,2,1)` method generates a tensor of [B,16384,32]. The same process applies to this tensor. and This facilitates subsequent attention calculations.

[0084] , (6)

[0085] in This is a cross-modal cross-attention feature map. Let be the dimension of the key vector. For normalized exponential functions, This is the matrix transpose operation.

[0086] First, flatten the... and Matrix multiplication is performed to obtain the attention score matrix; then Softmax normalization is performed along the key dimension (the last dimension) to obtain the attention weights; then multiplied with the value matrix; finally, the spatial dimension is restored to output the size [B,32,128,128]; to reduce memory usage, this embodiment adopts a window attention mechanism (window size is 16×16), which restricts the attention calculation to a local window, reducing memory usage from about 4GB to about 200MB, while maintaining the global receptive field through feature transfer between windows.

[0087] , (7)

[0088] in, To fuse feature maps (single scale). For layer normalization operation, It is a multilayer perceptron.

[0089] First, perform a residual join, and... The channel size is adjusted to 32 channels via 1×1 convolution. Then, the gradient and activation values ​​are stabilized during training through layer normalization. Next, a nonlinear transformation is performed using two MLP layers (the first layer is Linear 32→128 with GELU activation, and the second layer is Linear 128→32). Finally, a second residual connection is executed to output the size [B, 32, 128, 128]. In this embodiment, four identical Transformer blocks are stacked, with each block having its own parameters. The input of the second block and subsequent blocks is the output of the previous block. Visible light characteristics Residual joins are only involved in the first block.

[0090] , (8)

[0091] in, For multi-scale fused feature maps, This is a downsampling operation with step sizes of 1, 2, and 3. This refers to the splicing operation at the channel dimension.

[0092] Define three parallel downsampling branches. Branch 1 (s=1) directly takes... Without downsampling, branch 2 (s=2) uses average pooling (kernel size 2×2, stride 2) to output size [32, 64, 64], and branch 3 (s=4) uses average pooling (kernel size 4×4, stride 4) to output size [32, 32, 32]. Then, branches 2 and 3 are upsampled back to [32, 128, 128] using bilinear interpolation. Finally, they are concatenated along the channel dimension. Output size [B, 96, 128, 128]; features at different scales contain information of different granularities: the original scale (s=1) preserves fine texture for detecting tiny stains, s=2 captures medium-range structural information, and s=4 provides global semantic context for perceiving large-area contamination.

[0093] S4: The fused feature map is input into the anti-interference recognition network. An adaptive mask is generated through the clutter suppression submodule to suppress specular reflections and specular noise. A cross-modal attention submodule enhances the feature response to minute and semi-transparent contaminants. The detection head outputs the contamination area mask and category information. This step utilizes the fused feature map for contamination detection, suppresses clutter interference such as specular reflections through adaptive masking, and enhances the response to minute and semi-transparent contaminant features, ultimately outputting accurate contamination area location and category.

[0094] In this embodiment, the anti-interference identification network employs an improved single-stage target detector based on the YOLO architecture, whose input directly receives the fused feature map output in step S3. The network backbone embeds the clutter suppression submodule and the cross-modal attention submodule described in step S4: the former uses a mask generated from dual-modal residuals to weight the feature map, physically blocking signal propagation in the specular reflection region; the latter recalibrates the feature response in the channel dimension and enhances the saliency of small and semi-transparent contamination regions in the spatial dimension through a cross-modal attention mechanism. Finally, the network classifies and regresses the processed features through a multi-layer detection head, outputting a contamination region mask and category information with confidence levels.

[0095] Specifically, in this embodiment, the clutter suppression submodule generating the adaptive mask in step S4 specifically includes:

[0096] S4.1 Calculate the pixel-level difference between the enhanced visible light image output by formula (3) and the enhanced infrared image output by formula (4) to obtain a bimodal residual map. In this embodiment, the clutter suppression submodule first calculates the pixel-level difference between the enhanced bimodal images output by formulas (3) and (4). Since the visible light image contains ambient reflected light while the infrared image only reflects thermal radiation, the difference between the two is extracted to form a bimodal residual map. In this residual map, real thermal pollution sources usually have small differences, while false highlights that only exist in visible light (such as light reflections) will show large numerical differences.

[0097] S4.2. Threshold segmentation and morphological processing are performed on the dual-modal residual map to generate an adaptive mask. Specular reflection and specular noise are represented as abnormally bright areas with no thermal radiation response in the dual-modal residual map. The adaptive mask is used to weight and attenuate the fused feature map to reduce the impact of reflective interference on subsequent contamination identification.

[0098] Subsequently, the system performs threshold segmentation and morphological processing on the dual-modal residual map and generates an adaptive mask according to formula (10). This mask can accurately locate these interference regions and perform weighted attenuation on the fused feature map output in step S3, effectively reducing the misleading effect of reflective clutter on subsequent pollution identification.

[0099] To suppress the interference of specular reflection and high-light noise on contamination identification, this embodiment defines a dual-modal residual-driven adaptive mask generation mechanism, as shown in formulas (9)-(10).

[0100] , (9)

[0101] in, For the dual-modal residual map at the pixel The grayscale value at that location reflects the pixel-level difference between visible light and infrared images. It is an absolute value operator.

[0102] First, the enhanced visible light image Convert to a single-channel grayscale image; then calculate pixel-by-pixel comparison with the infrared image. absolute value of the difference The output dimensions are [H, W]. Physically, specular reflective areas exhibit high brightness in visible light images but lack corresponding thermal radiation responses in infrared images. Therefore, the values ​​of reflective areas in the residual image are significantly higher than those of normal areas. Meanwhile, real contamination areas respond in both modes (visible light texture changes plus infrared thermal radiation differences), resulting in smaller residual values. This characteristic allows the residual image to effectively distinguish between reflective interference and real contamination. Finally, Linear mapping to the [0,255] interval facilitates subsequent threshold processing.

[0103] , (10)

[0104] in, To adaptively suppress the mask at the pixel The weight values ​​at each point are used to perform weighted attenuation on the fused feature map. and Residual plots The mean and standard deviation, The adjustable sensitivity coefficient controls the sharpness of the mask. This is the Sigmoid function.

[0105] First, calculate the residual plot. global mean and standard deviation and Then perform standardization; then multiply by the sensitivity coefficient. (Obtained through validation set optimization); finally, the Sigmoid function is used to map it to the [0,1] interval. Morphological closing operations (dilation followed by erosion, kernel size 3×3) are performed on the mask to eliminate isolated noise and fill holes; the mask is applied to the fused feature map to achieve channel-wise weighted attenuation; when there is no strong reflection in the scene, the mask tends to be around 0.5, which has limited attenuation of the fused features; when there is strong reflection, the reflective area tends to be close to 1 (high attenuation) after Sigmoid mapping, thus achieving accurate adaptive suppression.

[0106] S5: Construct a joint optimization loss function to optimize contamination recognition accuracy, and perform end-to-end training on the fusion network and the recognition network. This step designs a composite loss function oriented towards detection performance, simultaneously constraining image fusion quality and contamination recognition accuracy during training, achieving parameter sharing and collaborative optimization between the fusion network and the detection network.

[0107] Specifically, in this embodiment, step S5, constructing the joint optimization loss function, specifically includes:

[0108] S5.1 Set the pollution identification loss to have a dominant weight in the joint optimization loss function, so as to force the learning objective of the fusion network to conform to the final identification accuracy.

[0109] In this embodiment, the constructed joint optimization loss function consists of a weighted average of multiple sub-losses. The contamination identification loss (typically including classification loss and bounding box regression loss) is set as the dominant weight. This design forces the fusion network to prioritize the accuracy of downstream detection tasks during the learning process, rather than simply pursuing the generation of visually appealing fused images, thereby ensuring that the fused feature maps have practical value for the identification task.

[0110] S5.2 Introducing a contamination confidence feedback mechanism: When the contamination confidence output by the detection head is lower than a preset threshold, the auxiliary decomposition loss weight of the corresponding region is increased, forcing the fusion network to enhance the feature representation of that region in subsequent iterations. The system introduces a contamination confidence feedback mechanism to dynamically adjust the training process. In each iteration, if the contamination confidence output by the detection head is lower than a preset threshold (indicating that the region is difficult to identify), the system automatically increases the auxiliary decomposition loss weight corresponding to that region. This mechanism forces the fusion network to focus on these difficult samples in subsequent backpropagation, enhancing its feature representation ability in that region, thereby improving the model's sensitivity to difficult-to-identify contamination.

[0111] S5.3 By combining pixel loss and gradient loss, we ensure that the fused feature map improves recognition performance while preserving the edge structure and texture details of the contaminated area, thus avoiding feature distortion caused by overfitting.

[0112] To prevent the network from compromising image structure to meet detection requirements, the loss function incorporates both pixel loss and gradient loss. Pixel loss constrains the brightness consistency between the fused image and the source image, while gradient loss preserves edge structure and texture details in contaminated areas. This combined constraint effectively avoids feature distortion caused by overfitting to the detection target, ensuring the physical authenticity of the fused image.

[0113] To achieve end-to-end joint optimization guided by pollution identification accuracy, this embodiment defines a joint optimization loss function, as shown in formulas (11)-(13).

[0114] , (11)

[0115] in, To mitigate the loss in contamination identification, the fusion network is driven to conform to the detection accuracy target. For classification loss (such as cross-entropy). The bounding box regression loss (such as CIoU) is used. and These are the corresponding balance weight coefficients.

[0116] , (12)

[0117] in, To aid in loss decomposition, it is used to enhance feature representation in low-confidence regions. For confidence levels below a preset threshold pixel area, For pixels The feedback weight at each point increases as the confidence level decreases. To fuse feature maps at the pixel level The feature vector at that location, To guide feature maps at the pixel level The feature vector at that location, It is the Euclidean norm.

[0118] , (13)

[0119] in, To jointly optimize the total loss, To account for pixel loss, constrain the pixel consistency between the fused image and the source image. For gradient loss, the edge structure and texture details of the contaminated areas are preserved. These are the balancing weight coefficients for each loss term.

[0120] S6: Deploy the trained integrated fusion and recognition model to the edge computing platform. Perform preprocessing, fusion, and recognition on the real-time acquired dual-modal images sequentially, outputting a fused image with contaminated area annotations. This step integrates the trained lightweight model into the edge computing device, enabling full-process processing of image preprocessing, feature fusion, and contamination recognition for real-time video streams, and outputting visualized detection results.

[0121] In this embodiment, the trained fusion recognition model is first quantized (INT8) and channel pruning and compression is performed to adapt to the storage and computing power limitations of the edge computing platform (such as NVIDIA Jetson series or Rockchip RK3588 NPU). Then, the model is loaded through an inference engine (such as TensorRT or ONNX Runtime). During runtime, the edge platform receives real-time visible light and infrared video streams from shipborne or shore-based dual-spectrum cameras. It sequentially executes step S2 (degradation perception preprocessing), step S3 (global and local feature collaborative fusion), and step S4 (anti-interference pollution recognition) frame by frame. Finally, the detected pollution area mask is overlaid and rendered on the fused image, and the fused image with pollution area annotations is output. Alarm signals or detection coordinates can be uploaded to the host computer monitoring system simultaneously.

[0122] Specifically, in this embodiment, the edge computing platform is configured to support dynamic power consumption adjustment mode in step S6, which specifically includes:

[0123] S6.1 Monitor the average confidence level of pollution identification and ambient light intensity of the current processing frame to determine the complexity level of the current mirror scene.

[0124] In this embodiment, the edge computing platform has a built-in monitoring module for real-time collection and analysis of data metrics of the current processing frame. This module mainly monitors two dimensions: first, the average confidence score for contamination identification, which reflects the difficulty of identification in the current scene; and second, the ambient light intensity, which determines whether the scene is under extreme lighting conditions. Based on these two metrics, the system comprehensively evaluates and classifies the complexity level of the current mirror scene (e.g., high complexity or low complexity).

[0125] S6.2 When a low-complexity scene or a highly reflective environment is detected, some multi-head self-attention computation layers in the cross-modal Transformer fusion module are turned off or skipped, and the output of the semantic-driven fusion module is directly used as the recognition input. Based on the complexity level determined in step S6.1, the system performs dynamic computing power scheduling. When a low-complexity scene (such as a large clean area) or a highly reflective environment (where Transformer gain is limited) is detected, the system will automatically turn off or skip some multi-head self-attention computation layers in the cross-modal Transformer fusion module, and instead directly use the output of the preceding semantic-driven fusion module as the recognition input, thereby shortening the computation chain without significantly affecting accuracy.

[0126] S6.3. Under the premise of ensuring that the pollution identification recall rate meets the preset safety threshold, the power consumption and inference latency of the edge computing platform are reduced by dynamically reducing the network computing depth, so as to meet the long-term battery life requirements of mobile inspection equipment or online monitoring systems.

[0127] While ensuring that the pollution identification recall rate always meets the preset safety threshold (e.g., 95%), the aforementioned strategy of dynamically reducing network computing depth significantly reduces the power consumption and inference latency of the edge computing platform. This dynamic power consumption adjustment mode enables the device to flexibly allocate computing power according to the actual needs of the scenario, greatly extending the continuous working time and battery life of mobile inspection equipment or online monitoring systems.

[0128] Example 2

[0129] like Figure 2 As shown, this invention provides an architecture diagram of a mirror contamination recognition system combining visible light and infrared images, applied to the mirror contamination recognition method combining visible light and infrared images as described in Embodiment 1, comprising:

[0130] Image acquisition module 210 is used to acquire visible light images and infrared thermal images of the same mirror area. The visible light images reflect the surface texture and reflection information of the mirror, and the infrared thermal images reflect the differences in thermal radiation of pollutants.

[0131] The degradation perception preprocessing module 220 is used to input the acquired bimodal images into the fine-tuned pre-trained vision-language model, identify the degradation type of the mirror scene and match the enhancement strategy to obtain the enhanced visible light image and the enhanced infrared image.

[0132] The feature fusion module 230 is used to construct a global and local feature collaborative fusion network. Taking the enhanced bimodal image as input, it uses the semantic-driven fusion module to enhance the features of the polluted area and uses the cross-modal Transformer fusion module to realize multi-scale deep interaction of bimodal features to generate a fused feature map.

[0133] The contamination identification module 240 is used to input the fused feature map into the anti-interference identification network, generate an adaptive mask through the clutter suppression submodule to suppress specular reflection and specular noise, enhance the feature response of micro-contamination and semi-transparent contamination through the cross-modal attention submodule, and output the contamination area mask and category information through the detection head.

[0134] The joint training module 250 is used to construct a joint optimization loss function, with the aim of optimizing the accuracy of pollution identification, and to perform end-to-end training on the fusion network in the feature fusion module and the identification network in the pollution identification module.

[0135] The deployment inference module 260 is used to deploy the trained fusion recognition integrated model to the edge computing platform, and to perform preprocessing, fusion and recognition processing on the real-time acquired dual-modal images in sequence, and output a fusion image with pollution area annotations.

[0136] Figure 3 This is an electronic device provided in one embodiment of the present invention. For example... Figure 3 As shown, the electronic device includes at least the following components: processor 301 and memory 300, communication interface 303, and bus 302.

[0137] In this embodiment of the invention, the memory 300 is used to store executable instructions of the processor 301, which is configured to implement the method as described in the first aspect when executing the instructions.

[0138] In an embodiment of the present invention, a computer-readable storage medium includes instructions that instruct a device to perform the method as described in the first aspect. For example, the instructions instruct the device to perform... Figure 1 The method is shown in the process steps.

[0139] In one embodiment of the present invention, the program operating in the electronic device may be a program that controls a central processing unit (CPU) or similar device to achieve the functions of the above-described embodiments of the present invention (a program that enables the computer to function). Information processed by these systems is then temporarily stored in random access memory (RAM) during processing, and subsequently stored in various ROMs such as read-only memory (FlashROM) and hard disk drives (HDDs), and read, corrected, and written by the CPU as needed.

[0140] It should be noted that a portion of the electronic device described above can also be implemented using a computer. In this case, the program for implementing the control function can be recorded on a computer-readable recording medium, and the program recorded on the recording medium can be read into the computer and executed.

[0141] It should be noted that the computer mentioned here refers to a computer built into an electronic device, employing hardware including an operating system and peripheral devices. Furthermore, computer-readable recording media refers to removable media such as floppy disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage systems such as hard drives built into the computer.

[0142] Furthermore, computer-readable recording media can include: media that dynamically stores programs for short periods of time, such as communication lines used when transmitting programs via networks like the Internet or communication lines like telephone lines; and media that store programs for fixed periods of time, such as volatile memory inside a computer that serves as a server or client in this case. In addition, the aforementioned program can be a program used to implement the above-mentioned functions, or it can be a program that can implement the above-mentioned functions by combining them with programs already recorded in the computer.

[0143] Furthermore, the electronic device in the above embodiments can also be implemented as an assembly (system group) composed of multiple systems. Each system constituting the system group can possess some or all of the functions or functional blocks of the electronic device in the above embodiments. As a system group, it is sufficient to have all the functions or functional blocks of the electronic device.

[0144] Those skilled in the art should recognize that the above embodiments are merely illustrative of the present invention and are not intended to limit the present invention. Any appropriate changes and variations made to the above embodiments within the essential spirit and scope of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A method for identifying specular contamination by combining visible light and infrared images, characterized in that, Includes the following steps: S1: Acquire visible light and infrared thermal images of the same mirror area to reflect the differences in mirror surface texture, reflection information, and thermal radiation of pollutants, respectively. S2: Input the acquired bimodal images into the fine-tuned pre-trained vision-language model to identify the degradation type of the mirror scene and match the enhancement strategy to obtain the enhanced visible light image and infrared image; S3: Construct a global and local feature collaborative fusion network. Using the enhanced bimodal image as input, the semantic-driven fusion module enhances the features of the contaminated area, and the cross-modal Transformer fusion module realizes multi-scale deep interaction of bimodal features to generate a fused feature map. S4: Input the fused feature map into the anti-interference recognition network, generate an adaptive mask through the clutter suppression submodule to suppress specular reflection and specular noise, enhance the feature response of micro-contamination and semi-transparent contamination through the cross-modal attention submodule, and output the contamination area mask and category information through the detection head; S5: Construct a joint optimization loss function, guided by optimizing the accuracy of pollution identification, and perform end-to-end training on the fusion network and the identification network; S6: Deploy the trained fusion recognition model to the edge computing platform, preprocess, fuse and recognize the real-time acquired dual-modal images in sequence, and output a fused image with pollution area annotations.

2. The method according to claim 1, characterized in that, Step S1 specifically includes: The visible light image and the infrared thermal imaging image are acquired synchronously through a dual-spectrum imaging device to ensure that the two are strictly aligned in spatial geometry. The visible light image contains RGB three-channel color information, which is used to characterize the appearance texture of the mirror surface, the visual morphology of reflected objects and pollutants; The infrared thermal imaging image is a single-channel grayscale data. By recording the difference in thermal radiation intensity between the mirror surface and the pollutants, it characterizes the outline information and temperature distribution characteristics of the pollutants, and can still maintain stable imaging capabilities under strong mirror reflection environment.

3. The method according to claim 1, characterized in that, Step S2, which identifies the degradation type of the mirror scene, specifically includes: We use a pre-trained vision-language model finely tuned on a mirror contamination dataset to perform semantic understanding and analysis on input visible light images; The visual-language model outputs a degradation type label for the current scene, which includes at least one or more combinations of: strong reflective interference, low contrast, transparent / semi-transparent contamination, minor stains, and background clutter occlusion.

4. The method according to claim 3, characterized in that, The matching enhancement strategy in step S2 specifically includes: Construct a prior knowledge base containing the mapping relationship between different degradation types and image processing algorithms; Based on the identified degradation type label, the corresponding enhancement strategy is retrieved from the prior knowledge base and invoked: When strong reflective interference is detected, a highlight suppression and reflection removal algorithm is applied to the visible light image. When the image is determined to be low contrast or has small stains, adaptive histogram equalization or sharpening filtering is performed on the visible light image. Simultaneously, contrast-limited adaptive histogram equalization is uniformly applied to the infrared thermal imaging images to improve the clarity of the thermal radiation contours in the contaminated areas.

5. The method according to claim 1, characterized in that, The specific processing steps of the semantic-driven fusion module in step S3 include: Feature maps from the intermediate layers of the pre-trained segmentation network are extracted as high-level semantic guidance maps. Calculate the similarity weights between the high-level semantic guidance map and the visible light feature map and the infrared feature map; The dual-modal features are weighted and fused according to the similarity weights, so that the network focuses on regions containing contaminating semantic information and suppresses the response of mirror background and reflective regions.

6. The method according to claim 5, characterized in that, The specific processing steps of the cross-modal Transformer fusion module in step S3 include: The enhanced visible light image features and infrared image features are divided into sequences of local image blocks of the same size; Global dependencies within a single modality are captured using a multi-head self-attention mechanism. By calculating the correlation weights between visible light feature sequences and infrared feature sequences through a cross-attention mechanism, the deep interaction and information complementarity of dual-modal features in multi-scale space are realized, and a fused feature map is output.

7. The method according to claim 1, characterized in that, Step S4, where the clutter suppression submodule generates the adaptive mask, specifically includes: Calculate the pixel-level difference between the enhanced visible light image and the infrared image to obtain the dual-modal residual map; The dual-modal residual map is subjected to threshold segmentation and morphological processing to generate an adaptive mask; Among them, specular reflection and specular noise appear as abnormally bright areas without thermal radiation response in the dual-modal residual map. The adaptive mask is used to weight and attenuate the fused feature map to reduce the impact of reflection interference on subsequent contamination identification.

8. The method according to claim 1, characterized in that, Step S5, which involves constructing the joint optimization loss function, specifically includes: The contamination identification loss is set to have a dominant weight in the joint optimization loss function, so as to force the learning objective of the fusion network to conform to the final identification accuracy. A contamination confidence feedback mechanism is introduced. When the contamination confidence output by the detection head is lower than a preset threshold, the auxiliary decomposition loss weight of the corresponding region is increased, forcing the fusion network to enhance the feature representation of the region in subsequent iterations. By combining pixel loss and gradient loss, we ensure that the fused feature map improves recognition performance while preserving the edge structure and texture details of the contaminated area, thus avoiding feature distortion caused by overfitting.

9. The method according to claim 1, characterized in that, In step S6, the edge computing platform is configured to support dynamic power consumption adjustment mode, specifically including: Monitor the average confidence level of pollution identification and ambient light intensity of the current processing frame to determine the complexity level of the current mirror scene; When a low-complexity scene or a highly reflective environment is detected, some multi-head self-attention calculation layers in the cross-modal Transformer fusion module are turned off or skipped, and the output of the semantically driven fusion module is directly used as the recognition input. While ensuring that the pollution identification recall rate meets the preset safety threshold, the power consumption and inference latency of the edge computing platform are reduced by dynamically decreasing the network computing depth, so as to meet the long-term battery life requirements of mobile inspection equipment or online monitoring systems.

10. A mirror contamination identification system combining visible light and infrared images, applied to the method as described in any one of claims 1 to 9, characterized in that, include: The image acquisition module is used to acquire visible light images and infrared thermal images of the same mirror area. The visible light images reflect the surface texture and reflection information of the mirror, and the infrared thermal images reflect the differences in thermal radiation of pollutants. The degradation perception preprocessing module is used to input the acquired bimodal images into the fine-tuned pre-trained vision-language model, identify the degradation type of the mirror scene and match the enhancement strategy to obtain the enhanced visible light image and the enhanced infrared image. The feature fusion module is used to construct a global and local feature collaborative fusion network. Taking the enhanced bimodal image as input, the semantic-driven fusion module enhances the features of the polluted area, and the cross-modal Transformer fusion module realizes multi-scale deep interaction of bimodal features to generate a fused feature map. The contamination identification module is used to input the fused feature map into the anti-interference identification network, generate an adaptive mask through the clutter suppression submodule to suppress specular reflection and specular noise, enhance the feature response of micro-contamination and semi-transparent contamination through the cross-modal attention submodule, and output the contamination area mask and category information through the detection head. The joint training module is used to construct a joint optimization loss function, with the aim of optimizing the accuracy of contamination identification, and to perform end-to-end training on the fusion network in the feature fusion module and the identification network in the contamination identification module; The deployment inference module is used to deploy the trained fusion recognition model to the edge computing platform, and to perform preprocessing, fusion and recognition processing on the real-time acquired dual-modal images in sequence, and output a fused image with pollution area annotations.

Images