Scraped car broken object double-source image fusion method based on material semantic perception and dynamic compensation

Abstract

The invention discloses a scraped car broken object double-source image fusion method based on material semantic perception and dynamic compensation, and belongs to the field of image processing. The method comprises the following steps: synchronously acquiring infrared and visible light images, extracting an optical flow field through a dynamic motion compensation module, and realizing pixel-level space-time alignment of double-source images; acquiring a pixel-level material proportion label by utilizing a semantic pre-recognition and multi-material unmixing module; dynamically generating an infrared and visible light weight map based on material characteristics to realize material adaptive weighted fusion; and image enhancement is further carried out through edge constraint and texture constraint dual mechanisms, the detail integrity of the fused image is ensured, artifacts are eliminated through post-optimization processing, and final fused image generation is completed. The fusion result directly supports an automatic sorting system, and the image recognition precision and sorting efficiency of materials such as metal, plastic and rubber are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of image processing, specifically to a dual-source image fusion method for scrapped car debris based on material semantic perception and dynamic compensation. Background Technology

[0002] Currently, the sorting of scrapped vehicle debris mainly relies on general image fusion technologies. Existing solutions are mostly based on static or general multimodal image fusion algorithms, such as simple weighted fusion of infrared and visible light images, and multi-scale fusion methods based on pyramid or wavelet transform. While these technologies can improve the complementarity of image information in general scenarios, none of them are adapted to the special scenarios of scrapped vehicle debris dynamic transport on sorting lines, multi-material mixing and superposition, and real-time requirements in industrial environments. Existing algorithms typically use fixed-weight fusion or globally uniform fusion strategies, lacking targeted processing for moving objects, material semantics, and texture details, making it difficult to achieve accurate and stable image fusion in complex industrial environments.

[0003] However, the aforementioned general fusion techniques exhibit significant limitations in the sorting scenario of scrapped vehicle shreds. First, the lack of a dynamic motion compensation mechanism fails to address the spatiotemporal misalignment of dual-source images caused by rapid movement on the sorting line, severely impacting the geometric consistency of the fused image. Second, the complexity of multi-material overlay is not considered, resulting in the output of only a single material label, failing to characterize the material mixing ratio and leading to image recognition confusion and missing semantic information. Furthermore, fixed-weight fusion strategies struggle to balance the characteristic differences of different materials (such as metal, plastic, and rubber), causing key sorting features (such as metal edges and plastic textures) to be weakened or lost during the fusion process. Finally, existing algorithms often ignore industrial real-time constraints (15-30fps), lacking image recognition optimization and enhancement mechanisms specifically for scrapped vehicle shreds scenarios. Complex algorithms are difficult to deploy, while lightweight algorithms sacrifice scenario adaptability, severely restricting the accuracy and efficiency of automated sorting. Therefore, a dedicated dual-source image fusion method specifically designed for scrapped vehicle shreds scenarios, incorporating motion compensation, semantic demixing, and dynamic weighting, is urgently needed. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a dual-source image fusion method for scrapped vehicle debris based on material semantic perception and dynamic compensation.

[0005] To achieve the above technology, the present invention adopts an overall architecture of dual-channel parallel processing and cross-channel collaborative fusion, wherein the dual channels include an infrared channel and a visible light channel; Furthermore, the infrared channel is sequentially connected to an IR input module, an IR normalization module, an IR denoising module, an IR dynamic alignment module, and an IR weight application module; the application modules sequentially complete the preprocessing and feature adjustment of the infrared image, retaining the thermal radiation characteristics of materials such as metals, while eliminating noise and motion distortion in the infrared image. Furthermore, the visible light channel is first connected to the RGB input module, the RGB white balance module and the RGB frame buffer module in sequence. Then, the RGB white balance module is connected in parallel to the RGB semantic pre-recognition module and the RGB weight + texture module. Furthermore, the cross-channel collaborative processing module includes: a dynamic motion compensation module, a multi-material semantic demixing module, a dynamic weight generation module, and a dual-constraint fusion + post-optimization module, which are responsible for realizing spatiotemporal synchronization of dual-source images, accurate demixing of material ratios, adaptive weight allocation, and detailed optimization of the fused image. Furthermore, the results of the RGB frame buffer module are input into the dynamic motion compensation module in the cross-channel collaborative fusion layer, which is also connected to the IR dynamic alignment module. The results of the IR dynamic alignment module and the RGB semantic pre-identification module are also used as inputs to the multi-material semantic demixing module. The output of the multi-material semantic demixing module is used as input to the dynamic weight generation module to obtain the infrared weight map and the visible light weight map. The weight map is also used as input to the IR weight application module and the RGB weight + texture module. Finally, the IR weight application module and the RGB weight + texture module are processed by the dual-constraint fusion + post-optimization module to obtain a high-precision fused image. The output module completes the format adaptation and interface conversion and transmits it to the downstream automated sorting system to support the precise sorting of crushed materials.

[0006] The following is a description of each module; Infrared channel: IR (Infrared) Input Module The IR input module is the signal input for the infrared channel of the dual-source image fusion system for scrapped vehicle debris, receiving infrared images of the scrapped vehicle debris acquired by the infrared thermal imaging camera; its input is a 16-bit grayscale infrared image with a resolution of [resolution missing]. , recorded as (in x , y (where the pixel coordinates are [0, 65535]). This implementation adapts the hardware interface and standardizes the data format, directly transmitting the image to the subsequent IR normalization module to ensure distortion-free reception of the original infrared signal.

[0007] IR normalization module The IR normalization module takes a 16-bit infrared image as input. First, dynamically calculate the effective grayscale range for each frame: after traversing the image and statistically analyzing the global grayscale distribution, remove extreme outliers to obtain the minimum effective grayscale value. and effective maximum grayscale Then, the 16-bit pixel value is converted to 8 bits using a linear mapping formula, as shown below: in, The result is the mapped 8-bit pixel value. If the calculation result exceeds [0, 255], it is corrected to 0 or 255 by clamping logic. This module outputs 8-bit infrared images. The IR noise reduction module ensures that the thermal radiation characteristics of different materials are not compressed.

[0008] IR noise reduction module The IR denoising module inputs the 8-bit infrared image output from the IR normalization module. (resolution) , x , y (Pixel coordinates, pixel value range [0, 255]), using a 3×3 Gaussian filtering algorithm, filter kernel... The calculation formula is: In the formula, and Represents the relative coordinates of the filter kernel. ; The standard deviation of the Gaussian function is used in the IR denoising module. ; through the Each pixel is averaged using a neighborhood weighted average, which effectively smooths out random thermal noise.

[0009] IR dynamic alignment module The input to the IR dynamic alignment module consists of two parts: one is the 8-bit denoised infrared image output by the IR denoising module. (Resolution 640×480) x , y The first is the pixel coordinates, and the second is the pixel-level optical flow field output by the dynamic motion compensation module (pixel value range [0, 255]). ( The x-direction motion vector. for y Directional motion vector, unit is pixel); the core processing is based on the inverse coordinate mapping of the optical flow field, that is, for each pixel of the aligned image ( x,y The corresponding coordinates of the infrared image in the original denoised image are calculated using the following formula: In the formula, This represents the coordinates in the original infrared image used to calculate the aligned pixel values; because and The pixel value may be non-integer or exceed the range of the original image. The module uses bilinear interpolation to complete the pixel value, with the following formula: In the formula, This indicates dynamic alignment of infrared image grayscale values; and They represent and The domain offset index, ; The weights are represented by the decimal part of the coordinates (e.g., , , To ensure a smooth transition of pixel values ​​after interpolation and avoid jagged edges, among other things... and They represent and The decimal part.

[0010] IR Weighting Application Module The input to the IR weighting application module consists of two parts: one is the 8-bit aligned infrared image output from the IR dynamic alignment module. (Resolution 640×480, x, y are pixel coordinates, pixel value range [0, 255]); Second, the infrared weight map output by the dynamic weight generation module. (Single channel, pixel value range [0, 1], reflecting the fusion ratio of infrared features at that pixel); the core processing is pixel-level weighted calculation, the formula is: In the formula, This is an 8-bit weighted infrared image; through this calculation, the metal region... Higher infrared thermal radiation information is enhanced in areas such as plastics and rubber. Lower infrared information is appropriately weakened; The module outputs an 8-bit weighted infrared image. (Resolution 640×480, pixel value range [0,255]), directly passed to the dual-constraint fusion + post-optimization module, and worked with visible light weighted features to complete the final fusion.

[0011] Visible light channel RGB (Visible Light) Input Module The RGB input module is the signal input to the visible light channel of the dual-source image fusion system for scrapped vehicle debris. It receives images of scrapped vehicle debris captured by a visible light camera. Its input is an 8-bit RGB visible light image with a resolution of [resolution missing]. , recorded as (where (c=R,G,B) are color channels, with pixel values ​​ranging from [0,255]); This module implements hardware interface adaptation and data format standardization, directly transmitting the image to the subsequent RGB white balance module to ensure distortion-free reception of the original visible light signal.

[0012] RGB white balance module RGB white balance module input RGB input module output 8-bit RGB image (resolution) , x , y Let c be the pixel coordinates, and G and B be the color channels, with pixel values ​​ranging from [0, 255]. First, calculate the mean grayscale value and the mean value of each channel of the image: traverse all pixels of the image and obtain the mean values ​​of the R, G, and B channels respectively. , and Then through the formula Calculate the global grayscale mean; White balance is then achieved by adjusting the channel ratio, using the following formula: In the formula, This represents the pixel value after white balance. This represents the average value of the corresponding channel (R / G / B). By adjusting this ratio, the grayscale distribution of each channel is brought closer to the global grayscale average, thus eliminating color cast.

[0013] RGB frame buffer module The RGB frame buffer module takes as input an 8-bit RGB image output from the RGB white balance module. (resolution) , x , y (where c is the pixel coordinate, G and B are the color channels, and the pixel value range is [0, 255]). The module adopts a FIFO buffer structure with a buffer depth of 2 frames. When a new frame is received... When the time comes, store it in the buffer queue, and at the same time output the earliest stored frame in the queue and the current frame to form two consecutive frames of image; This module outputs two consecutive frames of 8-bit RGB images (denoted as...). , The optical flow extraction module is connected to the dynamic motion compensation module to ensure that the optical flow extraction can accurately calculate the motion vector of the broken object, thus ensuring data continuity and spatiotemporal synchronization accuracy.

[0014] RGB semantic pre-recognition module The RGB semantic pre-recognition module inputs the 8-bit RGB image output by the RGB white balance module. (Resolution 640×480) x , y For pixel coordinates, c=R, G, B are color channels, and pixel value range [0, 255]); the module adopts an improved MobileNetV2 network and achieves semantic recognition through image recognition technology. The network structure includes: a 3×3 initial convolutional layer (32 channels, stride 2, activation function ReLU6), multiple sets of dilated residual blocks (dilation rates of 1, 2, and 4, output channel numbers of 16, 24, and 32 respectively), and an SE attention layer (compression ratio 16, through...). To achieve adaptive adjustment of channel weights, where As a channel feature, The first and second weights of the fully connected layer, (with Sigmoid activation), 3 transposed convolutional upsampling layers; During the training phase, a self-made dataset of scrapped car debris was used. The loss function was cross-entropy loss + Dice loss (weight ratio 1:1), with the following formulas: In the formula, An index indicating the material category; Indicates the actual material label; This represents the network's predicted probability; the optimizer chosen is AdamW (initial learning rate 1e). -4 (Cosine annealing strategy is used for attenuation). This module outputs a material probability map. The cross-channel multi-material semantic demixing module provides accurate preliminary classification results for semantic demixing in multi-material overlay scenarios.

[0015] RGB weights + texture module The input to the RGB weighting and texture module consists of two parts: one is the 8-bit white balance visible light image output by the RGB white balance module. (Resolution 640×480, c=R, G, B are color channels, pixel value range [0, 255]); Second, the visible light weight map output by the dynamic weight generation module. (Single channel, pixel value range [0, 1], reflecting the fusion ratio of visible light features at that pixel); The core processing flow includes: (1) Application of visible light weighting Pixel-level weighting is applied to the RGB three channels to enhance the visible light characteristics of areas with rich textures. The formula is as follows: in, This is an 8-bit weighted visible light image; through this calculation, the plastic... The color and texture information of areas with higher elevations and rubber is preserved with emphasis, while the metal... Visible light information in lower regions is appropriately weakened; (2) Texture feature extraction Multi-scale Gabor filtering is used to extract texture features from visible light images, capturing texture details of different materials; the Gabor kernel function formula is: In the formula, This represents the response of the Gabor filter at the pixel coordinates; and Represents the rotated pixel coordinates, where, , , x , y Set pixel coordinates; set scale (Adapts to the size of broken object texture), direction (Covering the main texture direction), phase Standard deviation Aspect Ratio ; right grayscale image Perform 4-directional Gabor filtering, and take the mean of the filtering results as the final texture map, as shown in the following expression: In the formula, An index indicating direction; Indicates the first Gabor filter kernels in each direction; * indicates a convolution operation; Represents the grayscale version of a visible light image; This is a single-channel texture value (range [0,255]), and a higher texture value indicates a richer texture in that area; (3) Output and performance The module outputs two types of data to the dual-constraint fusion + post-optimization module: one is a weighted visible light image. (3 channels, 640×480, [0,255]); the second is the texture map. (Single channel, 640×480, [0,255]).

[0016] Cross-channel collaboration layer Dynamic motion compensation module The dynamic motion compensation module is a core, pioneering module for cross-channel collaborative processing. It is specifically designed to solve the problem of spatiotemporal misalignment between infrared and visible light images caused by the movement of scrapped car debris on the sorting line, providing precise motion vectors for pixel-level alignment of dual-source images. Its core logic is to generate a pixel-level optical flow field by analyzing the motion characteristics of consecutive frames of visible light images. After verification by motion region masking and alignment accuracy, the field is output to the infrared dynamic alignment module to achieve reverse motion compensation of the infrared image. Specifically, the input to this module is two consecutive 8-bit RGB images (denoted as RGB) output by the RGB frame buffer module. and (Resolution 640×480, c=R, G, B, frame interval 10ms), output is pixel-level optical flow field. ( for x Directional motion vector, for y Direction of motion vector and motion mask The optical flow field is transmitted to the IR dynamic alignment module, and the motion mask is used for fault diagnosis backup. The core submodule processing flow includes: (1) Optical flow extraction submodule (core computing unit) A lightweight RAFT-Lite network is adopted to meet the real-time requirements of industrial applications, and the motion vector between two frames is extracted through feature matching and iterative optimization. The network structure includes: Feature extraction layer: 3×3 convolution (32 channels, stride 2, ReLU activation), compresses the input image into a 1 / 2 resolution feature map, preserving texture details; Expanded residual block group: 3 groups of residual blocks (expansion rate 1, 2, 4), output channels 16, 24, 32, to enhance the ability to capture different motion scales; Correlation layer + optical flow prediction head: Feature similarity is calculated through cross-correlation operation, and the initial optical flow field is output by combining GRU iterative optimization (6 iterations). Then, it is upsampled to restore the resolution to 640×480 to obtain the final optical flow field. ; (2) Motion mask generation submodule (region filtering unit) To distinguish moving regions from static backgrounds, a binary motion mask is generated based on the optical flow field; the processing logic is as follows: Calculate the motion vector magnitude for each pixel: , where mag reflects the intensity of the exercise; Binarization: When When the pixel value is less than 1, it is considered a moving area. Otherwise, it is a static region. ; Morphological post-processing: Perform a 3×3 rectangular kernel closing operation on the binary mask (iteration 1) to fill in small area gaps caused by noise and ensure the integrity of the contour of the moving area; (3) Dynamic alignment verification submodule (precision verification unit) The alignment effect of the optical flow field is verified by mutual information (MI) calculation to avoid the influence of erroneous motion vectors on subsequent fusion. Specifically, the input includes the optical flow field. IR noise reduction module output and the output of the RGB white balance module The processing steps are as follows: use right Perform initial alignment to obtain dynamically aligned infrared image grayscale values. ; calculate and The mutual information is expressed as: In the formula, This represents the joint gray-level probability distribution of the two images; and Let each represent its own marginal probability distribution; when If the alignment is deemed satisfactory, the optical flow field is output; otherwise, a retry mechanism is triggered (optical flow is re-extracted) to ensure the reliability of the output optical flow.

[0017] The optical flow field output by this module directly supports the IR dynamic alignment module, enabling pixel-level spatiotemporal synchronization of infrared and visible light images.

[0018] Multi-material semantic demixing module The input to the multi-material semantic demixing module consists of two parts: one is the 8-bit infrared aligned image output by the IR dynamic alignment module. (Resolution 640×480, pixel value range [0,255]); secondly, the material probability map output by the RGB semantic pre-recognition module. (Values ​​[0,1]); The core processing flow includes: (1) Overlay region detection submodule First, identify the material overlay region (single material regions do not require demixing; the semantic pre-identification result is directly retained); the processing logic is as follows: Calculate the "Top 1 material probability" for each pixel. Probability of Top 2 Material The difference is calculated using the following formula: ;when When the region is identified as a material overlay area, a binary mask is generated. Otherwise, it is a single-material area. ; (2) Dimensional Feature Construction Submodule For the superimposed region, multimodal features from infrared and visible light are fused to construct a 4-dimensional feature vector for demixing, specifically: In the formula, The visible light channel values ​​output by the RGB white balance module. The grayscale values ​​of the infrared aligned image are used to enhance material differentiation through multimodal feature fusion; (3) Sparse unmixing submodule Based on a pre-trained pure material feature library, the material ratio of the overlapping region is solved through sparse coding, and the optimization objective is: In the formula, Represents the multimodal feature vector (4-dimensional) of the current pixel; Indicates the first pre-training Material-like feature vectors; Indicates material The proportion (satisfying) ); This indicates that the sparsity of the material proportions is constrained by the sparse regularization coefficients, meaning that each pixel is dominated by only a few materials. Through this optimization, the preliminary material proportions of the overlay region are obtained. ; (4) Proportional smoothing submodule Spatially smooth the material proportions obtained from demixing to eliminate pixel-level abrupt changes in proportions; a 3×3 Gaussian filter is used, with the following formula: In the formula, This represents the Gaussian filter kernel. In the proportional smoothing submodule ; (5) Semantic-proportional fusion submodule The original semantic probability of a single material region is fused with the smoothing ratio of the overlay region to obtain the final semantic label map. : This label image has 5 channels, with each channel corresponding to the mixing ratio of a type of material (meeting the following requirements). ), and pass it to the dynamic weight generation module.

[0019] Dynamic weight generation module The input to the dynamic weight generation module is the semantic label map output by the multi-material semantic demixing module. (5 channels, Corresponding to metals, plastics, rubber, glass, and fabrics. For pixels ( x,y Material The proportion, satisfying ; The core processing flow includes: (1) Material-Weight Template Preset Based on the characteristics of the 5 material types, a preset basic weight template is provided: Metal( =1): Significant differences in infrared thermal radiation, preset infrared weights. Visible light weight ; plastic( =2), rubber ( =3): Rich visible light texture, preset , ; Glass( =4): Strong infrared transmittance and obvious visible light reflectivity, preset , ; Fabrics ( =5): Visible light color and texture are the main features, preset , ; (2) Initial weight calculation Based on the material proportions of the semantic tag map, a weighted fusion of the basic weight template is performed to obtain pixel-level initial weights: In the formula, For the initial infrared weights, The initial weights for visible light vary dynamically with the material proportions (e.g., if a pixel contains 60% metal and 40% plastic, then...). , ); (3) Weight smoothing To avoid weight jumps caused by abrupt changes in the material ratio of adjacent pixels (leading to artifacts in the fused image), the initial weights are spatially smoothed using a 1×3 horizontal + 3×1 vertical cross-shaped mean filter, with the following formula: Boundary pixels only take the average of the effective neighborhood, and the weight changes are made continuous by smoothing, which preserves the material boundary while eliminating high-frequency noise. (4) Weight constraints and output The smoothed weights are constrained to ensure they remain within the effective range and satisfy complementarity. Final output infrared weighted map (Transferred to the IR weighting application module) and visible light weighting map (Passed to the RGB weight + texture module), both are 640×480 single-channel images with pixel values ​​ranging from [0, 1].

[0020] Dual-constraint fusion + post-optimization module The input to the dual-constraint fusion + post-optimization module includes four types of data: firstly, the 8-bit weighted infrared image output by the IR weighting application module. (Resolution 640×480, pixel value range [0,255]); the second is an 8-bit weighted visible light image output by the RGB weight + texture module. (3 channels, c=R,G,B, [0,255]); the third is the output texture map. (Single channel, [0,255]); Fourth is the pre-calculated infrared edge map. Visible light edge map (Extracted using the Canny operator, with a high threshold of 150, a low threshold of 50, and binary edge values ​​of 0 / 255). The core processing flow includes: (1) Basic Fusion Submodule Preliminary fusion of dual-source features is achieved through weighted summation, using the following formula: ; In the formula, Represents the base fused image (3 channels). and The weight graph output by the dynamic weight generation module (satisfying) This step, through weight adjustment, initially achieves basic fusion by retaining infrared features and visible light features as needed. (2) Edge Constraint Submodule To address the edge blurring issue that may occur in basic fusion (such as metal edges being clear in infrared but blurry in visible light, which are easily weakened after fusion), edge difference compensation is introduced. The processing logic is as follows: Calculate the difference between the infrared and visible light edges: (A coefficient of 0.1 controls the compensation intensity to avoid overshoot). Edge enhancement is performed on the base fused image: ; (3) Texture Constraint Submodule To address the issue of missing texture in infrared images, visible light texture maps are used to supplement the texture details of the fused image; the processing logic is as follows: Calculate texture deviation value: Mean represents the global texture mean, with a coefficient of 0.08 controlling the texture enhancement level. Texture enhancement for edge-constrained images: ; (4) Post-optimization submodule The texture-constrained image undergoes detail optimization and image enhancement to eliminate artifacts during the fusion process (such as color blocks caused by weight jumps and overexposure caused by edge compensation). Pixel value cropping: Ensure that pixel values ​​are within the valid range; Contrast Enhancement: CLAHE (Limited Contrast Adaptive Histogram Equalization) is used with a block size of 8×8 and a contrast limit of 2.0 to improve local contrast (such as the distinction between metallic reflective areas and dark areas). Smoothing and noise reduction: 5×5 bilateral filtering (spatial standard deviation) is used. grayscale standard deviation ); (5) Output and performance The module then optimizes the results of the sub-modules as input and outputs a fused image. (24-bit RGB, resolution 640×480, pixel value range [0,255]), is passed to the final output module to complete the entire image generation process. This module is the final shaping stage of dual-source image fusion, and its output high-quality fused image directly determines the accuracy and efficiency of sorting scrapped car debris.

[0021] Beneficial effects of the present invention The dynamic motion compensation technology of this invention can eliminate the misalignment of dual-source images caused by the movement of scrapped car debris on the sorting line through scene-based optical flow extraction and reverse alignment. Compared with general static alignment algorithms, it can provide a spatially synchronized image basis for subsequent fusion and adapt to the dynamic transmission characteristics of the sorting line.

[0022] The multi-material semantic demixing technology of this invention is designed for the superimposed scenario of scrapped car debris. It can characterize the material ratio distribution, avoid the identification ambiguity caused by single material labels in general algorithms, provide accurate material basis for dynamic weight generation, and adapt to the material mixing characteristics of scrapped car debris.

[0023] The dynamic weight generation technology of this invention assigns fusion weights based on material characteristics. Infrared features of metal areas can be selectively preserved, while visible light textures of plastic and rubber areas can be prioritized. Compared with general fixed weights, it can balance the feature requirements of different materials and adapt to the material differences of scrapped car debris.

[0024] The edge-texture dual constraint and post-optimization technology of this invention can enhance key sorting features such as metal edges and plastic scratches, make up for the shortcomings of infrared and visible light images, and output more detailed features that better meet the sorting requirements compared with the soft blur problem of general algorithms, thus adapting to the sorting and identification needs of scrapped car debris. Attached Figure Description

[0025] Figure 1 This is a flowchart of the overall model of the present invention; Figure 2 This is a flowchart of the IR normalization module of the present invention; Figure 3 This is a flowchart of the IR noise reduction module of the present invention; Figure 4 This is a flowchart of the IR dynamic alignment module of the present invention; Figure 5 This is a flowchart of the IR weighting application module of the present invention; Figure 6 This is a flowchart of the RGB white balance module of the present invention; Figure 7 This is a flowchart of the RGB frame buffer module of the present invention; Figure 8 This is a flowchart of the RGB semantic pre-recognition module of the present invention; Figure 9 This is a flowchart of the RGB weight + texture module of the present invention; Figure 10 This is a flowchart of the dynamic motion compensation module of the present invention; Figure 11 This is a flowchart of the optical flow extraction submodule in the dynamic motion compensation module of the present invention; Figure 12 This is a flowchart of the motion mask generation submodule in the dynamic motion compensation module of the present invention; Figure 13 This is a flowchart of the dynamic alignment verification submodule in the dynamic motion compensation module of the present invention. Figure 14 This is a flowchart of the multi-material semantic demixing module of the present invention; Figure 15 This is a flowchart of the dynamic weight generation module of the present invention; Figure 16 This is a flowchart of the dual-constraint fusion + post-optimization module of the present invention. Detailed Implementation

[0026] The present invention will be further described in detail below with reference to specific embodiments.

[0027] like Figure 1As shown, a dual-source image fusion method for scrapped car debris based on material semantic perception and dynamic compensation adopts an overall architecture of dual-channel parallel processing and cross-channel collaborative fusion, wherein the dual channels include an infrared channel and a visible light channel. Furthermore, the infrared channel is sequentially connected to an IR input module, an IR normalization module, an IR denoising module, an IR dynamic alignment module, and an IR weight application module; the application modules sequentially complete the preprocessing and feature adjustment of the infrared image, retaining the thermal radiation characteristics of materials such as metals, while eliminating noise and motion distortion in the infrared image. Furthermore, the visible light channel is first connected to the RGB input module, the RGB white balance module and the RGB frame buffer module in sequence. Then, the RGB white balance module is connected in parallel to the RGB semantic pre-recognition module and the RGB weight + texture module. Furthermore, the cross-channel collaborative processing module includes: a dynamic motion compensation module, a multi-material semantic demixing module, a dynamic weight generation module, and a dual-constraint fusion + post-optimization module, which are responsible for realizing spatiotemporal synchronization of dual-source images, accurate demixing of material ratios, adaptive weight allocation, and detailed optimization of the fused image. Furthermore, the results of the RGB frame buffer module are input into the dynamic motion compensation module in the cross-channel collaborative fusion layer, which is also connected to the IR dynamic alignment module. The results of the IR dynamic alignment module and the RGB semantic pre-identification module are also used as inputs to the multi-material semantic demixing module. The output of the multi-material semantic demixing module is used as input to the dynamic weight generation module to obtain the infrared weight map and the visible light weight map. The weight map is also used as input to the IR weight application module and the RGB weight + texture module. Finally, the IR weight application module and the RGB weight + texture module are processed by the dual-constraint fusion + post-optimization module to obtain a high-precision fused image. The output module completes the format adaptation and interface conversion and transmits it to the downstream automated sorting system to support the precise sorting of crushed materials.

[0028] The following is a description of each module; Infrared channel: IR (Infrared) Input Module Specifically, the IR input module is the signal input to the infrared channel of the dual-source image fusion system for scrapped vehicle debris, receiving infrared images of the scrapped vehicle debris acquired by the infrared thermal imaging camera; its input is a 16-bit grayscale infrared image with a resolution of [resolution missing]. , recorded as (in x , y (where the pixel coordinates are [0, 65535]). This implementation adapts the hardware interface and standardizes the data format, directly transmitting the image to the subsequent IR normalization module to ensure distortion-free reception of the original infrared signal.

[0029] IR normalization module like Figure 2 As shown, the IR normalization module converts the 16-bit infrared image (pixel value range [0, 65535]) output by the IR input module into an 8-bit standard image (pixel value range [0, 255]), compressing the amount of data while preserving the differences in thermal radiation of broken materials such as metal, plastic, and rubber. Specifically, the input is a 16-bit infrared image. First, dynamically calculate the effective grayscale range for each frame: after traversing the image and statistically analyzing the global grayscale distribution, remove extreme outliers to obtain the minimum effective grayscale value. and effective maximum grayscale Then, the 16-bit pixel value is converted to 8 bits using a linear mapping formula, as shown below: in, The result is the mapped 8-bit pixel value. If the calculation result exceeds [0, 255], it is corrected to 0 or 255 by clamping logic. This module outputs 8-bit infrared images. The IR noise reduction module ensures that the thermal radiation characteristics of different materials are not compressed.

[0030] IR noise reduction module like Figure 3 As shown, the IR denoising module suppresses thermal noise in infrared images caused by ambient temperature fluctuations and camera sensor noise, while preserving the edge contours of materials such as metal and plastic. Specifically, the 8-bit infrared image output by the input IR normalization module (resolution) , x , y (Pixel coordinates, pixel value range [0, 255]), using a 3×3 Gaussian filtering algorithm, filter kernel... The calculation formula is: In the formula, and Represents the relative coordinates of the filter kernel. ; The standard deviation of the Gaussian function is used in the IR denoising module. ; through the Each pixel is averaged using a neighborhood weighted method to effectively smooth out random thermal noise; This module outputs a denoised infrared image. The infrared dynamic alignment module ensures that the edge features of the material are clearly distinguishable during the subsequent dynamic alignment process, providing a reliable infrared image for pixel-level alignment.

[0031] IR dynamic alignment module like Figure 4 As shown, the IR dynamic alignment module uses the optical flow field output by the dynamic motion compensation module to perform reverse motion compensation on the denoised infrared image, eliminating the misalignment between the infrared and visible light images caused by the movement of scrapped car debris, and achieving pixel-level spatiotemporal synchronization of dual-source images. Specifically, the input to this module consists of two parts: one is the 8-bit denoised infrared image output by the IR denoising module. (Resolution 640×480) x , y The first is the pixel coordinates, and the second is the pixel-level optical flow field output by the dynamic motion compensation module (pixel value range [0, 255]). ( The x-direction motion vector. for y Directional motion vector, unit is pixel); the core processing is based on the inverse coordinate mapping of the optical flow field, that is, for each pixel of the aligned image ( x,y The corresponding coordinates of the infrared image in the original denoised image are calculated using the following formula: In the formula, This represents the coordinates in the original infrared image used to calculate the aligned pixel values; because and The pixel value may be non-integer or exceed the range of the original image. The module uses bilinear interpolation to complete the pixel value, with the following formula: In the formula, This indicates dynamic alignment of infrared image grayscale values; and They represent and The domain offset index, ; The weights are represented by the decimal part of the coordinates (e.g., , , To ensure a smooth transition of pixel values ​​after interpolation and avoid jagged edges, among other things... and They represent and The decimal part; The module outputs an aligned 8-bit infrared image. The values ​​are respectively passed to the IR weighting application module and the cross-channel multi-material semantic demixing module.

[0032] IR Weighting Application Module like Figure 5 As shown, the IR weighting application module performs pixel-level weighted fusion of the infrared weight map output by the dynamic weighting generation module and the aligned infrared image, so that material regions with significant infrared features, such as metals, retain richer infrared thermal radiation information, providing weighted infrared features for subsequent dual-source image fusion. Specifically, the input to this module consists of two parts: one is the 8-bit aligned infrared image output by the IR dynamic alignment module. (Resolution 640×480, x, y are pixel coordinates, pixel value range [0, 255]); Second, the infrared weight map output by the dynamic weight generation module. (Single channel, pixel value range [0, 1], reflecting the fusion ratio of infrared features at that pixel); the core processing is pixel-level weighted calculation, the formula is: In the formula, This is an 8-bit weighted infrared image; through this calculation, the metal region... Higher infrared thermal radiation information is enhanced in areas such as plastics and rubber. Lower infrared information is appropriately weakened; The module outputs an 8-bit weighted infrared image. (Resolution 640×480, pixel value range [0,255]), directly passed to the dual-constraint fusion + post-optimization module, and worked with visible light weighted features to complete the final fusion.

[0033] Visible light channel RGB (Visible Light) Input Module The RGB input module is the signal input to the visible light channel of the dual-source image fusion system for scrapped vehicle debris. It receives images of scrapped vehicle debris captured by a visible light camera. Its input is an 8-bit RGB visible light image with a resolution of [resolution missing]. , recorded as (where (c=R,G,B) are color channels, with pixel values ​​ranging from [0,255]); This module implements hardware interface adaptation and data format standardization, directly transmitting the image to the subsequent RGB white balance module to ensure distortion-free reception of the original visible light signal.

[0034] RGB white balance module like Figure 6 As shown, the RGB white balance module corrects the white balance of the visible light image through the gray world algorithm, eliminates the color shift caused by the lighting in the sorting workshop, and ensures that the gray distribution of the R, G, and B channels tends to be neutral, providing a base image without color bias for subsequent semantic pre-recognition. Specifically, the input RGB input module outputs an 8-bit RGB image. (resolution) , x , y Let c be the pixel coordinates, and G and B be the color channels, with pixel values ​​ranging from [0, 255]. First, calculate the mean grayscale value and the mean value of each channel of the image: traverse all pixels of the image and obtain the mean values ​​of the R, G, and B channels respectively. , and Then through the formula Calculate the global grayscale mean; White balance is then achieved by adjusting the channel ratio, using the following formula: In the formula, This represents the pixel value after white balance. This represents the mean value of the corresponding channel (R / G / B). By adjusting this ratio, the grayscale distribution of each channel is brought closer to the global grayscale mean, thus eliminating color cast. This module outputs an 8-bit RGB image after white balance. The RGB frame buffer module ensures the accuracy of material color features during subsequent semantic pre-identification.

[0035] RGB frame buffer module like Figure 7 As shown, the RGB frame buffer module stores two consecutive frames of visible light images after white balance, providing continuous frame data for the optical flow extraction of the dynamic motion compensation module, thus solving the problem that motion vectors cannot be calculated from a single frame image. Specifically, the input is an 8-bit RGB image output from the RGB white balance module. (resolution) , x , y (where c is the pixel coordinate, G and B are the color channels, and the pixel value range is [0, 255]). The module adopts a FIFO buffer structure with a buffer depth of 2 frames. When a new frame is received... When the time comes, store it in the buffer queue, and at the same time output the earliest stored frame in the queue and the current frame to form two consecutive frames of image; This module outputs two consecutive frames of 8-bit RGB images (denoted as...). , The optical flow extraction is transferred to the dynamic motion compensation module to ensure that the motion vector of the broken object can be accurately calculated, thus guaranteeing data continuity and spatiotemporal synchronization accuracy.

[0036] RGB semantic pre-recognition module like Figure 8 As shown, the RGB semantic pre-identification module performs pixel-level material classification on visible light images, providing basic labels for multi-material semantic demixing; Specifically, the input is an 8-bit RGB image output from the RGB white balance module. (Resolution 640×480) x , y For pixel coordinates, c=R, G, B are color channels, and pixel value range [0, 255]); the module uses an improved MobileNetV2 network to achieve semantic recognition. The network structure includes: a 3×3 initial convolutional layer (32 channels, stride 2, activation function ReLU6), multiple sets of dilated residual blocks (dilation rates of 1, 2, and 4, output channel numbers of 16, 24, and 32 respectively), and an SE attention layer (compression ratio 16, through...). To achieve adaptive adjustment of channel weights, where As a channel feature, The first and second weights of the fully connected layer, (with Sigmoid activation), 3 transposed convolutional upsampling layers; During the training phase, a self-made dataset of scrapped car debris was used. The loss function was cross-entropy loss + Dice loss (weight ratio 1:1), with the following formulas: In the formula, An index indicating the material category; Indicates the actual material label; This represents the network's predicted probability; the optimizer chosen is AdamW (initial learning rate 1e). -4 (Cosine annealing strategy is used for attenuation). This module outputs a material probability map. The cross-channel multi-material semantic demixing module provides accurate preliminary classification results for semantic demixing in multi-material overlay scenarios.

[0037] RGB weights + texture module like Figure 9 As shown, the RGB weight + texture module applies dynamic weights to the visible light image to enhance the visible light characteristics of textured material areas such as plastic and rubber; on the other hand, it extracts texture information from the visible light image to make up for the lack of texture in the infrared image and provide fine-grained texture support for the fusion stage. Specifically, the input to this module consists of two parts: one is the 8-bit white balance visible light image output by the RGB white balance module. (Resolution 640×480, c=R, G, B are color channels, pixel value range [0, 255]); Second, the visible light weight map output by the dynamic weight generation module. (Single channel, pixel value range [0, 1], reflecting the fusion ratio of visible light features at that pixel); The core processing flow includes: (1) Application of visible light weighting Pixel-level weighting is applied to the RGB three channels to enhance the visible light characteristics of areas with rich textures. The formula is as follows: in, This is an 8-bit weighted visible light image; through this calculation, the plastic... The color and texture information of areas with higher elevations and rubber is preserved with emphasis, while the metal... Visible light information in lower regions is appropriately weakened; (2) Texture feature extraction Multi-scale Gabor filtering is used to extract texture features from visible light images, capturing texture details of different materials; the Gabor kernel function formula is: In the formula, This represents the response of the Gabor filter at the pixel coordinates; and Represents the rotated pixel coordinates, where, , , x , y Set pixel coordinates; set scale (Adapts to the size of broken object texture), direction (Covering the main texture direction), phase Standard deviation Aspect Ratio ; right grayscale image Perform 4-directional Gabor filtering, and take the mean of the filtering results as the final texture map, as shown in the following expression: In the formula, An index indicating direction; Indicates the first Gabor filter kernels in each direction; * indicates a convolution operation; Represents the grayscale version of a visible light image; This is a single-channel texture value (range [0,255]), and a higher texture value indicates a richer texture in that area; (3) Output and performance The module outputs two types of data to the dual-constraint fusion + post-optimization module: one is a weighted visible light image. (3 channels, 640×480, [0,255]); the second is the texture map. (Single channel, 640×480, [0,255]); It not only achieves targeted preservation of visible light features through weight adjustment, but also supplements the missing infrared details in the fused image through texture extraction, thereby improving the texture clarity of the fused result.

[0038] Cross-channel collaboration layer Dynamic motion compensation module like Figure 10 As shown, the dynamic motion compensation module is the core innovative module for cross-channel collaborative processing. It is specifically designed to solve the problem of spatiotemporal misalignment between infrared and visible light images caused by the movement of scrapped car debris on the sorting line, providing precise motion vectors for pixel-level alignment of dual-source images. Its core logic is to generate a pixel-level optical flow field by analyzing the motion characteristics of continuous frame visible light images. After verification by motion region masking and alignment accuracy, the field is output to the infrared dynamic alignment module to achieve reverse motion compensation of the infrared image. Specifically, the input to this module is two consecutive 8-bit RGB images (denoted as RGB) output by the RGB frame buffer module. and (Resolution 640×480, c=R, G, B, frame interval 10ms), output is pixel-level optical flow field. ( for x Directional motion vector, for y Direction of motion vector and motion mask The optical flow field is transmitted to the IR dynamic alignment module, and the motion mask is used for fault diagnosis backup. The core submodule processing flow includes: (1) Optical flow extraction submodule (core computing unit) like Figure 11 As shown, a lightweight RAFT-Lite network is used to adapt to the real-time requirements of industrial applications, and the motion vector between two frames is extracted through feature matching and iterative optimization. The network structure includes: Feature extraction layer: 3×3 convolution (32 channels, stride 2, ReLU activation), compresses the input image into a 1 / 2 resolution feature map, preserving texture details; Expanded residual block group: 3 groups of residual blocks (expansion rate 1, 2, 4), output channels 16, 24, 32, to enhance the ability to capture different motion scales; Correlation layer + optical flow prediction head: Feature similarity is calculated through cross-correlation operation, and the initial optical flow field is output by combining GRU iterative optimization (6 iterations). Then, it is upsampled to restore the resolution to 640×480 to obtain the final optical flow field. ; (2) Motion mask generation submodule (region filtering unit) like Figure 12As shown, to distinguish moving regions from static backgrounds, a binary motion mask is generated based on the optical flow field; the processing logic is as follows: Calculate the motion vector magnitude for each pixel: , where mag reflects the intensity of the exercise; Binarization: When When the pixel value is less than 1, it is considered a moving area. Otherwise, it is a static region. ; Morphological post-processing: Perform a 3×3 rectangular kernel closing operation on the binary mask (iteration 1) to fill small area gaps caused by noise and ensure the integrity of the contour of the moving area; (3) Dynamic alignment verification submodule (precision verification unit) like Figure 13 As shown, the alignment effect of the optical flow field is verified by mutual information (MI) calculation to avoid the influence of erroneous motion vectors on subsequent fusion; Specifically, the input includes the optical flow field. IR noise reduction module output and the output of the RGB white balance module The processing steps are as follows: use right Perform initial alignment to obtain dynamically aligned infrared image grayscale values. ; calculate and The mutual information is expressed as: In the formula, This represents the joint gray-level probability distribution of the two images; and Each represents its respective marginal probability distribution; when If the alignment is deemed satisfactory, the optical flow field is output; otherwise, a retry mechanism is triggered (optical flow is re-extracted) to ensure the reliability of the output optical flow.

[0039] The optical flow field output by this module directly supports the IR dynamic alignment module, enabling pixel-level spatiotemporal synchronization of infrared and visible light images.

[0040] Multi-material semantic demixing module like Figure 14 As shown, the multi-material semantic demixing module achieves accurate demixing of pixel-level material proportions by fusing infrared thermal radiation features and visible light semantic features, providing semantic-proportion fusion labels for dynamic weight generation; Specifically, its input consists of two parts: one is the 8-bit infrared aligned image output by the IR dynamic alignment module. (Resolution 640×480, pixel value range [0,255]); secondly, the material probability map output by the RGB semantic pre-recognition module. (Values ​​[0,1]); The core processing flow includes: (1) Overlay Region Detection Submodule First, identify the material overlay region (single material regions do not require demixing; the semantic pre-identification result is directly retained); the processing logic is as follows: Calculate the "Top 1 material probability" for each pixel. Probability of Top 2 Material The difference is calculated using the following formula: ;when When the region is identified as a material overlay area, a binary mask is generated. Otherwise, it is a single-material area. ; (2) Dimensional Feature Construction Submodule For the superimposed region, multimodal features from infrared and visible light are fused to construct a 4-dimensional feature vector for demixing, specifically: In the formula, The visible light channel values ​​output by the RGB white balance module. The grayscale values ​​of the infrared aligned image are used to enhance material differentiation through multimodal feature fusion; (3) Sparse unmixing submodule Based on a pre-trained pure material feature library, the material ratio of the overlapping region is solved through sparse coding, and the optimization objective is: In the formula, Represents the multimodal feature vector (4-dimensional) of the current pixel; Indicates the first pre-training Material-like feature vectors; Indicates material The proportion (satisfying) ); The sparsity regularization coefficient constrains the sparsity of the material proportions, meaning that each pixel is dominated by only a few materials; through this optimization, a preliminary material proportion for the overlay region is obtained. ; (4) Proportional smoothing submodule Spatially smooth the material proportions obtained from demixing to eliminate pixel-level abrupt changes in proportions; a 3×3 Gaussian filter is used, with the following formula: In the formula, This represents the Gaussian filter kernel. In the proportional smoothing submodule ; (5) Semantic-proportional fusion submodule The original semantic probability of a single material region is fused with the smoothing ratio of the overlay region to obtain the final semantic label map. : This label image has 5 channels, with each channel corresponding to the mixing ratio of a type of material (meeting the following requirements). ), and pass it to the dynamic weight generation module; The semantic label map output by this module is the core input of the dynamic weight generation module, which directly determines the fusion weight allocation of infrared and visible light features.

[0041] Dynamic weight generation module like Figure 15 As shown, the dynamic weight generation module dynamically generates fusion weights for infrared and visible light images based on the fine-grained material ratio information output by the multi-material semantic demixing module. This allows material regions with more prominent infrared features, such as metals, to retain infrared information, while material regions with richer visible light textures, such as plastics and rubber, to retain visible light information. Specifically, the input to this module is the semantic label map output by the multi-material semantic demixing module. (5 channels, Corresponding to metals, plastics, rubber, glass, and fabrics. For pixels ( x,y Material The proportion, satisfying ; The core processing flow includes: (1) Material-Weight Template Preset Based on the characteristics of the 5 material types, a preset basic weight template is provided: Metal( =1): Significant differences in infrared thermal radiation, preset infrared weights. Visible light weight ; plastic( =2), rubber ( =3): Rich visible light texture, preset , ; Glass( =4): Strong infrared transmittance and obvious visible light reflectivity, preset , ; Fabrics ( =5): Visible light color and texture are the main features, preset , ; (2) Initial weight calculation Based on the material proportions of the semantic tag map, a weighted fusion of the basic weight template is performed to obtain pixel-level initial weights: In the formula, For the initial infrared weights, The initial weights for visible light vary dynamically with the material proportions (e.g., if a pixel contains 60% metal and 40% plastic, then...). , ); (3) Weight smoothing To avoid weight jumps caused by abrupt changes in the material ratio of adjacent pixels (leading to artifacts in the fused image), the initial weights are spatially smoothed using a 1×3 horizontal + 3×1 vertical cross-shaped mean filter, with the following formula: Boundary pixels only take the average of the effective neighborhood, and the weight changes are made continuous by smoothing, which preserves the material boundary while eliminating high-frequency noise. (4) Weight constraints and output The smoothed weights are constrained to ensure they remain within the effective range and satisfy complementarity. Final output infrared weighted map (Transferred to the IR weighting application module) and visible light weighting map (Passed to RGB weight + texture module), both are 640×480 single-channel images with pixel value range [0, 1]; The dynamic weights generated by this module directly determine the fusion ratio of infrared and visible light features, which is a key step in achieving adaptive fusion of materials.

[0042] Dual-constraint fusion + post-optimization module like Figure 16 As shown, the dual-constraint fusion + post-optimization module is based on the weighted features of infrared and visible light. It improves the detail integrity of the fused image through edge constraints and texture constraints. Then, the post-optimization process eliminates fusion artifacts and outputs a high-precision fused image that can directly support the sorting of scrapped car debris. The input to this module includes four types of data: first, the 8-bit weighted infrared image output by the IR weighting application module. (Resolution 640×480, pixel value range [0,255]); the second is an 8-bit weighted visible light image output by the RGB weight + texture module. (3 channels, c=R,G,B, [0,255]); the third is the output texture map. (Single channel, [0,255]); Fourth is the pre-calculated infrared edge map. Visible light edge map (Extracted using the Canny operator, with a high threshold of 150, a low threshold of 50, and binary edge values ​​of 0 / 255). The core processing flow includes: (1) Basic Fusion Submodule Preliminary fusion of dual-source features is achieved through weighted summation, using the following formula: ; In the formula, Represents the base fused image (3 channels). and The weight graph output by the dynamic weight generation module (satisfying) This step, through weight adjustment, initially achieves basic fusion by retaining infrared features and visible light features as needed. (2) Edge Constraint Submodule To address the edge blurring issue that may occur in basic fusion (such as metal edges being clear in infrared but blurry in visible light, which are easily weakened after fusion), edge difference compensation is introduced. The processing logic is as follows: Calculate the difference between the infrared and visible light edges: (A coefficient of 0.1 controls the compensation intensity to avoid overshoot). Edge enhancement is performed on the base fused image: ; (3) Texture Constraint Submodule To address the issue of missing texture in infrared images, visible light texture maps are used to supplement the texture details of the fused image; the processing logic is as follows: Calculate texture deviation value: Mean represents the global texture mean, with a coefficient of 0.08 controlling the texture enhancement level. Texture enhancement for edge-constrained images: ; (4) Post-optimization submodule The texture-constrained image undergoes detail optimization to eliminate artifacts during the fusion process (such as color blocks caused by weight jumps and overexposure caused by edge compensation). Pixel value cropping: Ensure that pixel values ​​are within the valid range; Contrast Enhancement: CLAHE (Limited Contrast Adaptive Histogram Equalization) is used with a block size of 8×8 and a contrast limit of 2.0 to improve local contrast (such as the distinction between metallic reflective areas and dark areas). Smoothing and noise reduction: 5×5 bilateral filtering (spatial standard deviation) is used. grayscale standard deviation ); (5) Output and performance The module then optimizes the results of the sub-modules as input and outputs a fused image. (24-bit RGB, resolution 640×480, pixel value range [0,255]), is passed to the final output module to complete the entire image generation process. This module is the final shaping stage of dual-source image fusion, and its output high-quality fused image directly determines the accuracy and efficiency of sorting scrapped car debris.

[0043] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A dual-source image fusion method for scrapped vehicle debris based on material semantic perception and dynamic compensation, characterized in that: Infrared and visible light images are input, and a high-quality fused image is obtained through dual-channel parallel processing and a cross-channel collaborative fusion layer; The dual-channel system includes an infrared channel and a visible light channel; The infrared channel is sequentially connected to the IR input module, IR normalization module, IR denoising module, IR dynamic alignment module, and IR weighting application module; The visible light channel is connected in sequence to the RGB input module, the RGB white balance module, and the RGB frame buffer module; the RGB white balance module is connected in parallel to the RGB semantic pre-recognition module and the RGB weight + texture module. The cross-channel collaborative processing module includes: a dynamic motion compensation module, a multi-material semantic demixing module, a dynamic weight generation module, and a dual-constraint fusion + post-optimization module; The results from the RGB frame buffer module are input into the dynamic motion compensation module, which is also connected to the IR dynamic alignment module. The results of the IR dynamic alignment module and the RGB semantic pre-identification module are also used as inputs to the multi-material semantic demixing module. The output of the multi-material semantic demixing module is used as inputs to the dynamic weight generation module to obtain the infrared weight map and the visible light weight map. The weight map is also used as inputs to the IR weight application module and the RGB weight + texture module. The IR weight application module and the RGB weight + texture module are processed by the dual-constraint fusion + post-optimization module to obtain a high-precision fused image.

2. The dual-source image fusion method for scrapped vehicle debris based on material semantic perception and dynamic compensation according to claim 1, characterized in that: In the infrared channel, the IR input module receives a 16-bit grayscale infrared image, and the IR normalization module converts the 16-bit infrared image output from the IR input module into an 8-bit standard image and inputs it into the IR denoising module. The input to the IR dynamic alignment module includes two parts: one is the 8-bit denoised infrared image output from the IR denoising module, and the other is the pixel-level optical flow field output from the dynamic motion compensation module. ,in, for x Directional motion vector, for y The direction of motion vector; the input of the IR weighting application module includes two parts: one is the 8-bit aligned infrared image output by the IR dynamic alignment module, and the other is the infrared weight map output by the dynamic weight generation module.

3. The dual-source image fusion method for scrapped vehicle debris based on material semantic perception and dynamic compensation according to claim 2, characterized in that: The expression used by the IR normalization module to convert the 16-bit infrared image output by the IR input module into an 8-bit standard image is as follows: in, These are the mapped 8-bit pixel values; Indicates the minimum effective gray level; Indicates the maximum effective gray level; Represents a 16-bit infrared image; The IR denoising module employs a 3×3 Gaussian filtering algorithm, with the standard deviation of the Gaussian function in the IR denoising module set to 1.

2. The alignment operation in the IR dynamic alignment module first performs reverse coordinate mapping, as shown in the following expression: In the formula, Represents the coordinates in the original infrared image used to calculate the aligned pixel values; x , y These are pixel coordinates; At the same time, bilinear interpolation is used to complete the pixel values, as shown in the following expression: In the formula, This indicates dynamic alignment of infrared image grayscale values; and They represent and The domain offset index, ; Indicates weight; This indicates the output of the IR denoising module; The IR weighting application module performs a weighting operation, expressed as follows: In the formula, An 8-bit weighted infrared image; This represents the infrared weighted graph.

4. The dual-source image fusion method for scrapped vehicle debris based on material semantic perception and dynamic compensation according to claim 1, characterized in that: In the visible light channel, the RGB input module takes an 8-bit RGB visible light image as input and directly transmits it to the RGB white balance module; the RGB frame buffer module takes an 8-bit RGB image output by the RGB white balance module as input and outputs two consecutive 8-bit RGB images; the RGB semantic pre-recognition module takes an 8-bit RGB image output by the RGB white balance module as input and outputs a material probability map. The input to the RGB weight + texture module consists of two parts: one is the 8-bit white balance visible light image output by the RGB white balance module, and the other is the visible light weight map output by the dynamic weight generation module.

5. The dual-source image fusion method for scrapped vehicle debris based on material semantic perception and dynamic compensation according to claim 4, characterized in that: In the RGB white balance module, all pixels of the image are first traversed to obtain the mean values ​​of the R, G, and B channels. , and Then through the formula Calculate the global grayscale mean; White balance is then achieved by adjusting the channel ratio, using the following formula: In the formula, This represents the pixel value after white balance. represents the mean value of the corresponding channel (R / G / B); c represents the color channel; The RGB frame buffer module adopts a FIFO buffer structure with a buffer depth of 2 frames. The RGB semantic pre-recognition module employs an improved MobileNetV2 network, including: a 3×3 initial convolutional layer with 32 channels, a stride of 2, and a ReLU activation function; multiple sets of dilated residual blocks with dilation rates of 1, 2-to-4, and output channels of 16, 24, and 32 respectively; and an SE attention layer with a compression ratio of 16. To achieve adaptive adjustment of channel weights, As a channel feature, The first and second weights of the fully connected layer, The system uses Sigmoid activation; three transposed convolutional upsampling layers; a self-made dataset of scrapped car debris is used during training; the loss functions are cross-entropy loss and Dice loss with a weight ratio of 1:1; the optimizer is AdamW, with an initial learning rate of 1e. -4 Cosine annealing strategy is used for attenuation; The processing flow of the RGB weight + texture module includes: Visible light weighting application: Pixel-level weighting is applied to the RGB three channels to enhance the visible light characteristics of texture-rich areas. The formula is: in, An 8-bit weighted visible light image; This represents the pixel value after white balance. Represents the visible light weighting graph; Texture feature extraction: Multi-scale Gabor filtering is used to extract texture features from visible light images, capturing texture details of different materials; The Gabor kernel function formula is: In the formula, This represents the response of the Gabor filter at the pixel coordinates; and Represents the rotated pixel coordinates, where, , , x , y Set pixel coordinates; set scale ,direction Phase Standard deviation Aspect Ratio ; right The grayscale image is subjected to 4-directional Gabor filtering, and the mean of the filtering results is taken as the final texture map, as shown in the following expression: In the formula, An index indicating direction; Indicates the first Gabor filter kernels in each direction; * indicates a convolution operation; Represents the grayscale version of a visible light image; This is a single-channel texture value; Output and performance: Output two types of data: one is a weighted visible light image; the other is a texture map.

6. The dual-source image fusion method for scrapped vehicle debris based on material semantic perception and dynamic compensation according to claim 1, characterized in that: In the cross-channel collaborative layer, the dynamic motion compensation module takes two consecutive 8-bit RGB images output by the RGB frame buffer module as input and outputs an optical flow field to achieve pixel-level spatiotemporal synchronization of infrared and visible light images. The multi-material semantic demixing module takes two inputs: an 8-bit infrared aligned image output by the IR dynamic alignment module and a material probability map output by the RGB semantic pre-identification module, and outputs a semantic label map. The dynamic weight generation module takes the semantic label map output by the multi-material semantic demixing module as input and finally outputs an infrared weight map and a visible light weight map. The dual-constraint fusion + post-optimization module takes four inputs: an 8-bit weighted infrared image output by the IR weight application module, an 8-bit weighted visible light image output by the RGB weight + texture module, an output texture map, and infrared edge maps and visible light edge maps extracted by the Canny operator.

7. The dual-source image fusion method for scrapped vehicle debris based on material semantic perception and dynamic compensation according to claim 6, characterized in that: The processing of the dynamic motion compensation module includes: Optical flow extraction submodule: Employs a lightweight RAFT-Lite network; the network structure includes: Feature extraction layer: 3×3 convolution, set to 32 channels, stride 2, ReLU activation, compresses the input image into a 1 / 2 resolution feature map, preserving texture details; Expanded residual block group: 3 groups of residual blocks, with expansion rates of 1, 2 and 4, and output channels of 16, 24 and 32, to enhance the ability to capture different motion scales; Correlation layer + optical flow prediction head: Feature similarity is calculated through cross-correlation operation, combined with GRU iterative optimization, set to 6 iterations, to output the initial optical flow field, which is then upsampled to restore the resolution to 640×480 to obtain the final optical flow field. ; Motion mask generation submodule: To distinguish moving regions from static backgrounds, a binary motion mask is generated based on the optical flow field. The processing logic is as follows: Calculate the motion vector magnitude for each pixel: , where mag represents the intensity of the exercise; x , y These are pixel coordinates; Binarization: When When the pixel value is less than 1, it is considered a moving area. Otherwise, it is a static region. ; Morphological post-processing: Perform a 3×3 rectangular kernel closing operation on the binary mask, set to iterate once, to fill small area gaps caused by noise and ensure the integrity of the contour of the moving area; Dynamic alignment verification submodule: Input includes optical flow field IR noise reduction module output and the output of the RGB white balance module The processing steps are as follows: use right Perform initial alignment to obtain dynamically aligned infrared image grayscale values. ; calculate and The mutual information is expressed as: In the formula, This represents the joint gray-level probability distribution of the two images; and Let each represent its own marginal probability distribution; when If the alignment is deemed satisfactory, the optical flow field is output; otherwise, the optical flow is re-extracted to ensure the reliability of the output optical flow.

8. The dual-source image fusion method for scrapped vehicle debris based on material semantic perception and dynamic compensation according to claim 6, characterized in that: The processing flow of the multi-material semantic demixing module includes: Overlay region detection submodule: First, it identifies the material overlay region. Simultaneously, single-material regions do not require demixing; the semantic pre-identification result is directly retained. The processing logic is as follows: Calculate the "Top 1 material probability" for each pixel. Probability of Top 2 Material The difference is calculated using the following formula: , x , y For pixel coordinates; when When the region is identified as a material overlay area, a binary mask is generated. Otherwise, it is a single-material area. ; 4D Feature Construction Submodule: For the overlapping region, multimodal features of infrared and visible light are fused to construct a 4D feature vector for demixing, specifically: In the formula, The visible light channel values ​​output by the RGB white balance module. The grayscale values ​​of the infrared aligned image; The sparse unmixing module, based on a pre-trained pure material feature library, solves for the material ratio of the superimposed region through sparse coding. The optimization objective is: In the formula, This represents the 4-dimensional multimodal feature vector of the current pixel; Indicates the first pre-training Material-like feature vectors; Indicates material The proportion satisfies ; The sparsity of the material ratio is constrained by the sparse regularization coefficient; through this optimization, the preliminary material ratio of the overlay region is obtained. ; The scaling smoothing submodule: performs spatial smoothing on the material scaling obtained from demixing, eliminating pixel-level scaling abrupt changes. It uses a 3×3 Gaussian filter, with the following formula: In the formula, This represents the Gaussian filter kernel. and Represents the relative coordinates of the filter kernel. In the proportional smoothing submodule ; Semantic-Scale Fusion Submodule: This module fuses the original semantic probabilities of a single material region with the smooth scale of the overlay region to obtain the final semantic label map. The expression is: The label image has 5 channels, with each channel corresponding to the mixing ratio of a type of material, satisfying... ; It can be used with metals, plastics, rubber, glass, and fabrics.

9. The dual-source image fusion method for scrapped vehicle debris based on material semantic perception and dynamic compensation according to claim 6, characterized in that: The process of the dynamic weight generation module includes: Material-Weight Template Presets: Based on the characteristics of the 5 material types, a preset basic weight template is provided: Metal =1: Significant differences in infrared thermal radiation, preset infrared weights. Visible light weight ; plastic =2. Rubber =3: Rich visible light texture, preset , ; Glass =4: Strong infrared transmittance and obvious visible light reflectivity, preset. , ; fabric =5: Visible light color and texture are the main features, preset. , ; Initial weight calculation: Based on the material ratio of the semantic tag map, the basic weight template is weighted and fused to obtain the pixel-level initial weights, as shown in the following expression: In the formula, For the initial infrared weights, Initial weights for visible light; Represents the visible light weighting graph; x , y These are pixel coordinates; Weight smoothing: Spatial smoothing is applied to the initial weights using a 1×3 horizontal + 3×1 vertical cross-shaped mean filter, with the following formula: Boundary pixels only take the average of the effective neighborhood, and the weight changes are made continuous by smoothing, which preserves the material boundary while eliminating high-frequency noise. Weight Constraints and Output: Range constraints are applied to the smoothed weights to ensure they remain within the effective interval and satisfy complementarity. The expression is as follows: Final output infrared weighted map The data is then passed to the IR weighting application module and the visible light weighting map. The image is passed to the RGB weight + texture module. Both are 640×480 single-channel images with pixel values ​​ranging from [0, 1].

10. A dual-source image fusion method for scrapped vehicle debris based on material semantic perception and dynamic compensation according to claim 6, characterized in that: The process of dual-constraint fusion + post-optimization module includes: Basic fusion submodule: Achieves preliminary fusion of dual-source features through weighted summation, using the following formula: ； In the formula, This represents the base fused image with 3 channels. and The weight graph output by the dynamic weight generation module satisfies ,Right now Represents the infrared weighted graph; Represents the visible light weighting graph; This represents the 8-bit weighted infrared image output by the IR weighting application module; This represents an 8-bit weighted visible light image output by the RGB weights and texture module. Edge Constraint Submodule: Introduces edge difference compensation to address potential edge blurring issues in basic fusion. The processing logic is as follows: Calculate the difference between the infrared and visible light edges: A coefficient of 0.1 controls the compensation intensity to avoid overshoot; Edge enhancement is performed on the base fused image: Texture Constraint Submodule: Addressing the issue of missing textures in infrared images, this module utilizes visible light texture maps to supplement the texture details of the fused image. The processing logic is as follows: Calculate texture deviation value: Mean represents the global texture mean, with a coefficient of 0.08 controlling the texture enhancement level. This is a single-channel texture value. Represents visible light texture map Texture enhancement for edge-constrained images: ; Post-optimization submodule: Performs detail optimization on the texture-constrained image to eliminate artifacts during the fusion process. Steps include: Pixel value cropping: Ensure that pixel values ​​are within the valid range; Contrast enhancement: CLAHE is used with a block size of 8×8 and a contrast limit of 2.0 to improve local contrast. Smoothing and noise reduction: 5×5 bilateral filtering is used, and the spatial standard deviation is set. grayscale standard deviation ; Output and Performance: The module takes the results of subsequent optimization sub-modules as input and outputs a fused image. .

Images