A large model guidance-based multi-modal fusion plate defect detection method

Abstract

The application provides a multi-modal fusion plate defect detection method based on large model guidance, belonging to the technical field of plate determination. The method obtains visible light images, infrared thermal images and environmental parameters of the plate under the same space-time; the images are globally semantically analyzed by a lightweight visual language large model to obtain a defect region of interest, and a language large model is used to generate a modal adaptive fusion weight in combination with the environmental parameters; after extracting a local feature sequence in the region of interest, a cross-modal state space model is input, the hidden states of the visible light and infrared thermal modal are mutually permeated and interacted in the sequence scanning, and the interaction process is guided by the adaptive fusion weight; finally, the defect category and position are output based on the cross-modal fusion features. The method dynamically adjusts the fusion strategy and cross-modal deep interaction through environmental perception, overcomes the defect that the fixed fusion is difficult to adapt to the working condition change, and significantly improves the robustness and precision of the plate defect detection in complex environments.

Description

Technical Field

[0001] This invention belongs to the field of plate material testing technology, specifically relating to a multimodal fusion plate material defect detection method based on a large model-guided approach. Background Technology

[0002] During the wood processing and production process, defects such as dead knots, cracks, mold, and internal pores are easily generated on the surface and inside of boards, directly affecting their physical properties and economic value. Existing automated inspection methods mainly rely on single-modal data for defect identification, which has obvious technical limitations. When using visible light images alone, they are easily interfered with by factors such as the natural texture of the wood surface, water stains, and shadows, leading to misjudging normal textures as defects or missing real defects. When using infrared thermal imaging data alone, although it can reflect the density changes and pore information inside the board, it loses the fine color and texture features of the surface, making it difficult to distinguish between different types of surface defects.

[0003] Some existing solutions attempt to perform multimodal fusion detection by combining visible light images with infrared thermal images, but their fusion strategies typically employ a fixed early cascaded network structure and fixed fusion attention weights. In real-world production environments, wood species frequently change, and the texture characteristics of different species vary significantly. Ambient temperature and humidity fluctuate with seasons and weather, affecting the penetration and imaging quality of infrared thermal imaging. Production line speeds are also adjusted according to production plans, impacting image acquisition quality. Fixed fusion weights cannot adapt to these dynamic changes in operating conditions. When environmental conditions change, previously effective fusion strategies may degrade in performance, leading to increased false alarm rates or false negative rates.

[0004] Therefore, how to adaptively adjust the degree of fusion of different modes according to real-time production conditions during multimodal fusion detection, so that the detection system can automatically adjust the degree of reliance on visible light information and infrared thermal information according to changes in environmental parameters such as wood species, temperature and humidity, and linear velocity, is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] In view of the above-mentioned defects or deficiencies in the existing technology, a multimodal fusion plate defect detection method based on a large model is provided, including the following steps: Acquire visible light images, infrared thermal images, and environmental parameter data of the board material at the same spatial location and time stamp; The visible light image and the infrared thermal image are input into a lightweight visual language model for global semantic analysis to obtain the region of interest of potential defects; and the environmental parameter data are input into the language model for semantic reasoning to generate adaptive fusion weights for the visible light mode and the infrared thermal mode under the current working conditions. Based on the region of interest, local images at corresponding positions are cropped from the original resolution visible light image and infrared thermal image, and local feature sequences are extracted. The local feature sequences of the visible light mode and the infrared thermal mode are input into the cross-modal state space model. During the sequence scanning process, the hidden states of the two modes are allowed to interpenetrate and interact to obtain cross-modal fusion features. The cross-modal state space model is guided by the adaptive fusion weights during the interaction process. Based on the cross-modal fusion features, defect category prediction and defect location regression are performed, and defect detection results are output.

[0006] According to the technical solution provided in this application, the visible light image and the infrared thermal image are obtained by using an industrial high-definition RGB camera and an infrared thermal imager deployed on the production line, and by adopting a hardware pulse synchronization triggering mechanism to collect the same plate position at the same time stamp; After acquisition, the infrared thermal image is mapped to the spatial coordinate system of the visible light image through a perspective transformation matrix, and spatial alignment is achieved by bilinear interpolation resampling. The sampling frequency of the environmental parameter data is lower than the image acquisition frame rate. A time proximity window matching mechanism is used to align the environmental parameter data with the image data in the time dimension. Specifically, for each set of image timestamps, the environmental parameter data with the smallest absolute time deviation and a deviation not exceeding the preset maximum tolerable time delay threshold is retrieved from the timestamp sequence of the environmental parameter data, and it is bound to the current image frame pair.

[0007] According to the technical solution provided in this application, the step of inputting the visible light image and the infrared thermal image into a lightweight visual language large model for global semantic analysis to obtain the region of interest for potential defects includes the following steps: After downsampling and reducing the resolution of the visible light image and the infrared thermal image, a global multimodal image tensor is constructed. The global multimodal image tensor is input into the visual encoder of the lightweight visual language large model, and the output is a probabilistic semantic map representing the possibility of defects. An adaptive threshold is used to perform threshold activation and connected component analysis on the probabilistic semantic graph to extract a set of coordinate boxes for potential defect regions of interest with high confidence.

[0008] According to the technical solution provided in this application, the step of inputting the environmental parameter data into a language large model for semantic reasoning to generate adaptive fusion weights for the visible light mode and the infrared thermal mode under the current working condition includes the following steps: The environmental state vector is input into the prompt word embedding layer of the language big model; the environmental state vector is composed of the unique thermal code of the wood species, the standardized environmental temperature value, the environmental humidity value and the pipeline linear velocity value. The language model, combined with the built-in domain expert knowledge base, performs semantic reasoning on the texture artifact characteristics of wood species in the current production environment and the physical changes in infrared thermal imaging penetration under the current temperature and humidity, and outputs continuous visual cue word features related to the current working conditions. The visual cue word features are input into a weighted evolution network composed of a multilayer perceptron, and the visible light mode weights and infrared thermal mode weights are calculated using the Softmax activation function.

[0009] According to the technical solution provided in this application, the step of cropping local images at corresponding positions from the original resolution visible light image and infrared thermal image based on the region of interest, and extracting local feature sequences, includes the following steps: Based on the region of interest coordinate boxes generated by the large model, the original full-resolution visible light image and infrared thermal image are locally cropped with high definition to obtain visible light local images and infrared thermal local images. The visible light local image and the infrared thermal local image are respectively input into a shallow residual network to extract local feature maps with edge information; The local feature maps are flattened along the spatial dimension into a one-dimensional sequence to obtain visible light local feature sequences and infrared thermal local feature sequences, respectively.

[0010] According to the technical solution provided in this application, the cross-modal state space model is a cross-modal interactive scanning model constructed based on a selective state space model. The hidden states of the two modalities interpenetrate and interact with each other to obtain cross-modal fusion features, which is achieved through the following methods: At each time step of the sequence scan, the current hidden state of the visible light mode is determined by the following inputs: the result of the visible light mode's previous hidden state after transformation by the visible light state space matrix and the input control matrix, and the input of the infrared thermal mode's previous hidden state after adjustment by the infrared thermal state space matrix, the infrared thermal input control matrix, and the cross-modal information penetration coefficient. The current hidden state of the infrared thermal mode is determined by the following inputs: the result of the transformation of the infrared thermal state space matrix and the input control matrix of the previous hidden state of the infrared thermal mode itself, and the input of the previous hidden state of the visible light mode after adjustment by the visible light state space matrix, the visible light input control matrix and the cross-modal information penetration coefficient. The cross-modal information penetration coefficient is obtained by dynamic mapping of the cross-correlation characteristics of the two sets of modes at the current moment.

[0011] According to the technical solution provided in this application, the cross-modal information penetration coefficient is also dynamically adjusted by the adaptive fusion weights, specifically as follows: The adaptive fusion weights include visible light mode weights and infrared thermal mode weights; The visible light mode weights are fed forward to the cross-modal state space model to adjust the magnitude of the cross-modal information penetration coefficient when the infrared thermal mode hidden state penetrates into the visible light mode; The infrared thermal mode weights are fed forward to the cross-modal state space model to adjust the magnitude of the cross-modal information penetration coefficient when the visible light mode hidden state penetrates into the infrared thermal mode.

[0012] According to the technical solution provided in this application, obtaining the cross-modal fusion feature includes the following steps: The hidden state sequence output after scanning the cross-modal state space model is remapped back to two-dimensional space to obtain visible light fusion feature map and infrared thermal fusion feature map. The visible light fusion feature map and the infrared thermal fusion feature map are fed into a multi-scale attention fusion component. The multi-scale attention fusion component uses dilated convolution with different receptive fields and spatial attention mechanism to further fuse the probabilistic semantic map representing the possibility of defects output by the large model with the local defect features extracted by the cross-modal state space model, and outputs a fusion feature tensor.

[0013] According to the technical solution provided in this application, the defect category prediction and defect location regression based on the cross-modal fusion features, and the output of defect detection results, are achieved through a multi-task detection head. The multi-task detection head includes a classification branch and a regression branch; the classification branch takes the cross-modal fusion features as input and outputs a probability distribution prediction of the board defect category; the regression branch takes the cross-modal fusion features as input and outputs the geometric parameters of the defect bounding box; the defect categories include dead knots, cracks, holes, mold, and no defects; During model training, a joint loss function is used for end-to-end training. The joint loss function is composed of a weighted sum of classification cross-entropy loss, regression mean squared error loss, and cross-modal alignment loss. The cross-modal alignment loss is the alignment loss between the semantic features generated by the large model and the cross-modal fused features.

[0014] According to the technical solution provided in this application, a real-time detection step is also included: Visible light image data, infrared thermal image data, and environmental parameter data of the board material are collected on-site simultaneously, or historical data stored in a multimodal database are retrieved for spatiotemporal alignment preprocessing. The preprocessed data is input into the trained and converged defect detection model to perform forward inference and output the defect detection results, which include defect category, location coordinates and confidence level. The defect detection results are used to generate a diagnostic report and store it as text data. This report is then uploaded to the production line quality traceability database as a test record for this batch of boards and reserved as backup data for subsequent model fine-tuning and knowledge base updates.

[0015] Compared with the prior art, the beneficial effects of this application are as follows: First, by introducing a large language model as an environmental perception agent, real-time collected environmental parameters such as wood species, ambient temperature, humidity, and production line speed are input into the large language model. Combined with a built-in domain expert knowledge base, semantic reasoning is performed on the current production conditions to dynamically generate adaptive fusion weights for visible light and infrared thermal modes. These weights are then fed forward into the cross-modal state space model to adjust the degree of information penetration between modes. This mechanism enables the detection system to adaptively adjust its trust level for different modes according to changes in the production environment. When excessively high humidity leads to a decrease in infrared thermal imaging penetration, the system automatically reduces the fusion weight of the infrared thermal mode, effectively shielding the detection performance fluctuations caused by environmental interference in a single mode, and significantly improving the robustness of detection under complex industrial conditions.

[0016] Second, by constructing a cross-modal state space model, the hidden states of the visible light mode and the infrared thermal mode can interact and permeate each other during sequential scanning. The internal density and moisture mutation information carried by the infrared thermal mode can directly affect the hidden state update process of the visible light mode in linear temporal scanning, correcting the feature deviations of the visible light mode caused by artifacts such as surface water stains and dust. This achieves deep complementarity between the two modes at the underlying feature level, avoiding the problem of insufficient information interaction between modes in traditional simple cascade fusion methods, and improving the expressive power of defect features.

[0017] Third, by employing a hierarchical collaborative architecture that combines a large model for global semantic screening with a small model for local fine-grained detection, the large model first performs a rapid coarse screening on the downsampled global image to extract regions of interest (ROIs) for potential defects. Then, a lightweight network performs fine-grained feature extraction only on the local high-resolution images corresponding to the ROIs. This architecture significantly reduces the computational cost of high-resolution feature extraction across the entire image. At the same time, the open-domain generalization capability of the large model enables the detection system to discover complex and unknown superimposed defects, improving the overall detection accuracy and coverage. Attached Figure Description

[0018] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 The flowchart of the multimodal fusion plate defect detection method based on large model guidance provided in this application; Figure 2The structural diagram of the multimodal fusion plate defect detection system based on large model guidance provided in this application; Figure 3 The structural diagram of the Mamba feature dynamic fusion module based on cross-modal interaction provided in this application. Detailed Implementation

[0019] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] Example 1 As mentioned in the background section, this application proposes a multimodal fusion method for detecting defects in sheet metal based on a large model, such as... Figure 1 As shown, it includes the following steps: S1. Acquire visible light images, infrared thermal images, and environmental parameter data of the board material at the same spatial location and time stamp; S2. Input the visible light image and the infrared thermal image into a lightweight visual language model for global semantic analysis to obtain the region of interest of potential defects; and input the environmental parameter data into the language model for semantic reasoning to generate adaptive fusion weights for the visible light mode and the infrared thermal mode under the current working conditions. S3. Based on the region of interest, crop the corresponding local images from the original resolution visible light image and infrared thermal image, and extract the local feature sequence. S4. Input the local feature sequences of the visible light mode and the local feature sequences of the infrared thermal mode into the cross-modal state space model. During the sequence scanning process, allow the hidden states of the two modes to interpenetrate and interact to obtain cross-modal fusion features. The cross-modal state space model is guided by the adaptive fusion weights during the interaction process. S5. Based on the cross-modal fusion features, perform defect category prediction and defect location regression, and output the defect detection results.

[0022] Specifically, both the lightweight visual-language big model and the language big model are deep learning models pre-trained on general-purpose data. The region of interest (ROI) refers to the local image location identified by the lightweight visual-language big model as potentially containing defects, represented by a rectangular coordinate box. Adaptive fusion weights refer to the contribution ratios of the two modalities during fusion, dynamically calculated based on environmental parameters. The cross-modal state space model refers to a sequence processing network built based on a selective state space model. Hidden state interpenetration and interaction mean that during sequence scanning, each modality's state update relies not only on its own historical state but also receives input from the historical state of another modality. Defect category prediction determines the type of defect the currently detected target belongs to, and defect location regression outputs the bounding box coordinates of the defect in the image. In actual deployment, this method is executed according to the following steps.

[0023] S1, Data Acquisition. An industrial high-definition RGB camera and an infrared thermal imager are installed above the production line, both connected to the same hardware pulse trigger signal source. When the board arrives at the inspection station, the trigger source emits a synchronization pulse, and the two imaging devices simultaneously expose and acquire visible light and infrared thermal images of the same location on the same board. Both images carry the same timestamp. Environmental parameter data is read in real time from the production line control system, including the wood species of the current batch of boards, ambient temperature sensor readings, ambient humidity sensor readings, and production line operating speed values, and is stored in conjunction with the timestamp.

[0024] S2, a dual-path processing architecture for the large-scale model. One path downscales the visible light and infrared thermal images, then stitches them along the channel dimension before inputting them into the visual language large-scale model. The model performs forward inference on the input image, outputting a probabilistic semantic map corresponding to the input size. The value at each location in the map represents the probability of a defect at that location. Thresholding and connected component extraction are performed on this probabilistic map to obtain several rectangular coordinate boxes as regions of interest. The other path organizes four parameters—wood species, ambient temperature, ambient humidity, and production line speed—into a single input text, which is then input into the language large-scale model. Based on its learned knowledge of wood processing during training, the language large-scale model infers the imaging characteristics under the current working conditions, combining these parameters to generate a set of visual cue word features composed of continuous numerical values. This feature is input into a multilayer perceptron network. The network's endpoint is processed by a normalized exponential function, outputting two weight values ​​as the visible light mode weight and the infrared thermal mode weight, respectively.

[0025] S3, Local Feature Extraction. Using the region-of-interest (ROI) bounding boxes obtained in step S2, local image patches corresponding to each bounding box are cropped from the original high-resolution visible light image. Simultaneously, local image patches at the same locations are cropped from the original high-resolution infrared thermal image. Each local image patch is input into a residual network consisting of several convolutional layers and skip connections. The network performs layer-by-layer convolution and pooling operations on the image patch, outputting a local feature map. The height and width dimensions of the local feature map are merged and flattened into a one-dimensional sequence, where each element corresponds to a feature vector at a spatial location in the original feature map.

[0026] S4, Cross-modal fusion. The visible light local feature sequence and the infrared thermal local feature sequence are input into the cross-modal state space model. The model processes each position sequentially along the sequence. At each position, the current state of the visible light branch is obtained by adding two parts: one part is the result of a linear transformation of the previous position's state of the visible light branch, and the other part is the result of a linear transformation of the previous position's state of the infrared thermal branch multiplied by a penetration coefficient. Similarly, the current state of the infrared thermal branch is obtained by adding its own previous position's state transformation result and the result of the visible light branch's previous position's state transformation multiplied by the penetration coefficient. The value of the penetration coefficient is adjusted by the adaptive fusion weights generated in step S2. When the visible light modal weight is large, the penetration coefficient from infrared to visible light is increased. When the infrared thermal modal weight is large, the penetration coefficient from visible light to infrared thermal is increased. After processing the sequences of both branches, the output hidden state sequence is rearranged back into a two-dimensional feature map form, serving as the cross-modal fusion feature.

[0027] S5, Defect Detection Output. The cross-modal fusion features are input into the detection head, which is split into two paths. One path outputs the probability of each defect category via a fully connected layer. Defect categories include five types: dead knots, cracks, holes, mold, and no defects. The other path outputs the center point coordinates and width / height values ​​of the bounding box via the fully connected layer. The two outputs are merged into the final defect detection result.

[0028] In a preferred embodiment, the visible light image and the infrared thermal image are obtained by using an industrial high-definition RGB camera and an infrared thermal imager deployed on the production line, and by employing a hardware pulse synchronization triggering mechanism to collect the same material location at the same time stamp; After acquisition, the infrared thermal image is mapped to the spatial coordinate system of the visible light image through a perspective transformation matrix, and spatial alignment is achieved by bilinear interpolation resampling. The sampling frequency of the environmental parameter data is lower than the image acquisition frame rate. A time proximity window matching mechanism is used to align the environmental parameter data with the image data in the time dimension. Specifically, for each set of image timestamps, the environmental parameter data with the smallest absolute time deviation and a deviation not exceeding the preset maximum tolerable time delay threshold is retrieved from the timestamp sequence of the environmental parameter data, and it is bound to the current image frame pair.

[0029] Specifically, the spatial alignment process involves joint calibration of the industrial high-definition RGB camera and the infrared thermal imager before use. A calibration board with a checkerboard or dot matrix pattern is placed in the detection area, and both visible light and infrared thermal images of the calibration board are acquired simultaneously. Pixel coordinates of feature points on the calibration board are extracted from both images, forming several sets of coordinate correspondences. A 3x3 perspective transformation matrix is ​​calculated using these coordinate correspondences, with each element obtained by solving a system of linear equations. During detection, after each acquisition of an infrared thermal image, each pixel position is traversed, and its x and y coordinates are multiplied by the perspective transformation matrix to obtain its corresponding coordinates in the visible light image coordinate system. Since the calculated corresponding coordinates are often not integers, a weighted average of the four nearest integer pixel values ​​is calculated as the pixel value at that position. The weights of the four pixels are inversely proportional to their distances from the target coordinates; this is bilinear interpolation. After these operations, each pixel in the infrared thermal image corresponds to a pixel at the same spatial location in the visible light image.

[0030] The specific operation of time alignment is as follows: the sampling frequency of environmental parameter data is typically once per second or lower, while the image acquisition frame rate can reach tens of frames per second. When a set of images is acquired, the timestamp of that set of images is taken. The absolute value of the difference between the timestamp of each environmental parameter record and the timestamp of the image is calculated sequentially in the stored environmental parameter data sequence. The environmental parameter record with the smallest absolute difference is found. If this difference does not exceed a preset maximum tolerance time delay threshold, this environmental parameter record is associated and bound to the current image frame pair. The maximum tolerance time delay threshold is set according to the pipeline speed and detection accuracy requirements. If the minimum difference still exceeds the threshold, the environmental parameter record already bound to the previous frame is used.

[0031] In a preferred embodiment, the step of inputting the visible light image and the infrared thermal image into a lightweight visual language large model for global semantic analysis to obtain the region of interest for potential defects includes the following steps: After downsampling and reducing the resolution of the visible light image and the infrared thermal image, a global multimodal image tensor is constructed. The global multimodal image tensor is input into the visual encoder of the lightweight visual language large model, and the output is a probabilistic semantic map representing the possibility of defects. An adaptive threshold is used to perform threshold activation and connected component analysis on the probabilistic semantic graph to extract a set of coordinate boxes for potential defect regions of interest with high confidence.

[0032] Specifically, the downsampling and resolution reduction process involves dividing the original image into several grids after determining the target reduction ratio. Each grid corresponds to a pixel in the downsampled image. The arithmetic mean of the multiple original pixel values ​​contained in each grid is taken, and this average value is used as the pixel value at that location after downsampling. Visible light images and infrared thermal images use the same reduction ratio to ensure that the spatial dimensions of the two images are consistent after downsampling. The specific operation for constructing the global multimodal image tensor is as follows: visible light images typically contain three color channels (red, green, and blue), while infrared thermal images contain one temperature channel. The data from all four channels are concatenated and stacked along the channel dimension to form a multimodal tensor.

[0033] After inputting the multimodal tensor into the large-scale visual language model, the model employs multi-layered self-attention operations. Each image patch, when calculating its own features, considers information from all image patches globally to determine the presence of defects at each location. The model outputs a probabilistic semantic map by mapping the feature vector to a scalar value between zero and one for each spatial location in the last feature map layer; this value represents the defect probability at that location. The size of the output probabilistic semantic map is the same as the size of the downsampled input image.

[0034] The specific steps for extracting the region of interest (ROI) bounding boxes are as follows: First, determine the binarization threshold and calculate the average and standard deviation of all pixel values ​​in the probabilistic semantic map. Set the threshold to the average plus twice the standard deviation. Then, iterate through each pixel in the probabilistic semantic map, marking pixels with probability values ​​greater than the threshold as foreground and those less than or equal to the threshold as background. Next, perform connected component analysis on the foreground pixels using a two-pass scanning method. In the first pass, scan row by row and assign temporary connected component numbers to each foreground pixel. In the second pass, merge adjacent equivalent connected components and assign them unified numbers. Calculate the minimum bounding rectangle for each connected component, taking the minimum and maximum x-coordinates and y-coordinates of all foreground pixels in that connected component to form the left, right, top, and bottom boundaries of the rectangle. The sum of the bounding boxes of all connected components constitutes the ROI bounding box set.

[0035] In a preferred embodiment, the step of inputting the environmental parameter data into a large language model for semantic reasoning to generate adaptive fusion weights for the visible light mode and the infrared thermal mode under the current operating conditions includes the following steps: The environmental state vector is input into the prompt word embedding layer of the language big model; the environmental state vector is composed of the unique thermal code of the wood species, the standardized environmental temperature value, the environmental humidity value and the pipeline linear velocity value. The language model, combined with the built-in domain expert knowledge base, performs semantic reasoning on the texture artifact characteristics of wood species in the current production environment and the physical changes in infrared thermal imaging penetration under the current temperature and humidity, and outputs continuous visual cue word features related to the current working conditions. The visual cue word features are input into a weighted evolution network composed of a multilayer perceptron, and the visible light mode weights and infrared thermal mode weights are calculated using the Softmax activation function.

[0036] Specifically, the environmental state vector is composed of four concatenated components. The first component is the unique thermal code of the wood species. Unique thermal coding refers to using a vector of length equal to the total number of wood species to represent the current species, where only the bit corresponding to the current species is set to 1, and the rest are zero. The second component is the standardized ambient temperature value. Standardization involves subtracting the mean temperature from the original temperature value and then dividing by the standard deviation to bring parameters of different dimensions within a similar numerical range. The third component is the ambient humidity value, which is also standardized. The fourth component is the production line linear velocity value, which is also standardized. These four components are concatenated sequentially to form a single vector, thus obtaining the environmental state vector.

[0037] The environmental state vector is input into the cue word embedding layer of the language big data model. Specifically, each component of the environmental state vector is mapped to a corresponding word embedding vector. A word embedding vector is a representation that maps discrete or continuous numerical values ​​to a high-dimensional continuous vector space. During the pre-training phase, the language big data model has learned a wealth of professional knowledge in the wood processing field, including the surface texture features of different wood species and the physical property changes of infrared thermal imaging under different temperature and humidity conditions. Based on this built-in knowledge, the model performs layer-by-layer self-attention calculations and feedforward transformations on the environmental state vector, outputting a set of continuous numerical values ​​in the middle layer of the network. This set of numerical values ​​constitutes the visual cue word features. The visual cue word features typically have hundreds of dimensions, with each dimension encoding semantic information related to a specific aspect of the current working condition.

[0038] The visual cue word features are input into a weighted evolution network composed of a multilayer perceptron. The multilayer perceptron consists of several fully connected layers and activation functions stacked alternately. Each fully connected layer performs a linear transformation on the input vector, while the activation function introduces non-linearity. The last fully connected layer outputs two values, representing the original weight scores for the visible light mode and the infrared thermal mode, respectively. These two scores are then input into a Softmax activation function for normalization, yielding the normalized weight scores for the visible light mode and the normalized weight scores for the infrared thermal mode.

[0039] The technical principle of this solution lies in leveraging the cross-domain knowledge accumulated during pre-training of a large language model to transform discrete environmental parameters into continuous feature representations containing physical semantics. These representations are then mapped to specific modality fusion weights by a lightweight multilayer perceptron. The technical effect is the achievement of condition-adaptive adjustment of the fusion weights. When high ambient humidity leads to a decrease in infrared transmittance, the large language model can infer this change based on its physical knowledge and output corresponding visual cue word features, causing the weight evolution network to automatically reduce the weight of the infrared thermal mode.

[0040] In a preferred embodiment, the step of cropping local images at corresponding locations from the original resolution visible light image and infrared thermal image according to the region of interest, and extracting local feature sequences, includes the following steps: Based on the region of interest coordinate boxes generated by the large model, the original full-resolution visible light image and infrared thermal image are locally cropped with high definition to obtain visible light local images and infrared thermal local images. The visible light local image and the infrared thermal local image are respectively input into a shallow residual network to extract local feature maps with edge information; The local feature maps are flattened along the spatial dimension into a one-dimensional sequence to obtain visible light local feature sequences and infrared thermal local feature sequences, respectively.

[0041] Specifically, the operation of local high-resolution cropping using the region of interest (ROI) coordinate boxes generated by the large model is as follows: Each coordinate box in the ROI coordinate box set is traversed, and its left, top, right, and bottom boundary coordinates are obtained. On the original full-resolution visible light image, a rectangular cropping region is defined starting from the left and top boundaries, with the width equal to the difference between the right and left boundaries, and the height equal to the difference between the bottom and top boundaries. The pixel data within this region is copied and extracted to form a visible light local image. On the original full-resolution infrared thermal image, the same rectangular cropping region is defined using the exact same four coordinate values. The pixel data within this region is copied and extracted to form an infrared thermal local image. If the coordinate box exceeds the image boundary, the excess portion is filled with the pixel values ​​of the image edge.

[0042] Local visible light images and infrared thermal images are input into a shallow residual network. The shallow residual network consists of several residual blocks connected in series. Each residual block contains two convolutional layers and one skip connection. The convolutional layers operate by using several learnable convolutional kernels to perform sliding window-style inner product operations on the input feature maps. Each kernel extracts a local feature pattern and outputs a corresponding feature map channel. The skip connection directly adds the input of the residual block to the output of the last convolutional layer within that block, and then performs a non-linear transformation through an activation function before outputting the final image. Skip connections allow gradients to flow directly through the network during backpropagation training, avoiding the vanishing gradient problem common in deep networks. After layer-by-layer processing through several residual blocks, the network outputs local feature maps.

[0043] The specific operation of flattening a local feature map into a one-dimensional sequence along the spatial dimensions is as follows: The local feature map is a three-dimensional tensor containing three dimensions: height, width, and number of channels. Iterates through each spatial position in the height and width dimensions in a top-to-bottom, left-to-right order, forming a feature vector from the feature values ​​of all channels corresponding to each spatial position. Arrange the feature vectors of all spatial positions sequentially according to the traversal order to form a one-dimensional sequence. The length of the sequence equals the height multiplied by the width, and the dimension of each element in the sequence equals the number of channels.

[0044] The technical principle of this solution lies in concentrating the computational burden of fine feature extraction in local suspicious areas using a region-of-interest (ROI) bounding box. A shallow residual network extracts local features rich in edge and texture information, and a spatial flattening operation converts the two-dimensional image features into a format suitable for sequence model processing. The technical advantage is that it avoids the enormous computational overhead of full-image high-resolution feature extraction while preserving high-frequency detail information in the defective local areas.

[0045] In a preferred embodiment, the cross-modal state space model is a cross-modal interactive scanning model constructed based on a selective state space model. The hidden states of the two modalities interpenetrate and interact to obtain cross-modal fusion features, which is achieved in the following way: At each time step of the sequence scan, the current hidden state of the visible light mode is determined by the following inputs: the result of the visible light mode's previous hidden state after transformation by the visible light state space matrix and the input control matrix, and the input of the infrared thermal mode's previous hidden state after adjustment by the infrared thermal state space matrix, the infrared thermal input control matrix, and the cross-modal information penetration coefficient. The current hidden state of the infrared thermal mode is determined by the following inputs: the result of the transformation of the infrared thermal state space matrix and the input control matrix of the previous hidden state of the infrared thermal mode itself, and the input of the previous hidden state of the visible light mode after adjustment by the visible light state space matrix, the visible light input control matrix and the cross-modal information penetration coefficient. The cross-modal information penetration coefficient is obtained by dynamic mapping of the cross-correlation characteristics of the two sets of modes at the current moment.

[0046] Specifically, the cross-modal state-space model is built upon the selective state-space model. The selective state-space model is a neural network model for processing sequential data. Its core idea is to maintain a hidden state vector at each time step. This hidden state vector is updated based on the current input and the hidden state at the previous time step, and the direction and magnitude of the update are controlled by the state-space matrix and the input control matrix. This scheme introduces a cross-modal interaction mechanism, enabling the hidden states of two modalities to cross-transfer information during sequence scanning.

[0047] In actual operation, the model scans and processes sequentially from the first time step to the last. At each time step, the current hidden state of the visible light mode is obtained by adding two update variables. The first update variable is the result of linearly transforming the hidden state of the visible light mode itself in the previous time step through the visible light state space matrix and the visible light input control matrix. This update variable reflects the temporal evolution of the visible light mode's own state. The second update variable is the hidden state of the infrared thermal mode in the previous time step. It is first linearly transformed through the infrared thermal state space matrix and the infrared thermal input control matrix, and then the transformation result is multiplied by the cross-modal information penetration coefficient. This update variable reflects the amount of information transferred from the infrared thermal mode to the visible light mode.

[0048] The current hidden state of the infrared thermal mode is also obtained by adding two update variables. The first update variable is the result of linearly transforming the hidden state of the infrared thermal mode itself in the previous time step through the infrared thermal state space matrix and the infrared thermal input control matrix. The second update variable is the hidden state of the visible light mode in the previous time step, which is first linearly transformed through the visible light state space matrix and the visible light input control matrix, and then the transformation result is multiplied by the cross-modal information penetration coefficient.

[0049] All the state space matrices and input control matrices mentioned above are learnable parameter matrices, updated during model training via gradient backpropagation. The cross-modal information penetration coefficient is dynamically mapped from the cross-correlation features of the two modalities at the current time. Specifically, the input features of the visible light mode and the infrared thermal mode at the current time are taken, the cross-correlation matrix between them is calculated, the cross-correlation matrix is ​​mapped to a scalar value through a fully connected layer, and then transformed into the cross-modal information penetration coefficient through an activation function.

[0050] The technical principle of this solution lies in enabling the internal density and temperature anomaly information carried by the infrared thermal mode to directly permeate into the feature update process of the visible light mode through cross-modal hidden state transfer. Simultaneously, the surface texture information carried by the visible light mode can also permeate into the feature update process of the infrared thermal mode. The two modes continuously exchange information at each time step of the sequence scan, achieving deep complementarity. The technical effect is that when the visible light image is interfered with by surface water stains or dust, the infrared thermal mode can provide additional internal structural information through hidden state transfer, correcting erroneous features in the visible light branch and improving the robustness and expressive power of the fused features.

[0051] In another embodiment of this example, to further enhance the refinement of environmental perception, the visual cue word features generated by the large language model are not only used to calculate adaptive fusion weights, but also injected as conditional information into each time step of the cross-modal state space model. Specifically, the visual cue word features are transformed through a fully connected layer to align with the hidden state dimension of the cross-modal state space model, resulting in a conditional injection vector. At each time step of the sequence scan, the current hidden state of the visible light modality is updated by two additional updates, plus a third update based on the conditional injection vector mapped by the visible light conditional transformation matrix. Similarly, the current hidden state of the infrared thermal modality is also updated by the third update based on the conditional injection vector mapped by the infrared thermal conditional transformation matrix. Through this dual injection mechanism, the environmental perception results both adjust the information penetration between modalities at the macroscopic level and directly participate in the hidden state update at each sequence position at the microscopic level, enabling the fusion features to possess more refined adaptive capabilities. During model training, the visible light conditional transformation matrix and the infrared thermal conditional transformation matrix are used as learnable parameters and optimized end-to-end along with other model parameters.

[0052] In a preferred embodiment, the cross-modal information penetration coefficient is further dynamically adjusted by the adaptive fusion weights, specifically as follows: The adaptive fusion weights include visible light mode weights and infrared thermal mode weights; The visible light mode weights are fed forward to the cross-modal state space model to adjust the magnitude of the cross-modal information penetration coefficient when the infrared thermal mode hidden state penetrates into the visible light mode; The infrared thermal mode weights are fed forward to the cross-modal state space model to adjust the magnitude of the cross-modal information penetration coefficient when the visible light mode hidden state penetrates into the infrared thermal mode.

[0053] In practical implementation, cross-modal information permeation exists in two directions within the cross-modal state-space model. One direction is the permeation of the hidden states of the infrared thermal mode into the visible light mode, the degree of which is controlled by a first permeation coefficient. The other direction is the permeation of the hidden states of the visible light mode into the infrared thermal mode, the degree of which is controlled by a second permeation coefficient. The cross-modal information permeation coefficient includes the permeation coefficients of both directions. Both are initially obtained by dynamically mapping the cross-correlation features of the two sets of modes at the current time, and then adjusted by adaptive fusion weights to obtain the final value.

[0054] The visible light mode weights and infrared thermal mode weights are derived from the output of the large language model. A higher visible light mode weight indicates higher reliability of the visible light image under the current conditions, while infrared thermal imaging may be affected by factors such as environmental humidity, leading to a decrease in the signal-to-noise ratio. In this case, the system increases the penetration coefficient when the infrared thermal mode penetrates into the visible light mode, allowing the visible light mode to fully absorb the still effective internal density information from the infrared thermal mode. Even if the overall quality of the infrared thermal imaging decreases, the effective information can still be extracted and utilized. Simultaneously, the system decreases the penetration coefficient when the visible light mode penetrates into the infrared thermal mode to avoid excessive transmission of noise or artifact information from the visible light mode to the infrared thermal mode.

[0055] When the infrared thermal mode weight is high, it indicates that the reliability of the infrared thermal imaging is high under the current operating conditions, while the visible light image may be interfered with by factors such as water stains and texture artifacts on the wood surface. In this case, the system increases the penetration coefficient when the visible light mode penetrates into the infrared thermal mode, while simultaneously decreasing the penetration coefficient when the infrared thermal mode penetrates into the visible light mode. Through this asymmetric bidirectional adjustment mechanism, the degree of information penetration between the two modes is dynamically balanced according to their respective reliability. The mode with higher reliability transmits more information to the mode with lower reliability, while the mode with lower reliability transmits less information to the mode with higher reliability.

[0056] The specific operation of weight feedforward to the cross-modal state space model is as follows: at each time step of model inference, the initial values ​​of the two penetration coefficients are multiplied by their respective modal weight values ​​to obtain the adjusted penetration coefficients. Then, the adjusted penetration coefficients are substituted into the hidden state update formula for calculation.

[0057] The technical principle of this solution lies in utilizing the modal credibility assessment results generated by the large-model environment-aware agent to achieve asymmetric cross-modal information flow control at the micro level of the state-space model. The technical effect is that it enables the cross-modal interaction process to have adaptive capabilities under different operating conditions, automatically adjusting the direction and intensity of information penetration in different production environments, thus improving the flexibility and robustness of the fusion strategy.

[0058] In a preferred embodiment, obtaining the cross-modal fusion feature includes the following steps: The hidden state sequence output after scanning the cross-modal state space model is remapped back to two-dimensional space to obtain visible light fusion feature map and infrared thermal fusion feature map. The visible light fusion feature map and the infrared thermal fusion feature map are fed into a multi-scale attention fusion component. The multi-scale attention fusion component uses dilated convolution with different receptive fields and spatial attention mechanism to further fuse the probabilistic semantic map representing the possibility of defects output by the large model with the local defect features extracted by the cross-modal state space model, and outputs a fusion feature tensor.

[0059] Specifically, after the cross-modal state space model scan is completed, the output consists of two one-dimensional hidden state sequences, corresponding to the visible light branch and the infrared thermal branch, respectively. The specific operation for remapping the hidden state sequences back to two-dimensional space is as follows: Take the hidden state sequence of the visible light branch; the length of this sequence is equal to the length of the sequence after flattening the local feature map. According to the spatial position correspondence before flattening, place the feature vector of each position in the sequence back into its original spatial position in the feature map. That is, flattening is done by traversing from top to bottom and from left to right; when mapping back to two dimensions, the sequence elements are filled back into the height and width dimensions row by row and column by column in the same order, restoring the visible light fused feature map. Perform the same operation on the hidden state sequence of the infrared thermal branch to restore the infrared thermal fused feature map.

[0060] The visible light fused feature map and the infrared thermal fused feature map are fed into a multi-scale attention fusion component for further fusion. The specific structure of the multi-scale attention fusion component is as follows: First, the two fused feature maps are concatenated along the channel dimension to obtain a concatenated feature map. Then, the concatenated feature map is fed into three parallel dilated convolution branches with different dilation rates (the spacing between sampling points of the convolution kernel). The branch with a dilation rate of one uses ordinary convolution, resulting in a smaller receptive field and extracting local detail features. The branch with a dilation rate of three has a sampling point spacing of two, resulting in a medium receptive field and extracting medium-range contextual features. The branch with a dilation rate of five has a sampling point spacing of four, resulting in a larger receptive field and extracting a large range of global features. The outputs of the three branches are concatenated again along the channel dimension and then fused through an ordinary convolutional layer to obtain the multi-scale feature map.

[0061] Next, a spatial attention mechanism is applied to the multi-scale feature map. The spatial attention is calculated by performing global average pooling and global max pooling along the channel dimension of the multi-scale feature map, resulting in two single-channel spatial description maps. These two single-channel maps are concatenated along the channel dimension, then passed through a convolutional layer and activation function transformation to output a spatial attention weight map, where the value at each position is between zero and one. The multi-scale feature map and the spatial attention weight map are multiplied channel-by-channel and spatially by position to obtain a weighted multi-scale feature map. Regions with high attention weights are enhanced, while regions with low weights are suppressed.

[0062] Finally, the probabilistic semantic map output by the large model is upsampled to make its spatial size similar to that of the weighted multi-scale features. Figure 1 Upsampling is achieved through bilinear interpolation. The upsampled probabilistic semantic map is appended as an additional channel to the weighted multi-scale feature map, and then fused through a convolutional layer to output the final fused feature tensor. This fused feature tensor contains both global semantic guidance information from the large model and local multimodal detail information extracted from the cross-modal state space model.

[0063] The technical principle of this solution lies in capturing contextual information of different ranges through multi-scale dilated convolution, adaptively enhancing the response of defect regions through spatial attention mechanisms, and deeply fusing high-level semantic guidance from the large model with low-level features through probabilistic semantic map backfilling. The technical effect is that the fused feature tensor possesses both global semantic understanding and local detail resolution capabilities, which is beneficial for the subsequent detection head to accurately classify and locate defects.

[0064] In a preferred embodiment, the defect category prediction and defect location regression based on the cross-modal fusion features, and the output of defect detection results, are achieved through a multi-task detection head; The multi-task detection head includes a classification branch and a regression branch; the classification branch takes the cross-modal fusion features as input and outputs a probability distribution prediction of the board defect category; the regression branch takes the cross-modal fusion features as input and outputs the geometric parameters of the defect bounding box; the defect categories include dead knots, cracks, holes, mold, and no defects; During model training, a joint loss function is used for end-to-end training. The joint loss function is composed of a weighted sum of classification cross-entropy loss, regression mean squared error loss, and cross-modal alignment loss. The cross-modal alignment loss is the alignment loss between the semantic features generated by the large model and the cross-modal fused features.

[0065] In implementation, the multi-task detection head includes a classification branch and a regression branch. Both branches share the same cross-modal fusion feature as input, but each has independent network parameters. The classification branch consists of several stacked fully connected layers, with the output dimension of the last fully connected layer equal to the total number of defect categories. The defect categories include five types: dead knots, cracks, holes, mold, and no defects. A Softmax activation function is applied to the output of the last fully connected layer, converting each element in the output vector into a corresponding category probability value; the sum of all probability values ​​is one. The category with the highest probability value is taken as the classification prediction result. The regression branch also consists of several stacked fully connected layers, with the output dimension of the last fully connected layer being four, corresponding to the x-coordinate of the bounding box's center point, the y-coordinate of the center point, the width of the bounding box, and the height of the bounding box. The output of the regression branch is a continuous numerical value without undergoing an activation function.

[0066] Model training is performed offline, requiring a pre-labeled training dataset beforehand. Each training sample includes a visible light image, an infrared thermal image, environmental parameter data, and manually labeled defect categories and ground truth bounding box coordinates. During training, a joint loss function is used for end-to-end optimization; this joint loss function consists of a weighted sum of three parts.

[0067] The first part is the classification cross-entropy loss, which measures the difference between the class probability distribution output by the classification branch and the true class label. It is calculated by taking the negative natural logarithm of the predicted probability value corresponding to the true class and using this value as the classification loss. The smaller this loss, the more accurate the classification prediction.

[0068] The second part is the regression mean squared error loss, which measures the difference between the bounding box coordinates output by the regression branch and the true bounding box coordinates. It is calculated by squared the difference between the predicted and true values ​​for each of the four coordinate parameters of the bounding box, and then averaging the four squared values. A smaller loss indicates more accurate localization.

[0069] The third part is the cross-modal alignment loss, which ensures the consistency between the semantic features generated by the large model and the cross-modal fused features in the feature space. Specifically, the semantic feature map output from the intermediate layer of the large model's visual encoder and the output fused feature tensor are both subjected to global average pooling to obtain two feature vectors. The cosine similarity between the two feature vectors is calculated; the closer the cosine similarity is to one, the more consistent the directions of the two feature vectors. The cross-modal alignment loss is defined as a negative cosine similarity or mean squared error. Minimizing this loss ensures that the fused features do not deviate from the high-level semantic guidance of the large model.

[0070] The three loss components are multiplied by preset balancing coefficients and then summed to obtain the total loss value. The balancing coefficients are used to adjust the relative importance of the three loss components during training. The gradient of the total loss with respect to all learnable parameters of the model is calculated using the gradient backpropagation algorithm, and the optimizer updates the parameter values ​​based on the gradients. This process is repeated iteratively multiple times on the training dataset until the total loss converges to a stable value, at which point model training is complete.

[0071] The technical principle of this solution lies in achieving knowledge transfer between classification and regression tasks through multi-task shared feature representation. It uses cross-modal alignment loss in the joint loss function to constrain the consistency between the underlying fused features and the high-level semantic features, enabling the model to simultaneously optimize detection accuracy and semantic alignment during training. The technical effect is that the trained model can accurately determine the defect category and precisely output the defect location, and the fused features will not deviate from the semantic understanding of the larger model due to data bias during training.

[0072] In a preferred embodiment, a real-time detection step is also included: Visible light image data, infrared thermal image data, and environmental parameter data of the board material are collected on-site simultaneously, or historical data stored in a multimodal database are retrieved for spatiotemporal alignment preprocessing. The preprocessed data is input into the trained and converged defect detection model to perform forward inference and output the defect detection results, which include defect category, location coordinates and confidence level. The defect detection results are used to generate a diagnostic report and store it as text data. This report is then uploaded to the production line quality traceability database as a test record for this batch of boards and reserved as backup data for subsequent model fine-tuning and knowledge base updates.

[0073] In practice, data acquisition takes two approaches. The first is on-site synchronous acquisition, utilizing a hardware pulse synchronization triggering mechanism. Industrial high-definition RGB cameras and infrared thermal imagers capture real-time images of the boards currently passing through the inspection station, while simultaneously reading the wood species, ambient temperature, ambient humidity, and production line speed of the current batch of boards from the production line control system. The second approach involves retrieving historical data stored in a multimodal database. This multimodal database refers to a data storage system deployed on the production line server or edge computing device, which stores previously acquired visible light images, infrared thermal images, and environmental parameter data by timestamp and batch number, for use in quality traceability or offline re-inspection.

[0074] After acquiring the data, it is preprocessed for spatiotemporal alignment according to spatial and temporal alignment methods to ensure that the visible light image, infrared thermal image and environmental parameter data are strictly corresponding in spatial and temporal dimensions.

[0075] The preprocessed data is input into the trained and converged defect detection model for forward inference. "Trained and converged" means the model has completed end-to-end training, the joint loss function has converged to a stable value, and the learnable parameters of each layer are fixed. During forward inference, the model executes the steps in the aforementioned order and outputs the defect detection results. The entire forward inference process does not involve gradient calculation or parameter updates; it only calculates layer by layer along the forward direction of the network until the output layer.

[0076] The defect detection result output by the model includes three pieces of information. The first is the defect category, which is the category with the highest probability value output by the classification branch. The second is the location coordinates, namely the x-coordinate, y-coordinate, width, and height of the center point of the bounding box. The third is the confidence score, which is the probability value corresponding to the defect category, ranging from zero to one.

[0077] The defect detection results are used to generate a diagnostic report and stored as text data. The report is generated by organizing the defect category name, location coordinates, confidence level, batch number of the current board material, and inspection timestamp according to a preset text template format, which can be JSON, XML, or plain text table format. This text data is then uploaded to the production line quality traceability database via a database connection interface. The batch number and inspection timestamp are used as the combined index key during upload to ensure that each inspection record is uniquely retrieved in the database. This inspection record is permanently stored as a daily inspection and maintenance record for that batch of boards.

[0078] The detection records are also retained as backup data for subsequent model fine-tuning and knowledge base updates. Model fine-tuning refers to adding manually verified samples from newly accumulated detection records to the training dataset after the model has been deployed and run for a period of time, and then performing a small number of iterative training iterations to adapt the model to newly emerging wood species or defect types. Knowledge base updates refer to supplementing the newly discovered defect feature patterns and corresponding environmental condition information into the domain expert knowledge base built into the language model, expanding the model's cognitive scope of the production environment, and improving the accuracy of subsequent adaptive fusion weight generation.

[0079] Example 2 This embodiment proposes a multimodal fusion plate defect detection system based on a large model, such as... Figure 2 As shown, the system used to implement the method described in Embodiment 1 mainly consists of four core modules: a multimodal data acquisition and environmental perception module, a large model environmental perception agent and multimodal adaptive prompting and guidance module, a Mamba feature dynamic fusion module based on cross-modal interaction, a defect multi-task detection and network training module, and a defect real-time detection module.

[0080] The specific steps and plan are as follows: First, there is the multimodal data acquisition and environmental perception module: This module is primarily used for real-time acquisition, preprocessing, and multi-dimensional alignment of multimodal information of sheet materials on the production line. It includes real-time synchronous acquisition and strict spatiotemporal alignment of visible light image data, infrared thermal image data, and environmentally perceived text data. The steps it performs are as follows: (1) Multimodal image acquisition of the sheet material: Employing a synchronous triggering mechanism, visible light images of the same board material location are simultaneously acquired using industrial high-definition RGB cameras and infrared thermal imagers deployed on the production line. and infrared thermal imaging images The acquired images undergo preprocessing, primarily including image enhancement, position correction, and image resizing. The preprocessed data from both types are then rigorously aligned in both temporal and spatial dimensions.

[0081] (2) Environmental and parameter perception: Collect real-time environmental data and material parameters of the board production line, including wood species (such as pine, birch, and oak), ambient temperature, humidity, and production line operating speed, and encode them into environmental status text vectors:

[0082] in, Unique thermal code representing wood species, For standardized ambient temperature values, For ambient humidity, The linear velocity of the assembly line.

[0083] (3) Multimodal synchronous alignment acquisition: In order to fuse high-frequency image feature streams with low-frequency environmental feature vectors in the same computational stream, this step implements joint spatial and temporal alignment.

[0084] Strict alignment of spatial dimensions: Employing a high-precision hardware pulse synchronization triggering mechanism, an industrial high-definition RGB camera and an infrared thermal imager deployed directly above the conveyor line are used to synchronize the pulses at the same timestamp. Collect visible light images of the same spatial area of ​​the board. With infrared thermal images Through perspective transformation matrix By mapping infrared thermal images to the spatial coordinate system of visible light images, strict spatiotemporal alignment is achieved.

[0085] in, This represents the bilinear interpolation resampling space mapping operator.

[0086] Time-dimension multi-rate cascade alignment: To address the issue of inconsistency between the environmental state vector sampling frequency and the image acquisition frame rate (i.e., the image timestamp sequence) The density is much greater than that of environmental timestamp sequences. This step introduces a temporal proximity window matching mechanism. For any set of image timestamps... By minimizing the absolute time deviation, the nearest operating condition is retrieved in the environmental timestamp sequence:

[0087] Let the aligned environment state vector be When satisfied ( When the maximum tolerable time delay threshold for industry is set, Dynamically bind to the current image frame pair.

[0088] Finally, this module outputs a timestamp. Strictly spatiotemporally aligned three-modal industrial inspection data This information is then used as a unified input for pipeline defect feature mining and fed into the subsequent large model's initial screening layer.

[0089] Secondly, the large-model environment-aware agent and multimodal adaptive prompting and guidance module: This module introduces a lightweight visual language model (VLM) and a large language model (LLM) as a context agent to achieve the collaborative generation of global semantic screening and adaptive modality weights. The steps it performs are as follows: (1) Global semantic screening and region of interest mapping: Strictly spatiotemporally aligned image pairs and Perform downsampling and resolution reduction processing to construct a global multimodal image tensor. The input is then fed into the visual encoder of the Lightweight Visual Language Model (VLM), leveraging the VLM's generalization and understanding capabilities in open domains for global coarse screening, and outputting a probabilistic semantic map representing the likelihood of defects. Subsequently, an adaptive segmentation threshold was set. Extracting the coordinates of the Region of Interest (RoI) with high confidence for potential defects:

[0090] in, For threshold activation and connected component analysis functions, This step involves creating a set of bounding box coordinates that contain potential defects. Through this process, large models can quickly filter out defect-free, normal wood background areas, significantly reducing the computational cost of subsequent high-frequency detail feature extraction.

[0091] (2) Adaptive modal weight evolution based on environment agent: Align the environment state vector The input is the prompt word embedding layer of the Large Language Model (LLM). The LLM, combined with a built-in domain expert knowledge base, performs semantic reasoning on the current physical production environment (such as the texture artifact characteristics of specific wood species, and the physical changes in thermal infrared transmittance under high humidity), outputting continuous visual prompt word features highly relevant to the current working conditions. .

[0092] Then, The input is fed into a weight evolution network composed of a multilayer perceptron (MLP), activated by the Softmax function, to dynamically calculate the sensitivity weight vectors of the two image modalities under the current operating conditions. :

[0093] in, and For learnable weight matrices and biases, and These represent the visible light mode weights and the infrared thermal mode weights, respectively.

[0094] Calculated dynamic weights This information will be directly fed forward to the subsequent feature dynamic fusion module. This mechanism establishes at the underlying level that the detection system can adaptively adjust its reliance on visible light and infrared thermal information in response to the natural fluctuations in physical parameters such as wood material, temperature, humidity, and linear speed of the production line, effectively shielding against single-mode interference caused by sudden changes in the industrial environment.

[0095] Secondly, a dynamic feature fusion module based on cross-modal interaction of Mamba features: This module is the core fusion network of the system, such as Figure 3 As shown, this paper abandons the traditional simple cascading or attention fusion approach and introduces a selective state-space model (Mamba). It also designs a cross-modal interactive scanning mechanism (Cross-modalScan) to achieve deep complementarity between texture information and internal density information during the discretization process of the continuous state-space model. The specific steps it performs are as follows: Step 1: High-frequency fine local feature extraction: Coordinates generated based on the large model For the original full resolution and High-resolution cropping is performed on local areas, and each crop is fed into a shallow residual network to extract local feature maps with rich edge information. Flatten it along the spatial dimension into a one-dimensional sequence form: ,in .

[0096] Step 2: Discretization scan of the cross-modal state-space model: Based on the standard Mamba sequence scan, cross-modal hidden state transmission is introduced. Let... and Visible light and infrared thermal sequences respectively The hidden state at any given time, and the cross-modal interaction update rules are defined as follows:

[0097] in, The state space matrix; and This is the cross-modal information penetration coefficient, dynamically calculated from the feature interactions of the previous layer. Among them, , These represent the hidden states of visible light and infrared thermal modes in a continuous spatial dimension, respectively. , This is the system state transition matrix; , For input control matrix; and The cross-modal cooperative penetration parameters are derived from two sets of modes in The cross-correlation characteristics at time points are dynamically mapped.

[0098] To apply continuous systems to discrete image sequences, a time step parameter is introduced. , The system is discretized in first order using a zero-order hold:

[0099] The formula for cross-modal interaction iteration in discrete time is derived as follows:

[0100] Through this interaction equation, the internal density / moisture abrupt change information carried by infrared thermal features can be hidden in a linear time-series scan. The form directly corrects the hidden state of the visible light channel caused by surface artifacts (water stains, dust), thereby realizing the deep interweaving of multimodal features at the bottom layer.

[0101] Step 3: Multi-scale spatial reconstruction and fusion: The hidden state sequence after cross-modal scanning is remapped back to two-dimensional space to obtain... and The data is then fed into a multi-scale attention fusion component. By utilizing dilated convolutions with different receptive fields and spatial attention mechanisms, the output of the large model is transformed into a probabilistic semantic map representing the likelihood of defects. Local high-frequency defect texture information extracted based on cross-modal interaction Mamba , Further fusion results in a final output of a fusion feature tensor with highly rich content features. .

[0102] Secondly, there is the defect multi-task detection and network training module: This module utilizes a multi-task awareness head to fuse features. It performs high-precision feature extraction and detection classification, and completes offline training using a joint loss function. The specific steps it takes are as follows: (1) Defect multi-task fine prediction output: Fusion features Input to downstream multi-task inspection head. Classification branch outputs probability distribution predictions of sheet metal defects (covering dead knots, cracks, holes, mold, and no defects). The regression branch outputs detailed defect bounding box geometry parameters. .

[0103] (2) Multi-task joint loss function and model training: End-to-end training is performed using a joint loss function, combining the cross-entropy loss (CE) for classification and the mean squared error (MSE) loss for regression:

[0104] in, This is the balance coefficient; Cross-modal alignment loss between semantic features generated for large models and fused features is used to ensure that low-level visual features do not deviate from high-level semantic guidance.

[0105] The detection bounding boxes, defect categories, and confidence levels obtained from the integrated model architecture detection are used to output the final detection report, which is then stored in the database in text format.

[0106] Finally, there is the real-time defect detection module: this module is a deployed, trained, and converged board defect detection model. This module collects three types of data from the board simultaneously on-site (visible light image data, infrared thermal image data, and environmentally perceptible text data), or retrieves historical data collected daily and stored in a multimodal database, performing rigorous spatiotemporal alignment and other preprocessing. Subsequently, the processed unified data is input into the real-time defect detection module to perform forward inference and real-time defect detection. Finally, the defect detection results (including defect category, location coordinates, and confidence level) are used to generate a diagnostic report and stored as text data. This report is then uploaded to the production line quality traceability database as a daily inspection and maintenance record for this batch of boards, and is also retained as backup data for subsequent model fine-tuning and knowledge base updates.

[0107] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for detecting defects in sheet metal based on a large model-guided multimodal fusion, characterized in that, Includes the following steps: Acquire visible light images, infrared thermal images, and environmental parameter data of the board material at the same spatial location and time stamp; The visible light image and the infrared thermal image are input into a lightweight visual language model for global semantic analysis to obtain the region of interest of potential defects; and the environmental parameter data are input into the language model for semantic reasoning to generate adaptive fusion weights for the visible light mode and the infrared thermal mode under the current working conditions. Based on the region of interest, local images at corresponding positions are cropped from the original resolution visible light image and infrared thermal image, and local feature sequences are extracted. The local feature sequences of the visible light mode and the infrared thermal mode are input into the cross-modal state space model. During the sequence scanning process, the hidden states of the two modes are allowed to interpenetrate and interact to obtain cross-modal fusion features. The cross-modal state space model is guided by the adaptive fusion weights during the interaction process. Based on the cross-modal fusion features, defect category prediction and defect location regression are performed, and defect detection results are output. The step of inputting the environmental parameter data into a language model for semantic reasoning to generate adaptive fusion weights for the visible light mode and the infrared thermal mode under the current operating conditions includes the following steps: The environmental state vector is input into the prompt word embedding layer of the language big model; the environmental state vector is composed of the unique thermal code of the wood species, the standardized environmental temperature value, the environmental humidity value and the pipeline linear velocity value. The language model, combined with the built-in domain expert knowledge base, performs semantic reasoning on the texture artifact characteristics of wood species in the current production environment and the physical changes in infrared thermal imaging penetration under the current temperature and humidity, and outputs continuous visual cue word features related to the current working conditions. The visual cue word features are input into a weighted evolution network composed of a multilayer perceptron, and the visible light mode weights and infrared thermal mode weights are calculated using the Softmax activation function.

2. The method for detecting defects in sheet metal based on large model-guided multimodal fusion according to claim 1, characterized in that, The visible light image and the infrared thermal image are obtained by using an industrial high-definition RGB camera and an infrared thermal imager deployed on the production line, and by employing a hardware pulse synchronization triggering mechanism to collect the same material position at the same time stamp. After acquisition, the infrared thermal image is mapped to the spatial coordinate system of the visible light image through a perspective transformation matrix, and spatial alignment is achieved by bilinear interpolation resampling. The sampling frequency of the environmental parameter data is lower than the image acquisition frame rate. A time proximity window matching mechanism is used to align the environmental parameter data with the image data in the time dimension. Specifically, for each set of image timestamps, the environmental parameter data with the smallest absolute time deviation and a deviation not exceeding the preset maximum tolerable time delay threshold is retrieved from the timestamp sequence of the environmental parameter data and matched to the current image frame pair.

3. The method for detecting defects in sheet metal based on large model-guided multimodal fusion according to claim 1, characterized in that, The step of inputting the visible light image and the infrared thermal image into a lightweight visual language large model for global semantic analysis to obtain the region of interest for potential defects includes the following steps: After downsampling and reducing the resolution of the visible light image and the infrared thermal image, a global multimodal image tensor is constructed. The global multimodal image tensor is input into the visual encoder of the lightweight visual language large model, and the output is a probabilistic semantic map representing the possibility of defects. An adaptive threshold is used to perform threshold activation and connected component analysis on the probabilistic semantic graph to extract a set of coordinate boxes for potential defect regions of interest with high confidence.

4. The method for detecting defects in sheet metal based on large model-guided multimodal fusion according to claim 1, characterized in that, The step of cropping local images at corresponding locations from the original resolution visible light image and infrared thermal image based on the region of interest, and extracting local feature sequences, includes the following steps: Based on the region of interest coordinate boxes generated by the large model, local high-resolution cropping is performed on the original full-resolution visible light image and infrared thermal image to obtain visible light local image and infrared thermal local image respectively; The visible light local image and the infrared thermal local image are respectively input into a shallow residual network to extract local feature maps with edge information; The local feature maps are flattened along the spatial dimension into a one-dimensional sequence to obtain visible light local feature sequences and infrared thermal local feature sequences, respectively.

5. The method for detecting defects in sheet metal based on large model-guided multimodal fusion according to claim 1, characterized in that, The cross-modal state space model is a cross-modal interactive scanning model constructed based on the selective state space model. The hidden states of the two modalities interpenetrate and interact with each other to obtain cross-modal fusion features, which is achieved in the following way: At each time step of the sequence scan, the current hidden state of the visible light mode is determined by the following inputs: the result of the visible light mode's previous hidden state after transformation by the visible light state space matrix and the input control matrix, and the input of the infrared thermal mode's previous hidden state after adjustment by the infrared thermal state space matrix, the infrared thermal input control matrix, and the cross-modal information penetration coefficient. The current hidden state of the infrared thermal mode is determined by the following inputs: the result of the transformation of the infrared thermal state space matrix and the input control matrix of the previous hidden state of the infrared thermal mode itself, and the input of the previous hidden state of the visible light mode after adjustment by the visible light state space matrix, the visible light input control matrix and the cross-modal information penetration coefficient. The cross-modal information penetration coefficient is obtained by dynamic mapping of the cross-correlation characteristics of the two sets of modes at the current moment.

6. The method for detecting defects in sheet metal based on large model-guided multimodal fusion according to claim 5, characterized in that, The cross-modal information penetration coefficient is also dynamically adjusted by the adaptive fusion weights, specifically as follows: The adaptive fusion weights include visible light mode weights and infrared thermal mode weights; The visible light mode weights are fed forward to the cross-modal state space model to adjust the magnitude of the cross-modal information penetration coefficient when the infrared thermal mode hidden state penetrates into the visible light mode; The infrared thermal mode weights are fed forward to the cross-modal state space model to adjust the magnitude of the cross-modal information penetration coefficient when the visible light mode hidden state penetrates into the infrared thermal mode.

7. The method for detecting defects in sheet metal based on large model-guided multimodal fusion according to claim 1, characterized in that, The process of obtaining cross-modal fusion features includes the following steps: The hidden state sequence output after scanning the cross-modal state space model is remapped back to two-dimensional space to obtain visible light fusion feature map and infrared thermal fusion feature map. The visible light fusion feature map and the infrared thermal fusion feature map are fed into a multi-scale attention fusion component. The multi-scale attention fusion component uses dilated convolution with different receptive fields and spatial attention mechanism to further fuse the probabilistic semantic map representing the possibility of defects output by the large model with the local defect features extracted by the cross-modal state space model, and outputs a fusion feature tensor.

8. The method for detecting defects in sheet metal based on large model-guided multimodal fusion according to claim 1, characterized in that, The defect category prediction and defect location regression based on the cross-modal fusion features are performed, and the defect detection results are output through a multi-task detection head. The multi-task detection head includes a classification branch and a regression branch; the classification branch takes the cross-modal fusion features as input and outputs a probability distribution prediction of the plate defect category; The regression branch takes the cross-modal fusion features as input and outputs the geometric parameters of the defect bounding box; the defect categories include dead knots, cracks, holes, mold, and no defects; During model training, a joint loss function is used for end-to-end training. The joint loss function is composed of a weighted sum of classification cross-entropy loss, regression mean squared error loss, and cross-modal alignment loss. The cross-modal alignment loss is the alignment loss between the semantic features generated by the large model and the cross-modal fused features.

9. The method for detecting defects in sheet metal based on large model-guided multimodal fusion according to claim 1, characterized in that, It also includes a real-time detection step: Visible light image data, infrared thermal image data, and environmental parameter data of the board material are collected on-site simultaneously, or historical data stored in a multimodal database are retrieved for spatiotemporal alignment preprocessing. The preprocessed data is input into the trained and converged defect detection model to perform forward inference and output the defect detection results, which include defect category, location coordinates and confidence level. The defect detection results are used to generate a diagnostic report and store it as text data. This report is then uploaded to the production line quality traceability database as a test record for this batch of boards and reserved as backup data for subsequent model fine-tuning and knowledge base updates.

Images