Vision-rwkv polarization multimodal defect detection method and system

Abstract

The application relates to the technical field of computer vision and industrial intelligent detection, and discloses a Vision-RWKV polarization multimodal defect detection method and system, which comprises the following steps: firstly, a multimodal image is preprocessed and input into a feature extraction network based on Vision-RWKV; multi-scale features are extracted by using a direction perception shift operator and a local context enhancement unit through a multimodal image and a structure guiding block; subsequently, attention is calculated in a local block by using a block attention fusion method to realize cross-modal interaction, and a double-branch structure of a structure guiding feature enhancement method is used to generate an attention map to modulate semantic features by using geometric structure information; and finally, pixel-level defect positioning results are output through decoding. While maintaining linear computational complexity, the application improves the direction perception and boundary positioning accuracy of micro defects, and effectively solves the feature alignment and blur problems in multimodal fusion.

Description

Vision-RWKV Polarization Multimodal Defect Detection Method and System Technical Field

[0001] This invention relates to the fields of computer vision and industrial intelligent inspection technology, specifically the Vision-RWKV polarization multimodal defect detection method and system. Background Technology

[0002] As modern industrial manufacturing moves towards precision and intelligence, the standards for inspecting product surface quality are becoming increasingly stringent. In the field of industrial appearance defect inspection, automated optical inspection technology based on deep learning is gradually replacing traditional manual visual inspection. However, traditional inspection methods mainly rely on visible light or grayscale images, which often lead to missed or false detections when dealing with highly reflective metal surfaces, transparent materials, or low-contrast dark defects (such as tiny scratches and dark cracks) due to a lack of sufficient optical feature information. To address this issue, multimodal inspection technology incorporating polarization imaging or depth information has become a research hotspot. By utilizing the sensitivity of polarized light to physical properties such as surface roughness and texture direction, the visibility of defects can be effectively enhanced.

[0003] While multimodal data provides richer information, existing deep neural network architectures face significant computational bottlenecks when processing this high-dimensional, heterogeneous data. Currently, most mainstream detection algorithms are based on convolutional neural networks (CNNs) or visual Transformer architectures. Although CNNs offer fast inference speeds, their limited local receptive field makes it difficult to capture long-range contextual dependencies in images, potentially leading to insufficient overall recognition of large-scale or continuous defects. In contrast, while the Transformer architecture based on self-attention mechanisms possesses excellent global modeling capabilities, its computational complexity and memory consumption increase quadratically with the input image resolution. In industrial inspection scenarios, high-resolution images are often required to clearly see minute defects, making the computational overhead of the standard Transformer architecture too high to meet the timeliness requirements of real-time production line inspection.

[0004] Furthermore, existing technologies also have significant shortcomings in multimodal feature fusion and detail recovery. Simple feature stitching or element-wise addition often ignores the nonlinear correlation of spatial distribution of different modal data (such as polarization intensity and polarization angle), making it difficult to achieve deep information complementarity. While using a global attention mechanism for cross-modal fusion can improve alignment, it further increases the computational burden. Simultaneously, after downsampling operations through multiple layers of networks, edge structural information of subtle defects such as cracks and scratches is easily lost, resulting in blurred boundaries of the defect segmentation mask generated in the final decoding stage, making pixel-level precise localization difficult. Therefore, designing a detection method that maintains low computational complexity while efficiently integrating multimodal information and accurately preserving fine-grained structural features is a pressing technical challenge in the field of industrial defect detection. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a Vision-RWKV polarization multimodal defect detection method and system, which solves the problems in existing industrial multimodal defect detection technologies, such as the high computational complexity of the traditional Transformer architecture in processing high-resolution images, insufficient extraction of small directional defect features, and the tendency for boundary blurring and structural information loss during cross-modal feature fusion.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] The first aspect of the present invention provides a Vision-RWKV polarization multimodal defect detection method.

[0008] This method first acquires raw multimodal image data of the industrial surface to be inspected. Preprocessing is then used to eliminate environmental noise and geometric distortion, generating a standardized input tensor sequence. This input tensor sequence is then fed into a feature extraction network based on the Vision-RWKV architecture. This network employs an RWKV backbone network with linear computational complexity as its basic architecture, extracting multi-scale hierarchical features through multimodal images and structure guide blocks. This design maintains the long-range dependency modeling capabilities of the Transformer architecture while reducing the computational overhead that increases with sequence length, making it more suitable for high-resolution industrial image processing.

[0009] Furthermore, in the feature extraction stage, the multimodal image and structure guide block enhance the perception of the spatial structure of defects through a computational mechanism. Specifically, firstly, the input features are expanded using a direction-aware shift operator, and feature groups are divided along the channel dimension. Spatial shift operations are then performed along the four directions of left, right, up, and down. This deterministic shift mechanism enables the network to explicitly capture directional defect features (such as cracks and scratches) and establish spatial dependencies. Subsequently, the features are aggregated using a local context enhancement unit. This unit processes the shifted features through parallel global and local branches, extracting global context representations and local detail representations respectively. These representations, along with the original input, are then weighted and fused using adaptive weight parameters to output hierarchical features that combine macroscopic semantics and microscopic details.

[0010] After acquiring multi-scale features, this invention employs a block-based attention fusion method for cross-modal interaction. This method generates query and key representations through projection and divides these representations into multiple non-overlapping local blocks in spatial dimension. The calculation of attention weights is restricted to within each local block, i.e., calculating the scaled dot product of the query vector and key vector within the block. This block-based strategy reduces the computational complexity of global attention from quadratic to linear, while enabling more refined handling of cross-modal feature alignment within local regions. After weighted aggregation of the value vectors using the calculated attention weights, the system inversely reassembles the local features and performs residual connections with the main modality features, achieving effective complementarity of multimodal information.

[0011] To address the boundary blurring issue that arises after feature fusion, this invention further utilizes a structure-guided feature enhancement method to process the fused feature map. This method employs a dual-branch architecture to decouple features: the semantic feature branch extracts region discrimination information using cascaded convolutions; the structural feature branch extracts orientation-sensitive geometric cues using decomposed convolutions (concatenated 1x3 and 3x1 convolution kernels) and dilated convolutions. The spatial attention map generated by the structural feature branch modulates semantic features through element-wise multiplication, thereby suppressing background noise and enhancing defect boundary responses. Finally, the enhanced feature map undergoes decoding, mapping high-dimensional features to class probabilities, and is upsampled to output pixel-level defect localization results.

[0012] As a preferred technical solution, during the data acquisition and preprocessing stage, intensity images with four different polarization directions (0 degrees, 45 degrees, 90 degrees, and 135 degrees) are acquired, and linear polarization degree images and polarization angle images are calculated and generated, which are then combined with the intensity images; alternatively, RGB images and depth images are acquired and combined. The combined images are adjusted to a uniform resolution using a bilinear interpolation algorithm and then normalized.

[0013] As a preferred technical solution, the specific method for generating a spatial attention map during the structure-guided feature enhancement process includes performing channel-weighted summation and bias stacking on the structural features, and mapping them to the 0-1 interval using the Sigmoid activation function. The modulated features are then fused with the original fused features through residual connections and output after ReLU activation.

[0014] As a preferred technical solution, a composite loss function containing a binary cross-entropy loss function and a Dice loss function is constructed during the model training phase. The pixel-level classification error and the region overlap error are calculated respectively, and the weighted summation guides backpropagation and parameter updates.

[0015] A second aspect of the present invention provides the Vision-RWKV polarization multimodal defect detection system.

[0016] The system includes a data input module, a feature extraction module, a feature fusion module, a feature enhancement module, and a decoding output module.

[0017] The data input module is used to acquire the raw multimodal image data of the industrial surface to be inspected and perform standardized preprocessing to generate an input tensor sequence.

[0018] The feature extraction module receives the input tensor sequence and extracts multi-scale hierarchical features using multimodal images and structural guide blocks based on the Vision-RWKV architecture. This module integrates orientation-aware shift logic and local context enhancement logic, enabling efficient capture of the directional features and multi-scale contextual information of defects.

[0019] The feature fusion module is used to execute the block-based attention fusion strategy. This module achieves efficient alignment and complementarity of cross-modal features by dividing the feature map into local blocks and calculating attention weights within each block, thereby reducing computational resource consumption.

[0020] The feature enhancement module executes a structure-guided feature enhancement strategy. This module is configured with parallel semantic and structural branches. It uses geometric cues extracted from the structural branch to generate an attention map, modulating the features of the semantic branch to enhance the edge structural information of the defect.

[0021] The decoding output module is used to decode the enhanced feature map, map the feature dimensions to class probabilities, and output the final pixel-level defect localization result after upsampling.

[0022] This invention provides a method and system for detecting Vision-RWKV polarization multimodal defects. It offers the following advantages:

[0023] 1. This invention employs a multimodal image and structure guide block based on the Vision-RWKV architecture, combining the linear computational complexity of the RWKV backbone network with a direction-aware shift operator. This not only overcomes the bottleneck of the traditional Transformer architecture where the computational overhead increases quadratically with the sequence length when processing high-resolution industrial images, achieving feature extraction with linear complexity, but also endows the network with explicit perception capabilities for directional surface micro-defects such as cracks and scratches through multi-directional spatial pixel shift operations, thereby improving the feature representation efficiency and accuracy of the detection system against complex texture backgrounds.

[0024] 2. This invention utilizes a block-based attention fusion method to process multimodal data. By dividing the global feature map into non-overlapping local blocks and calculating scaled dot product attention within each block, the computational path for cross-modal interactions is optimized, reducing the complexity of attention computation to a linear level. At the same time, it enhances the spatial alignment capability between polarization and intensity modes at the local fine-grained level, ensuring that complementary information between different modes can be efficiently integrated, and avoiding feature misalignment or interference caused by modal differences.

[0025] 3. This invention employs a structure-guided feature enhancement method, utilizing a dual-branch architecture to explicitly decouple semantic features from geometric structural features. Through the combination of decomposed convolution and dilated convolution in the structural branch, the system can accurately extract the edges and texture cues of defects and generate a spatial attention map. This allows for element-wise modulation of semantic features, resolving the common boundary blurring problem after multimodal fusion. While suppressing background noise response, it enhances the boundary clarity of minute defects, improving the pixel-level positioning accuracy of the final decoding output. Attached Figure Description

[0026] Figure 1 is a schematic diagram of the overall system architecture and module connections of the present invention;

[0027] Figure 2 is a flowchart of the multimodal data acquisition and preprocessing process of the present invention;

[0028] Figure 3 is a schematic diagram of the feature extraction block structure of the present invention;

[0029] Figure 4 is a schematic diagram of the structure of the structure-guided feature enhancement module of the present invention.

[0030] The components include: 1. Image acquisition device; 2. Data processing device; 3. Display terminal; 21. Data input module; 22. Feature extraction module; 23. Feature fusion module; 24. Feature enhancement module; and 25. Decoding output module. Detailed Implementation

[0031] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Referring to Figure 1, this invention provides a polarization multimodal defect detection system based on the Vision-RWKV architecture. The system mainly includes an image acquisition device 1, a data processing device 2, and a display terminal 3. The image acquisition device 1 and the data processing device 2 are connected via a high-speed data transmission interface, and the data processing device 2 is communicatively connected to the display terminal 3.

[0033] Image acquisition device 1 is configured to acquire multimodal image data of an industrial surface to be inspected. In one specific embodiment, image acquisition device 1 includes a polarization camera for acquiring images at different polarization angles. The acquired data includes four polarization angle images, namely a 0-degree polarization image. 45-degree polarized image 90-degree polarization image and 135-degree polarized image Based on these original polarization images, image acquisition device 1 or data processing device 2 further calculates and generates a linear polarization degree image (DoLP) and a polarization angle image (AoP). In another specific embodiment, image acquisition device 1 is a binocular stereo vision system used to acquire RGB images and a depth image (Depth). Image acquisition device 1 will acquire the... One input image is transmitted to data processing device 2.

[0034] The data processing device 2 includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed, the computer program implements a polarization multimodal defect detection method based on the Vision-RWKV architecture. Logically, the data processing device 2 is divided into a data input module 21, a feature extraction module 22, a feature fusion module 23, a feature enhancement module 24, and a decoding output module 25.

[0035] Data input module 21 receives data from image acquisition device 1. The system takes one input image and preprocesses it. The preprocessing operation includes resizing all input images to a uniform resolution. In this embodiment, the input images are resized to... Pixels. Input data is represented as a set of tensors. Each input image , Indicates batch size. Indicates the number of channels. For polarization mode data, the input includes intensity and angle images, DoLP images, and AoP images; for RGB-D data, the input includes RGB images, grayscale images, and depth images.

[0036] The feature extraction module 22 is connected to the data input module 21 and is used to perform multi-scale hierarchical feature extraction on the input image for each modality. The feature extraction module 22 employs... The block serves as the core component. The block is built on the RWKV architecture and utilizes linear computational complexity for long-range dependency modeling. The feature extraction module 22 extracts a set of multi-scale hierarchical features for each input modality, represented as... The feature extraction module 22 includes an orientation-aware shift unit and a local context enhancement unit, which captures global contextual semantics and fine-grained surface geometry in the image through visually adaptive state space modeling.

[0037] The feature fusion module 23 is connected to the feature extraction module 22 and is configured to perform patch attention fusion (PAF). The feature fusion module 23 receives multi-scale hierarchical features from the feature extraction module 22 and processes the features spatially using the patch attention mechanism. The feature fusion module 23 establishes cross-modal correspondences through multi-path feature projection, divides the feature map into non-overlapping patches, and computes scaled dot product attention within each patch. This process achieves local, context-aware cross-modal interaction and outputs the fused feature representation. The feature fusion module 23 controls the fusion strength through a learnable residual scaling mechanism, achieving gradual integration of cross-modal information.

[0038] The feature enhancement module 24 is connected to the feature fusion module 23 and is configured to perform Structure-Guided Feature Enhancement (SGFE). The feature enhancement module 24 employs a dual-branch architecture, including a semantic branch and a structural branch. The structural branch utilizes decomposed convolutions to extract orientation-sensitive geometric cues, while the semantic branch extracts spatial features. The feature enhancement module 24 uses the features generated by the structural branch to generate an attention map and modulates the features of the semantic branch using this attention map. Finally, the feature enhancement module 24 outputs an enhanced feature map with enhanced geometric structural information and boundary details through a residual fusion strategy.

[0039] The decoding output module 25 is connected to the feature enhancement module 24 and is configured to map the enhanced feature map into pixel-level defect localization results. The decoding output module 25 generates the final output map. Each pixel value in the output image represents the probability or category of a defect at that location. Display terminal 3 displays this output image, visually presenting the location and shape of defects on the industrial surface. During the training phase of data processing device 2, the system uses a loss function calculation module to calculate the difference between the predicted results and the true labels. The loss function uses a weighted sum of binary cross-entropy loss and Dice loss. To evaluate the system's detection performance, data processing device 2 calculates the average cross-union ratio (CIU). Optimal dataset scale and optimal image scale Indicators such as these.

[0040] Among them, the average crossover ratio The calculation formula is as follows:

[0041] ;

[0042] In the formula, Indicates the number of categories. The category index representing the actual label. Index representing the predicted category, Indicates that the actual label is a category. However, it was predicted by the model to be of category The number of pixels, The correct prediction is categorized as follows: The number of pixels, Indicates that the actual label is a category. But it was predicted as a category The number of pixels.

[0043] Optimal dataset scalar The calculation formula is as follows:

[0044] ;

[0045] In the formula, Indicates the binarization threshold. Indicates at the threshold Precision rate Indicates at the threshold The recall rate is as follows. This indicates iterating through all thresholds. Take the maximum value.

[0046] Optimal image scale The calculation formula is as follows:

[0047] ;

[0048] In the formula, This indicates the total number of test images. For image indexing, Indicates the first Precision of images at the threshold Indicates the first Recall rate of images under the threshold Indicates that for the first Zhang image selection makes it The threshold for the highest score.

[0049] Referring to Figure 2, this embodiment details the specific process of image acquisition device 1 acquiring raw data and data input module 21 preprocessing the data. In the multimodal polarization data acquisition stage, image acquisition device 1 is configured to capture the polarization state information of the target surface. Specifically, image acquisition device 1 acquires intensity images of four different polarization directions in the same scene using a polarization imaging sensor, namely, 0-degree polarization images. 45-degree polarized image 90-degree polarization image and 135-degree polarized image These four components characterize the vibration intensity of light waves in different directions, forming the basic data source for calculating the Stokes vector and subsequent polarization parameters.

[0050] The data input module 21 receives the four polarization intensity images and calculates linear polarization degree (DoLP) and polarization angle (AoP) data based on these images. DoLP data quantifies the proportion of polarization components in the reflected light relative to the total light intensity; a higher DoLP value indicates a more pronounced polarization characteristic. This characteristic effectively highlights defect areas with surface microstructures, especially for scattering surface defects that appear as low-contrast in conventional intensity images. AoP data quantifies the principal vibration direction of polarized light, which is closely related to the surface normal direction. Therefore, AoP data can reveal subtle geometric changes and texture information of the surface. The data input module 21 combines the calculated DoLP and AoP image data with the original intensity and angle images to form the network's input multimodal data.

[0051] During the RGB-D data acquisition stage, image acquisition device 1 simultaneously acquires RGB color images and depth images using a binocular stereo vision system or a structured light sensor. The depth image is obtained by measuring the physical distance from each point on the object's surface to the sensor; the pixel values ​​directly correspond to spatial depth, thus providing three-dimensional geometric information about the degree of unevenness of defects (such as pits and scratches), supplementing the two-dimensional texture information contained only in the RGB image. Data input module 21 can also convert the RGB image to a grayscale image as needed, serving as an auxiliary modal input to reduce the interference of color noise on texture feature extraction.

[0052] The data input module 21 performs standardized preprocessing operations on the acquired multimodal images. First, the data input module 21 performs a combination operation to correct parallax and distortion between different sensors or different channels, ensuring that different modalities (such as RGB and Depth, or polarization intensity and DoLP / AoP) are strictly spatially corresponding in the pixel coordinate system, that is, the same coordinate position corresponds to the same point on the object surface.

[0053] Subsequently, the data input module 21 performs a resolution adjustment operation. Regardless of the original image resolution output by the image acquisition device 1, the data input module 21 uses an interpolation algorithm to uniformly adjust the spatial dimensions of all input images to a fixed size. In this embodiment, the uniform height... and width All are 512 pixels, that is .

[0054] Finally, the data input module 21 converts the adjusted image data into a tensor format acceptable to the model. Each input mode, the input data is constructed as a tensor sequence. Each tensor The dimension is set as .here, Represents the batch size during the training or inference process; The number of channels represents the modality; for example, for a single-channel DoLP image, AoP image, or grayscale image. For RGB images, The data input module 21 also normalizes the image pixel values, linearly mapping their numerical range to the [0,1] interval to eliminate the differences in numerical magnitude between different modal data and accelerate the convergence speed of the subsequent feature extraction module 22. The processed tensor sequence is then transmitted to the feature extraction module 22 for further processing.

[0055] Referring to Figure 3, in this embodiment, the feature extraction module 22 is built based on the Vision-RWKV architecture and utilizes... The block processes the input tensor sequence. The block aims to address the limitations of the receptive field in traditional convolutional neural networks and the quadratic increase in computational complexity of traditional Transformer architectures with sequence length. By introducing orientation-aware shift operators and local context enhancement units (LCEs), the RWKV backbone network is endowed with explicit perception of spatial structure while maintaining linear computational complexity.

[0056] Feature extraction module 22 first processes the input feature map The input direction-aware shift operator enhances spatial awareness through deterministic directional feature shifting. Specifically, the feature extraction module 22 first processes the input features... application The convolutional layer is then followed by a ReLU activation function for non-linear mapping to expand the channel dimension and generate extended features. Subsequently, the extended feature is uniformly divided into four feature groups along the channel dimension, with each feature group corresponding to a spatial shift direction. The feature extraction module 22 performs spatial shift operations along the left, right, up, and down directions on these four feature groups, respectively. This process is described by the following formula:

[0057] ;

[0058] In the formula, Indicates the first The output features after shifting each feature group; Indicates along direction And the pixel offset is Shift operation functions; Indicates the first One extended feature group; This represents the elements in the set of shift directions. Through this operation, pixels originally located in the feature map space can be aggregated from their four neighboring regions (distanced by ). The feature information of the four shifted feature groups is used to establish explicit spatial dependencies, which is geometrically adaptable for capturing directional defect patterns (such as cracks and scratches). The four shifted feature groups are then concatenated and processed through another... The convolutional layer projects back to the original feature dimension, thus aggregating directional information.

[0059] To further enhance the ability of features to represent local details and global context, feature extraction module 22... The block integrates a Local Context Enhancement (LCE) unit. The LCE unit employs parallel aggregation paths to capture multi-scale spatial context. First, the feature extraction module 22 pairs convolutional features... A dual pooling operation is applied, simultaneously performing average pooling and max pooling to extract complementary spatial statistics. Average pooling helps smooth background noise, while max pooling helps preserve defect features. This pooling process is described by the following formula:

[0060] ;

[0061] In the formula, This represents the hybrid features after pooling; This indicates an average pooling operation; This represents the max pooling operation; Represents the input convolutional features; This indicates an element-wise addition operation.

[0062] Based on pooled features, the feature extraction module 22 constructs two parallel processing branches. One branch is used to model global context information and generate a global representation. Another branch is used to preserve local details and generate local representations. The outputs of these two branches are the same as the original input. Adaptive fusion is performed using learnable weights to balance fine-grained boundary information with extensive contextual cues. This adaptive fusion process is described by the following formula:

[0063] ;

[0064] In the formula, This represents the final output characteristic of the LCE unit; This indicates a thinning convolution operation, used for feature reshaping of the fused features; and The weights are learnable parameters, set to 0.5 during the initial training phase, and automatically adjusted during training to optimize the ratio of global to local information. Represents global context features; Indicates local details; This represents the original input features of the residual connection. (Through...) The block-level processing and feature extraction module 22 outputs multi-scale hierarchical feature sequences for each input modality. These features contain both long-range dependency information and fine local geometric details.

[0065] In this embodiment, the feature fusion module 23 is configured to execute a block attention fusion algorithm, which aims to efficiently integrate spatial correspondence information from different modalities (e.g., polarization intensity features and DoLP / AoP features, or RGB features and depth features). The PAF mechanism solves the problem that the computational complexity of traditional global attention mechanisms increases with the square of resolution by decomposing the global feature map into local blocks and calculating attention within each block. At the same time, it enhances the cross-modal feature alignment capability for minor defects through local context interaction.

[0066] The feature fusion module 23 first performs a multi-path feature projection step to establish cross-modal feature correspondences. Given two input feature maps to be fused... and All of their dimensions are Feature fusion module 23 utilizes parallel processing. The convolutional layer performs a linear mapping on the input features, which are then normalized by a group normalization (GN) layer to generate a query representation for attention computation. Key representation This projection process maps heterogeneous modal features to a unified metric space, and the calculation formula is as follows:

[0067] ;

[0068] In the formula, This indicates a group normalization operation. Compared to batch normalization, it is not sensitive to batch size and is more suitable for small-batch, high-resolution training scenarios in industrial inspection. Indicates the kernel size as The convolution operation.

[0069] Subsequently, the feature fusion module 23 performs a block space partitioning operation. To maintain the local spatial correlation of features and reduce computational overhead, the feature fusion module 23 does not directly calculate attention across the entire image; instead, it uses the projected feature map... and The space is divided into multiple non-overlapping local patches. The spatial size of each patch is set to [value missing]. The feature map is reorganized into one containing A sequence of blocks, with features within each block flattened. The recombined query vector. and key vector It is expressed as follows:

[0070] ;

[0071] In the formula, This represents the total number of blocks divided, calculated as follows: The subspace dimension representing the feature channel; This represents the side length of the block in pixels. This operation reduces the complexity of attention computation from being proportional to the square of the total number of pixels in the image. Reduced to linear complexity related to block size .

[0072] Next, feature fusion module 23 performs scaled dot product attention calculations within each divided local block. For the first... Each block, module calculates the query vector. With key vector The correlation scores between the components are calculated and normalized using the Softmax function to obtain the attention weight matrix. At the same time, the module uses the key vector as the value vector. Involved in subsequent calculations. The calculation formula is as follows:

[0073] ;

[0074] In the formula, This represents the matrix transpose operation; This is a scaling factor used to adjust the magnitude of the dot product value and prevent the Softmax function from entering the saturation region where the gradient is minimal. The function converts the relevance scores into weights in the form of a probability distribution.

[0075] After obtaining the attention weights, the feature fusion module 23 performs weighted aggregation and feature reconstruction. The module utilizes the calculated weights... value vector Weighted summation is performed to generate local features that incorporate cross-modal information. Subsequently, the module performs a reverse recombination operation. All The features of each local block are reassembled according to its original spatial location to restore a complete feature map. The calculation formula is as follows:

[0076] ;

[0077] Finally, the feature fusion module 23 outputs the final fused features through a learnable residual scaling mechanism. To ensure model stability in the early stages of training and to allow the network to adaptively adjust the injection intensity of cross-modal information, the module introduces a learnable scalar parameter. The calculation formula is as follows:

[0078] ;

[0079] In the formula, It is initialized to 0 at the start of training. This means that in the initial stage, the output of the fusion module is directly equivalent to the input of the main modality. (i.e., identity mapping), as training progresses, the network gradually adjusts through backpropagation. The value of is used to smoothly introduce complementary information from auxiliary modes, thus avoiding gradient oscillations or feature destruction caused by excessive modal differences.

[0080] Referring to Figure 4, in this embodiment, the feature enhancement module 24 is configured to execute a structure-guided feature enhancement algorithm. This module, located after feature fusion, aims to address the issues of boundary blurring and loss of minor defect structures that occur during the fusion of multimodal features. The SGFE module explicitly decouples and reassembles the semantic content and geometric structure of features through a dual-branch architecture. It utilizes attention masks generated from structural information to enhance the responsiveness of semantic features to defect boundaries, while maintaining a low computational load through channel reduction.

[0081] Feature enhancement module 24 first receives the input feature map from feature fusion module 23. Its dimensions are To reduce the number of parameters and computational cost of dual-branch processing while maintaining feature representation capabilities, the module first performs channel compression. This is achieved through a... The convolutional layer will have a certain number of input channels. According to the preset compression ratio Reduce the number of intermediate feature channels. .

[0082] Subsequently, the compressed features are fed into two parallel processing branches: a semantic feature branch and a structural feature branch. The semantic feature branch aims to extract high-level contextual information and ontological features of defects. The feature enhancement module 24 employs a cascaded convolutional structure in this branch. Specifically, the features are processed... Convolution performs an initial transformation, followed by feature projection operations. Finally, after convolution Extracting local spatial patterns. This process generates semantic features. It retains rich regional discrimination information. The processing logic of this branch is described as follows:

[0083] ;

[0084] In the formula, Indicates by parameters Defined transformation function, Indicates the kernel size as The convolution operation is performed. The structural feature branch aims to capture orientation-sensitive geometric cues, such as the direction of cracks and the edges of scratches. To efficiently extract these fine-grained structures, the feature enhancement module 24 employs a strategy combining decomposed convolution and dilated convolution in this branch. Instead of using standard square convolution kernels, the module utilizes... convolution kernel and convolution kernel By using concatenation, this decomposition design can enhance edge response in both the horizontal and vertical directions, effectively capturing directional texture variations. Subsequently, the features are subjected to a dilation rate of... Hollow convolution This branch expands the receptive field without reducing resolution, thereby perceiving a wider range of structural continuity. The structural features generated by this branch... The description is as follows:

[0085] ;

[0086] In the formula, Indicates by parameters Defined transformation function.

[0087] In obtaining structural features Subsequently, the feature enhancement module 24 performs an attention generation operation. The module compresses the multi-channel structural features into a single-channel spatial attention map through weighted aggregation along the channel directions. Specifically, the module applies a weighted summation to each spatial location of the structural features and introduces a bias term. Then activated by the Sigmoid function. The values ​​are mapped to the (0,1) interval to generate a spatial attention map. This attention map exhibits a higher response value in regions with clearly defined structures (such as edges and contours). The calculation process is as follows:

[0088] ;

[0089] In the formula, Indicates the first Aggregated weights for each channel.

[0090] Finally, feature enhancement module 24 performs feature modulation and residual fusion. The module utilizes the generated attention map. semantic features Perform element-wise multiplication modulation, i.e. This step utilizes structural information to suppress noise response in the background region and enhance feature intensity at defect boundaries. To prevent feature degradation and maintain effective gradient propagation, the module employs a residual connection method, combining the modulated features with the original input features. (go through The transformation is fused (using matching dimensions). The fusion process introduces a learnable scaling factor. This allows the network to adaptively adjust the weights for structural enhancement based on training feedback. The final output features... After ReLU activation function The processing and calculation are as follows:

[0091] ;

[0092] In the formula, This indicates output projection convolution. This indicates the output bias. Through the above steps, the feature enhancement module 24 outputs a feature map with both rich semantics and accurate geometric structure, providing a high-quality feature foundation for the pixel-level classification of the subsequent decoding output module 25.

[0093] In this embodiment, the decoding output module 25, the model training unit, and the performance evaluation unit together constitute the system's output and optimization backend. This part is responsible for mapping deep features into visualized defect prediction results, guiding network parameter updates through a loss function, and quantifying detection performance using rigorous statistical metrics.

[0094] Decoding output module 25 is connected to feature enhancement module 24 to receive high-dimensional feature maps enhanced by structure-guided enhancement. Since feature extraction and processing involve changes in the number of channels, the decoding output module 25 first performs a dimensionality projection operation. Specifically, the module uses a... The convolutional layer is used as the prediction head to reduce the channel dimension of the feature map from... The output is compressed to the target number of categories. In a binary defect detection task (i.e., defect and background), the number of output channels is set to 1. Subsequently, to ensure that the output resolution is strictly consistent with the original input image (and ground truth labels), the decoding output module 25 performs an upsampling operation. The module uses a bilinear interpolation algorithm to restore the spatial size of the feature map to the target number of categories. Pixel. Finally, the upsampled feature map is processed by the Sigmoid activation function, mapping the value of each pixel to a continuous interval of [0,1]. This value represents the confidence probability that the corresponding location belongs to the defect category. By setting a threshold (e.g., 0.5), the probability map can be converted into a binary defect segmentation mask, where areas with a pixel value of 1 represent detected defects, and areas with a pixel value of 0 represent the background.

[0095] During the model training phase, data processing device 2 updates the network weights using the backpropagation algorithm. This embodiment employs the AdamW optimizer, which combines momentum and RMSProp algorithms and introduces a decoupled weight decay strategy, enabling it to maintain training stability and convergence speed when dealing with sparse gradients and non-stationary objective functions. The initial learning rate is set to 0.0005, and the weight decay coefficient is set to 0.0001 to prevent overfitting.

[0096] To address the severe imbalance between positive and negative samples (i.e., the number of pixels in the defect area is far less than the number of pixels in the background area) commonly encountered in industrial defect detection, this invention constructs a composite loss function. It is composed of a weighted sum of the binary cross-entropy loss and the Dice loss. Binary cross-entropy loss It focuses on pixel-level classification accuracy. It independently calculates the predicted probability and the true label for each pixel. The logarithmic error between the pixels is calculated and averaged over all pixels. This component helps the model correctly classify as many pixels as possible overall.

[0097] Dice loss It focuses on the spatial overlap between the predicted and ground truth regions. Based on the Sørensen-Dice coefficient, it directly optimizes the ratio of the intersection to the union of the predicted and ground truth masks. The Dice loss is insensitive to the size of the foreground (defect) region, thus effectively mitigating the problem of small target defects being overwhelmed by a large background in the loss calculation, forcing the model to pay more attention to the overall shape and boundary continuity of the defect.

[0098] To objectively quantify the detection accuracy and generalization ability of the model, this embodiment uses the average intersection-union ratio (AUC). ), optimal dataset scale ( ) and optimal image scale ( () is used as the core evaluation indicator.

[0099] The average intersection-union ratio (OCR) is calculated based on the confusion matrix. For each category (defect and background), the system calculates the ratio of the intersection (the number of correctly predicted pixels) to the union (the sum of the actual number of pixels and the predicted number of pixels minus the intersection). This is the arithmetic mean of the ratios across all categories. This metric comprehensively reflects the balance between precision and recall in a model and is a universal standard for measuring the quality of semantic segmentation.

[0100] Regarding the optimal dataset scale, this metric is used to evaluate the model's global performance at a fixed threshold. The system sets a series of discrete thresholds (e.g., from 0.01 to 0.99, with a step size of 0.01) within the probability interval [0,1]. For each threshold, the system binarizes all images in the entire test dataset and calculates the corresponding... The score (the harmonic mean of precision and recall). Defined as the threshold that, among all candidate thresholds, makes the average of the entire dataset... The one whose score reaches the maximum value Numerical value. This metric measures the robustness of a model when using a single fixed parameter in real-world applications.

[0101] Regarding the optimal image scale, this metric is used to evaluate the theoretically best performance of the model at adaptive thresholds. different, The system allows for independent threshold selection for each image in the test set. The system targets the first... For each image, iterate through all candidate thresholds to find the one that satisfies the condition. Find the threshold for maximizing the score and record that maximum. Fraction. Defined as the maximum value of each of the test images. The average score. This metric reflects the upper limit of the model's performance under ideal parameter settings for images with different lighting, materials, or defect types, revealing the model's adaptability to single-sample features.

Claims

1. A Vision-RWKV polarization multimodal defect detection method, characterized in that, Includes the following steps: The raw multimodal image data of the industrial surface to be inspected is acquired, and the raw multimodal image data is preprocessed to generate a standardized input tensor sequence; the input tensor sequence is input into a feature extraction network based on the Vision-RWKV architecture, and multi-scale hierarchical features are extracted through multimodal images and structural guide blocks; The multi-scale hierarchical features are processed across modalities using a block-based attention fusion method to generate a fused feature map; the fused feature map is then processed using a structure-guided feature enhancement method with a dual-branch architecture to generate an enhanced feature map. The enhanced feature map is decoded to map the feature dimensions to class probabilities, and then upsampled to output pixel-level defect localization results.

2. The Vision-RWKV polarization multimodal defect detection method according to claim 1, characterized in that, The steps of acquiring raw multimodal image data of the industrial surface to be inspected and preprocessing the raw multimodal image data specifically include: acquiring intensity images with four different polarization directions, namely 0 degrees, 45 degrees, 90 degrees and 135 degrees; calculating and generating linear polarization degree images and polarization angle images based on the intensity images, and combining the linear polarization degree images, the polarization angle images and the intensity images; or, acquiring RGB images and depth images and combining them, using bilinear interpolation algorithm to adjust the combined image to a uniform resolution, and normalizing the image pixel values ​​to construct an input tensor sequence with dimensions of batch size, number of channels, height and width.

3. The Vision-RWKV polarization multimodal defect detection method according to claim 1, characterized in that, The steps for extracting multi-scale hierarchical features through multimodal images and structure guide blocks specifically include: using an RWKV backbone network based on linear computational complexity as the basic architecture; in the multimodal images and structure guide blocks, firstly, a direction-aware shift operator is used to perform channel expansion and multi-directional spatial shift operations on the input features to capture directional features; then, a local context enhancement unit is used to aggregate the shifted features through parallel global and local branches, outputting multi-scale hierarchical features that combine long-range dependencies and local details.

4. The Vision-RWKV polarization multimodal defect detection method according to claim 3, characterized in that, The steps of using the direction-aware shift operator to perform channel expansion and multi-directional spatial shift operations on the input features specifically include: applying 1x1 convolution and a non-linear activation function to the input features to expand the channel dimension and generate expanded features; uniformly dividing the expanded features into four feature groups along the channel dimension; performing deterministic spatial pixel shifts along the left, right, up, and down directions on these four feature groups respectively; concatenating the four shifted feature groups and projecting them back to the original feature dimension through 1x1 convolution.

5. The Vision-RWKV polarization multimodal defect detection method according to claim 3, characterized in that, The step of using a local context enhancement unit to aggregate the shifted features through parallel global and local branches specifically includes: simultaneously performing average pooling and max pooling operations on the input features and adding the results of the two to obtain a hybrid feature; based on the hybrid feature, extracting global context representation through the global branch and extracting local detail representation through the local branch; and using learnable adaptive weight parameters to perform weighted summation and fusion of the global context representation, the local detail representation, and the original input features.

6. The Vision-RWKV polarization multimodal defect detection method according to claim 1, characterized in that, The steps for cross-modal processing of the multi-scale hierarchical features using the block-based attention fusion method specifically include: projecting the input features of different modalities onto the blocks using 1x1 convolutions and group normalization layers to generate query and key representations; dividing the query and key representations into multiple non-overlapping local blocks in the spatial dimension; calculating the scaling dot product attention weight between the query vector and the key vector within each local block; weighting and aggregating the value vectors using the attention weights to generate local fusion features; and inversely reorganizing the local fusion features of all local blocks according to their original spatial positions, and adding them to the main modal features using a learnable residual scaling factor.

7. The Vision-RWKV polarization multimodal defect detection method according to claim 1, characterized in that, The steps of processing the fused feature map using a structure-guided feature enhancement method with a dual-branch architecture specifically include: channel compression of the input fused feature map; splitting the compressed feature map into parallel semantic feature branches and structural feature branches; in the semantic feature branch, extracting semantic features containing region discrimination information using cascaded convolution operations; in the structural feature branch, extracting direction-sensitive geometric cues using decomposition convolution composed of cascaded 1x3 and 3x1 convolution kernels, and expanding the receptive field through dilated convolution to generate structural features; generating a spatial attention map using the structural features, and using the spatial attention map to perform element-wise multiplication modulation on the semantic features.

8. The Vision-RWKV polarization multimodal defect detection method according to claim 7, characterized in that, The steps of generating a spatial attention map using the structural features and modulating the semantic features using the spatial attention map through element-wise multiplication specifically include: performing weighted summation and bias stacking on the structural features along the channel direction; mapping the summation result to a spatial attention map with values ​​between 0 and 1 using the Sigmoid activation function; multiplying the spatial attention map with the semantic features at corresponding positions to suppress background noise response; and fusing the modulated features with the original input fused features through residual connections and outputting the result via the ReLU activation function.

9. The Vision-RWKV polarization multimode defect detection method according to claim 1, characterized in that, The model training steps also include: constructing a composite loss function that includes a binary cross-entropy loss function and a Dice loss function; calculating the pixel-level classification error using the binary cross-entropy loss function; calculating the spatial overlap error between the predicted region and the real region using the Dice loss function; performing a weighted summation of the binary cross-entropy loss function and the Dice loss function, and updating the network parameters using the backpropagation algorithm.

10. The Vision-RWKV polarization multimodal defect detection system, characterized in that, The Vision-RWKV polarization multimodal defect detection method applied to any one of claims 1-9 includes: a data input module (21) for acquiring original multimodal image data of the industrial surface to be inspected, and preprocessing the original multimodal image data to generate a standardized input tensor sequence; a feature extraction module (22) for receiving the input tensor sequence and extracting multi-scale hierarchical features using multimodal images and structure-guided blocks based on the Vision-RWKV architecture; a feature fusion module (23) for performing cross-modal processing on the multi-scale hierarchical features using a block attention fusion strategy to generate a fused feature map; a feature enhancement module (24) for processing the fused feature map using a structure-guided feature enhancement strategy through a dual-branch architecture to generate an enhanced feature map; and a decoding output module (25) for decoding the enhanced feature map, mapping the feature dimensions to class probabilities, and outputting pixel-level defect localization results after upsampling.

Images