Modal loss-oriented lightweight self-insight fusion RWKV crack segmentation method and system

Abstract

The invention relates to the technical field of computer vision and deep learning, and discloses a mode-missing-oriented lightweight self-insight fusion RWKV crack segmentation method and system, and the method comprises the following steps: recognizing an available mode of input data, executing feature extraction and channel calibration through a reconstruction feature generator, and generating a global reconstruction feature map; the feature extraction subsystem generates a sequence scanning strategy according to the global features, and controls an introspection perception unit to code an image sequence through linear recursive calculation; and the cross-modal interaction fusion module integrates the multi-level features, and finally reconstructs and outputs a crack prediction map step by step through a decoder. According to the method, the problem of feature failure caused by modal deficiency is solved through a dynamic aggregation mechanism, the long-distance dependency relationship of the crack is effectively captured while the model parameter quantity and the calculation burden are greatly reduced by utilizing an adaptive scanning strategy and a block codebook technology, and the robustness of crack detection in a complex environment is improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning technology, specifically to a lightweight self-insight fusion RWKV crack segmentation method and system for modality missing features. Background Technology

[0002] In infrastructure maintenance, such as the detection of surface cracks in roads, bridges, and dams, is crucial for ensuring public safety. With the development of sensor technology, multimodal crack segmentation using multi-source data, including visible light, depth information, and infrared thermal imaging, has become a mainstream trend. Multimodal data provides complementary information, helping algorithms to more accurately identify crack features even under uneven lighting or complex backgrounds.

[0003] In practical engineering applications, sensor data often suffers from partial loss due to hardware failures, data transmission errors, or environmental interference. Most existing multimodal segmentation networks are designed based on the assumption that all modal data is complete, typically employing a fixed multi-branch structure for parallel feature extraction. When modal loss occurs in the input data, the conventional approach is to directly fill the missing channels with zeros or replace them with the mean. This approach disrupts the statistical properties of the data, preventing the network from effectively establishing global contextual relationships and severely impacting the robustness of the segmentation results.

[0004] Furthermore, cracks typically exhibit long, thin, and continuous geometric structures, requiring algorithms to capture long-range dependencies. Traditional convolutional neural networks, limited by their local receptive fields, struggle to model the continuity of cracks globally, easily leading to fragmented segmentation results. While Transformer-based methods address the global receptive field issue through self-attention mechanisms, their computational complexity increases quadratically with image resolution, resulting in enormous computational resource consumption and failing to meet real-time processing requirements. How to effectively capture global long-range dependencies while maintaining linear computational complexity remains a significant challenge for current technology.

[0005] Meanwhile, crack detection equipment is typically mounted on mobile terminals such as drones and wall-climbing robots, which have very limited computing power and storage space. Existing high-performance segmentation models often come with a huge number of parameters and computational loads, making them difficult to deploy directly on edge computing devices. Although compression techniques such as model pruning or knowledge distillation exist, these methods often require complex post-processing or come at the cost of sacrificing some detection accuracy. Therefore, designing a lightweight model with low parameter count and low computational load characteristics at the structural design level is of great significance for the engineering implementation of crack detection. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a lightweight self-insight fusion RWKV crack segmentation method and system for modal missing scenarios, which solves the problems of incomplete feature extraction, high computational resource consumption, and difficulty in long-distance dependency modeling in existing multimodal crack segmentation technologies under modal missing scenarios.

[0007] To achieve the above objectives, this invention provides the following technical solution: a lightweight self-insight fusion RWKV crack segmentation method for modal missing features, comprising the following steps: The available modality index of the input data is identified, and the available modality data is input into the reconstruction feature generator. Feature extraction and channel calibration are performed on the available modality data to generate a global reconstruction feature map. The feature extraction subsystem generates a strategy distribution for sequence scanning based on the global reconstructed feature map, controls the internal introspective perception unit to encode the image feature sequence, and outputs a multi-level feature map. The cross-modal interaction fusion module receives the multi-level feature map, performs cross-modal interaction and integration of features from different modalities, and generates fused features; The decoder module receives the fused features, performs step-by-step sampling and reconstruction of the features, and outputs a pixel-level crack segmentation prediction map.

[0008] Preferably, the process by which the reconstruction feature generator generates the global reconstruction feature map is as follows: for each available modality, intermediate features are extracted using depthwise separable convolution; channel calibration is performed on the intermediate features to generate enhanced context features; gating coefficients of the original input features are calculated, and the original input features and the enhanced context features are weighted and combined using an adaptive gating fusion mechanism to obtain enhanced features; non-parametric element-wise aggregation is performed on all the enhanced features, and the global reconstruction feature map is generated through projection transformation. This step dynamically aggregates available modality information and constructs global semantic guidance at the input end, thereby reducing the dependence of subsequent feature extraction networks on specific modalities.

[0009] Preferably, the channel calibration process is as follows: performing global average pooling on the intermediate features to generate channel descriptors; processing the channel descriptors using one-dimensional convolution; generating channel weight vectors using the Sigmoid activation function; and multiplying the channel weight vectors element-wise with the intermediate features.

[0010] Preferably, the process by which the feature extraction subsystem generates the policy distribution is as follows: using a convolution classifier to process the global reconstructed feature map and extracting the unnormalized prediction score; The predicted scores are sampled using the Gumbel-Softmax reparameterization method to generate the policy distribution. A maximum indexing operation or a probability-based sampling operation is then performed on the policy distribution to obtain a defined scan permutation index. This scan permutation index is then used to rearrange the two-dimensional image features into a one-dimensional sequence. This step selects the sequence scan path based on the global semantics of the image, improving the linear complexity model's ability to capture the geometric continuity of cracks.

[0011] Preferably, the introspective perception unit includes a spatial mixing module, which performs the following steps: gating modulation of the input features of the spatial mixing module using the global reconstructed feature map; mapping the modulated features into an acceptability vector, a key vector, and a value vector using a linear projection layer; rearranging the key vector and the value vector according to the scan permutation index; performing SI-WKV recursive calculation on the rearranged sequence to generate an aggregated output vector; performing an inverse permutation operation to restore the spatial correspondence of the features; performing element-wise multiplication of the aggregated output vector and the acceptability vector; and outputting the spatially mixed features after transformation by the output linear projection layer.

[0012] Preferably, the introspective perception unit includes a channel mixing module, which performs the following steps: mapping the input features of the channel mixing module to the hidden layer space and applying the squared rectified linear unit activation function to generate intermediate features; processing the intermediate features using a structure containing two one-dimensional convolutional layers to generate channel attention weights and perform element-wise multiplication with the intermediate features; performing layer normalization on the multiplied features and generating a channel mixing output through a gating mechanism.

[0013] Preferably, the feature extraction subsystem employs a grouped codebook linear layer structure when performing linear transformation operations. This structure includes learnable codebook vectors and a static grouping graph. The codebook vectors store unique weight values, and the static grouping graph stores integer indices pointing to the codebook vectors. During computation, the static grouping graph is used as an address index to retrieve values ​​from the codebook vectors to dynamically synthesize a weight matrix, or the retrieved values ​​are directly used in conjunction with the input features. This structure reduces the number of model parameters and storage requirements through weight sharing and quantization strategies.

[0014] Preferably, the process by which the cross-modal interaction fusion module generates the fused features is as follows: the features of each modality are mapped to a unified feature space through an embedding function, and averaged and aggregated to form preliminary mixed features; the preliminary mixed features are processed using a cross-modal attention module, which integrates spatial convolution and channel attention mechanisms; the feature channels are reweighted through the channel attention mechanism to output the fused features.

[0015] Preferably, the decoder module adopts a hierarchical cascaded structure; for the current decoding scale, it receives the decoding features from the previous level and performs an upsampling operation; the upsampled features are concatenated with the fused features of the same scale in the channel dimension; the concatenated features are fused by the update module to generate the decoding features of the current scale; the final decoding output is mapped to a binarized crack segmentation result by the convolutional prediction head.

[0016] A second aspect of the present invention provides a lightweight self-insight fusion RWKV crack segmentation system for modal missing features, comprising: The reconstruction feature generator identifies the available modality index of the input data, receives the available modality data and performs feature extraction and channel calibration to generate a global reconstruction feature map; The feature extraction subsystem receives the global reconstructed feature map, generates a strategy distribution for sequence scanning based on the global reconstructed feature map, and controls the internal introspective perception unit to encode the image feature sequence and output a multi-level feature map. The cross-modal interaction fusion module receives the multi-level feature map, performs cross-modal interaction and integration of features from different modalities, and generates fused features; The decoder module receives the fused features, performs upsampling and reconstruction of the features step by step, and outputs a pixel-level crack segmentation prediction map.

[0017] This invention provides a lightweight self-insight fusion RWKV crack segmentation method and system for modal missing features. It has the following advantages: 1. This invention achieves adaptive enhancement and global guidance for modality-missing data through a reconstruction feature generator. This method can automatically identify currently available modality data, enhance effective features using a channel calibration mechanism, and construct a reconstructed feature map containing global semantics through a non-parametric aggregation strategy. This ensures that even with missing sensor data, the network can still obtain complete global contextual information, effectively avoiding the problem of discontinuous or failed feature extraction due to modality loss, and improving the robustness of crack segmentation tasks in complex environments.

[0018] 2. This invention employs an introspective sensing unit that combines an adaptive scanning strategy, effectively solving the problem of long-distance dependency modeling of crack features. By generating a targeted sequence scanning strategy through a feature extraction subsystem, the two-dimensional image features are converted into a one-dimensional sequence that best suits the current crack geometry distribution. Combined with recursive calculations with linear complexity, it can capture global correlation information in the image without increasing the computational burden. Compared with traditional methods that are limited by local receptive fields, it can more accurately maintain the continuity and integrity of the crack structure.

[0019] 3. This invention introduces a grouped codebook linear layer structure in the feature extraction process, which achieves a high degree of model lightweighting. By decomposing the dense weight matrix into a shareable codebook vector and a static grouped index graph, and using quantized lookup table to replace the traditional fully connected computation, the model parameter quantity and storage space are greatly compressed while retaining the model's feature transformation capability. This enables the segmentation system to be deployed on edge devices with limited computing resources, meeting the practical application requirements of real-time performance and low power consumption in industrial settings. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the system functional architecture of the present invention; Figure 2 This is a schematic diagram of the operation flow of the method of the present invention.

[0021] The module consists of: 100, Reconstruction Feature Generator; 200, Feature Extraction Subsystem; 300, Cross-Modal Interactive Fusion Module; and 400, Decoder Module. Detailed Implementation

[0022] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Please see the appendix Figure 1 This invention provides a lightweight self-insight fusion RWKV crack segmentation system for modality missing. The system is built on a deep neural network and is used to process data input consisting of multiple modalities, and to complete crack feature learning and segmentation when some modal data is missing.

[0024] A lightweight, self-insightful fusion RWKV crack segmentation method and system for modality missing features accepts a multimodal image set as input, containing image channels from multiple modalities. The system mainly consists of: a reconstruction feature generator 100, a feature extraction subsystem 200, a cross-modal interactive fusion module 300, and a decoder module 400. These modules are connected sequentially to form a computational path for image semantic segmentation.

[0025] A reconstructed feature generator 100 is located at the system's input. The reconstructed feature generator 100 is configured to receive the currently available modal portion from the input data. Internally, the reconstructed feature generator 100 includes depthwise separable convolutional units and channel calibration logic for extracting contextual dependency information of the available modalities. The reconstructed feature generator 100 generates a globally consistent feature representation, i.e., a global reconstructed feature map, by aggregating and mapping the available modal features. This global reconstructed feature map is used to characterize the semantic information required to fill in missing modalities and serves as a guiding signal for subsequent feature extraction processes.

[0026] The feature extraction subsystem 200 is connected to the output of the reconstruction feature generator 100. The feature extraction subsystem 200 contains several stacked introspective sensing units. The feature extraction subsystem 200 divides the input features of each modality into a sequence of image patches and embeds positional codes. The feature extraction subsystem 200 receives a global reconstructed feature map from the reconstruction feature generator 100 and uses this global reconstructed feature map to generate a scanning strategy. The feature extraction subsystem 200 performs linear complexity sequence modeling on the image patch sequence according to the scanning strategy, extracting multi-level feature maps containing crack topology and texture details. In the linear transformation operation of the feature extraction subsystem 200, a grouped codebook linear layer structure is adopted, and a weight matrix is ​​constructed by combining codebook vectors and static grouped index maps.

[0027] The cross-modal interactive fusion module 300 is connected to the output of the feature extraction subsystem 200. The cross-modal interactive fusion module 300 is configured to receive feature maps of different scales and modalities. The cross-modal interactive fusion module 300 performs residual enhancement processing on the input features and executes a cross-modal attention aggregation operation. The cross-modal interactive fusion module 300 aggregates feature representations from all modalities at each feature scale, weights and fuses the features based on channel dimensions and spatial regions, generating a fused feature sequence.

[0028] Decoder module 400 is connected to cross-modal interactive fusion module 300. Decoder module 400 adopts a hierarchical cascaded structure. Decoder module 400 restores low-resolution high-level semantic features to high resolution through upsampling operations, and concatenates the restored features with fused features of the same scale. Decoder module 400 maps the concatenated features to a single-channel prediction map through convolutional layers, and outputs the final binarized crack segmentation result.

[0029] Please see the appendix Figure 2 This invention provides a lightweight self-insight fusion RWKV crack segmentation method for modal missing features, which includes the following steps: S1. Identify the available modality index in the input multimodal data and input the available modality data into the reconstruction feature generator 100; The reconstruction feature generator 100 performs feature extraction and channel calibration on the available modality data, calculates the channel importance weights and adaptive gating coefficients, and generates a global reconstruction feature map containing global semantic information.

[0030] S2. The feature extraction subsystem 200 receives the global reconstructed feature map and generates a strategy distribution for the sequence scan based on the global reconstructed feature map. The feature extraction subsystem 200 uses the strategy distribution to control the internal introspective sensing unit to encode and process the image feature sequence, and outputs a multi-level feature map containing information at different resolutions.

[0031] S3, the cross-modal interaction fusion module 300 receives multi-level feature maps, performs cross-modal interaction and integration of features from different modalities, and generates fused features.

[0032] S4, the decoder module 400 receives the fused features, performs step-by-step upsampling and reconstruction of the features, and outputs a pixel-level crack segmentation prediction map.

[0033] S5. During the model training phase, the loss function value between the output crack segmentation prediction map and the real label is calculated, and the network parameters in the reconstruction feature generator 100, feature extraction subsystem 200, cross-modal interactive fusion module 300 and decoder module 400 are updated using the backpropagation algorithm.

[0034] Upon receiving input data, the reconstructed feature generator 100 performs preprocessing, context extraction, and channel-calibrated feature enhancement operations for available modal features. This process aims to extract highly robust feature representations from incomplete modal inputs, providing a foundation for subsequent missing information filling.

[0035] The reconstructed feature generator 100 identifies the modality index present at the current time step based on the state of the input data. The multimodal input set is defined as follows: ,in This represents the total number of modes. The available mode index set is defined as follows: Then it can be represented by a set of modal features as ,in Indicates the first Input feature maps of available modalities, , , These represent the number of channels, height, and width of the feature map, respectively. For the set... Each available modal feature The system performs depthwise separable convolution operations to extract local shape features and contextual dependencies. Depthwise separable convolution comprises two sub-operations: depthwise convolution and pointwise convolution. Depthwise convolution performs convolution operations independently for each input channel to extract spatial features; pointwise convolution... Convolutional kernels are linearly combined along the channel directions for feature fusion. Compared to standard convolution, this operation reduces the number of computational parameters. After depthwise separable convolution, intermediate features containing local contextual information are generated. .

[0036] In obtaining intermediate features Subsequently, the system introduces a channel calibration mechanism to adaptively adjust the response weights of different feature channels. This mechanism first adjusts the intermediate features... Perform global average pooling operation Global average pooling modulates the feature map in spatial dimensions. The above is compressed to generate a size of The channel descriptor aggregates global spatial information for each channel. Subsequently, one-dimensional convolution is used... Process the channel descriptors. One-dimensional convolution. The kernel size is This operation captures inter-channel dependencies through local cross-channel interactions without reducing channel dimensionality. Compared to fully connected layers, one-dimensional convolution avoids the problem of excessive parameter count in fully connected layers while preserving local correlations between channels.

[0037] The result after one-dimensional convolution is then activated by the Sigmoid function. A mapping is performed to obtain a channel weight vector ranging from (0,1). This weight vector reflects the importance of each channel for the crack feature representation. Finally, the calculated channel weight vector is compared with the original intermediate features. Element-wise multiplication (i.e., Hadamard product) is performed to complete the feature recalibration. The specific calculation process for the above channel calibration and feature enhancement is defined by the following formula: in, Indicates the enhanced number after channel calibration. Contextual features of each modality; This represents the Adama product operation; Represents the composition of functions; This represents the Sigmoid activation function; This represents a one-dimensional convolution operation; This represents the global average pooling operation. Through the above processing, the reconstruction feature generator 100 can suppress background noise channels that contribute little to the segmentation task, while enhancing key channels containing crack texture and edge information, outputting enhanced contextual features. This feature This approach not only preserves the spatial structure of the original input but also incorporates global statistical information along the channel dimension, providing a filtered and enhanced feature foundation for generating globally consistent reconstructed features. For the specific code implementation of depthwise separable convolutions and the calls to underlying operators, those skilled in the art can utilize standard library functions from existing deep learning frameworks.

[0038] After the reconstruction feature generator 100 completes channel calibration and context feature extraction for a single available modality, in order to balance the low-level details preserved in the original features with the high-level semantic context in the calibrated features, the system further performs an adaptive gating fusion operation. This operation dynamically determines the contribution ratio of the original input features and the context-enhanced features in the final representation through a learnable gating mechanism, thereby achieving optimal feature combination.

[0039] The reconstructed feature generator 100 generates a corresponding gating coefficient matrix for each available modality. These gating coefficients are obtained by applying the original input features... application The convolutional layer performs the calculations. Convolutional layers are used to map features to a gating space, and then the output values ​​are compressed to the (0,1) interval by the Sigmoid activation function to generate gating coefficients. The gating coefficient In terms of spatial location, it corresponds at the pixel level and is used to indicate whether each pixel is more inclined to retain the original information or accept contextual refinement information.

[0040] Using the generated gating coefficients The reconstructed feature generator 100 performs an element-wise convex combination operation, transforming the original input features... With channel-calibrated context features The network then performs fusion. This fusion mechanism allows the network to adaptively adjust the information flow based on the local characteristics of the image content. For regions requiring high-frequency details, such as crack edges, the network can assign higher weights to the original features; while for textures inside cracks or background regions, it utilizes contextual features more extensively. The final enhancement features of the single-modality model are then used. The formula for calculating is defined by the following formula: ; in, Indicates the fused first Feature representation of each modality; These are the original input features; The gating coefficient is calculated as follows: Where Conv_{1×1} represents a convolution operation with a kernel size of 1×1 (also known as point convolution), used to perform a linear transformation of the channel dimension of the features. The addition and multiplication in the formula are element-wise operations, ensuring that the spatial dimension of the feature map remains unchanged.

[0041] After obtaining the enhanced feature set of all currently available modalities Subsequently, the reconstructed feature generator 100 performs aggregation and generation of globally consistent features. Due to missing input modalities, the system employs a non-parametric element-wise aggregation strategy to handle variable-length feature sequences. This aggregation strategy applies to all existing features... Channel-by-channel statistical integration is performed to generate a unified feature map. This feature map is then transformed and smoothed in the feature space through a projection layer, and upsampling is performed as needed to match the spatial resolution required by subsequent modules, finally outputting a global reconstructed feature map. The global reconstruction feature map It integrates information from all available modalities and implicitly fills in the semantic gaps of missing modalities in the feature space, providing global prior guidance containing complete scene information for the subsequent feature extraction subsystem 200.

[0042] To reduce the number of model parameters while maintaining the network's feature transformation capabilities, this invention employs a Group Codebook Linear Layer (GCLinear) instead of the traditional fully connected linear layer in the feature extraction and linear projection stages. The Group Codebook Linear Layer is designed based on the principle of parameter reparameterization, decomposing the conventional dense weight matrix into a shared numerical pool and a fixed index mapping.

[0043] In a standard linear layer of a deep neural network, the input feature dimension is defined as... The output feature dimension is Its transformation process depends on a weight matrix. The matrix Each element in Typically, it is an independent trainable floating-point number, which means that a standard linear layer needs to maintain... This parameter is an independent parameter. In multimodal high-dimensional feature processing scenarios, the number of channels increases quadratically, leading to excessive model storage requirements.

[0044] The block codebook linear layer will use the above weight matrix The parameter space is decoupled into two independent components: the learnable codebook vector. and static grouping chart Learnable codebook vectors Defined as a one-dimensional vector, belonging to space ,in This represents the size of the codebook, which is the total number of unique weight values ​​allowed in this linear layer. The elements in the array are entity parameters updated during network training using the backpropagation algorithm, and their numerical distribution represents the numerical characteristics of the weights of that layer. The value is usually set much smaller than This allows for parameter compression.

[0045] Static grouping chart Defined as a dimension The index matrix has the same shape as the weight matrix of a standard linear layer. Static grouping diagram. It is a non-trainable constant matrix that remains fixed after being generated during the network initialization phase. Each element in All are integer indices, and their values ​​are strictly limited to the set. Internal. Static grouping diagram Its function is to indicate each position in the weight matrix. Codebook should be used Which value in the codebook? Through this structural design, the parameters of the linear layer are no longer directly stored in a large matrix, but rather through storing a small number of codebook values. It uses a predefined lookup table to implicitly represent the weights. This reparameterization approach reduces the storage complexity of the weights from... Reduced to (Ignoring inference-time storage of static index graphs, and only considering the number of learnable parameters), this makes the model more lightweight during the training and storage phases.

[0046] During the operation of the block codebook linear layer, the complete weight matrix It is dynamically synthesized through an index lookup mechanism. This mechanism utilizes a static grouped graph. As an address index, from the learnable codebook vector The corresponding values ​​are retrieved from the matrix to fill the positions of each element in the weight matrix.

[0047] Before performing forward propagation computation, the system bases its calculations on the static grouping graph. The weight matrix is ​​constructed based on the mapping relationship defined in the code. For any position in the weight matrix element at Its value is not stored independently, but is directly taken from the codebook vector. The corresponding position in the diagram. Specifically, the system reads the static grouping diagram in... index value at And use that index value as a pointer to access the codebook vector. The Each component is assigned a value to... The process of index lookup and assignment for this set of elements is defined by the following formula: ; in, Represents the weight matrix In the Line number The column's weight value; A codebook vector representing learnable parameters; Represents a static grouped diagram In the Line number The column-stored integer index value ranges from 0 to... ; The subscript in the codebook vector is The element value.

[0048] From the perspective of overall matrix composition, static grouping diagram The two-dimensional coordinate space of the weight matrix Implicitly divided into A set of mutually disjoint subsets or partitions. Definition For the set of all coordinates mapped into the codebook, i.e. Based on this partition definition, the complete weight matrix... Through the The matrix is ​​constructed by a weighted sum of basis matrices. Each basis matrix corresponds to a partition and is weighted by the corresponding values ​​in the codebook. The matrix synthesis process is defined by the following formula: ; in, This is the complete weight matrix after synthesis; The size of the codebook; For the first in the codebook vector A number; It is a with A binary indicator matrix of the same dimension. In the binary indicator matrix... In the middle, if coordinates Belongs to partition If the value is 1, then the element at that position is 1; otherwise, it is 0. Through linear combination, the system can preserve the structural integrity of the weight matrix while ensuring that a large number of elements in the matrix share the same learnable parameter, thus achieving parameter sharing and model compression. The completed weight matrix... This is then followed by conventional matrix multiplication to complete the linear mapping from input features to output features. For the specific implementation of matrix multiplication, those skilled in the art can utilize the existing Basic Linear Algebra Subroutine Library (BLAS) for computation.

[0049] To quantify the improvement in model storage efficiency brought about by this structure when constructing neural networks using block codebook linear layers, this invention defines a parameter compression ratio metric. This metric measures the ratio between the number of parameters in a standard linear layer and the number of learnable parameters in a block codebook linear layer.

[0050] In a standard linear layer, each element of the weight matrix is ​​an independent floating-point parameter, and the total number of parameters is denoted as . The calculation method is the product of the input feature dimension and the output feature dimension. In the grouped codebook linear layer of this embodiment, although the logical shape of the weight matrix remains unchanged, the actual parameters that need to be stored and updated are only the elements in the codebook vector, and the total number of parameters is denoted as... Based on this, the parameter compression ratio Defined by the following formula: ; in, Indicates the compression ratio; This represents the number of parameters in a standard linear layer with the same input and output dimensions. This represents the number of learnable parameters in the linear layer of the block codebook; Indicates the dimension of the output feature space; Indicates the dimension of the input feature space; This indicates the size of the codebook. Since in actual configurations, the codebook size... The value is usually much smaller than the product of the input and output dimensions. The ratio This is typically represented by a large value, indicating that the structure can reduce the gradient storage space and optimizer state storage space required during model training, thus achieving lightweight processing of the model size.

[0051] To address the operational efficiency of the model during actual deployment, the system introduces an inference acceleration strategy. During model training, the weight matrix needs to be dynamically reconstructed as the codebook is updated. However, during inference, due to the changes in the codebook vector... and static grouping chart All values ​​are fixed and will no longer be updated. The system uses a caching mechanism to eliminate the computational overhead caused by weight reconstruction.

[0052] The caching mechanism is configured to trigger during the first forward propagation in the inference process. When the system detects that it is in inference mode and the cache is empty, the complete weight matrix is ​​calculated in one go according to the aforementioned index lookup and matrix composition rules. Subsequently, the synthesized weight matrix is... It resides in the computation memory as a constant tensor. In all subsequent inference steps, the system no longer performs the mapping operation from the codebook to the matrix, but directly calls the weight matrix in the cache. Matrix multiplication is performed with the input features. This strategy ensures that the computational complexity of the block codebook linear layer during the inference phase is identical to that of the standard linear layer, avoiding additional memory addressing operations and thus maintaining low storage footprint while ensuring high inference speed. For the specific underlying implementation of the matrix multiplication operation, those skilled in the art can use the Generalized Matrix Multiplication (GEMM) algorithm.

[0053] In order to improve the efficiency of perceiving crack topology, the feature extraction subsystem 200 introduces a self-insight strategy (SIS) when processing image feature sequences. This strategy aims to dynamically plan the scanning order of the feature sequence based on global context information, rather than using a fixed raster scanning order. The self-insight perception unit uses the global reconstructed feature map from the reconstructed feature generator 100 as prior information to infer the optimal scanning arrangement strategy for the current input image.

[0054] The feature extraction subsystem 200 receives the globally reconstructed feature map. This feature map contains global semantic information after filling in missing modalities. To transform the high-dimensional feature map into a policy distribution that can guide sequence scanning, the system first utilizes a convolution classifier. The feature map is processed. Convolution classifier. Typically composed of several layers of convolutional neural networks, it is used to map feature channels to class log odds. Since directly performing discrete sampling on the log odds is not advisable, as it would prevent the backpropagation algorithm from updating parameters during training, the system introduces the Gumbel-Softmax reparameterization technique.

[0055] The Gumbel-Softmax operation allows sampling from the classification distribution, and its sampling process is differentiable with respect to the model parameters. The system uses this operation to generate a smooth policy distribution probability matrix. The values ​​in this matrix represent the probability distribution of each pixel being accessed preferentially when generating a one-dimensional sequence. The specific calculation of the policy distribution generation is defined by the following formula: GumbelSoftmax ; in, This represents the generated policy probability distribution matrix, the size of which matches the spatial resolution of the input features; This indicates the convolution classifier operation used to extract unnormalized prediction scores; represents the globally reconstructed feature map; GumbelSoftmax represents the sampling function applied by the Gumbel-Softmax distribution. Through this formula, the system transforms continuous feature representations into potential discrete scanning decisions.

[0056] To ensure the generated strategy Having practical physical significance—that is, guiding the network to prioritize crack regions or structures of significant topological importance—the system introduces an auxiliary supervision signal during the training phase. This supervision signal utilizes a real crack label map. Constraints are imposed on the policy generation process. The system constructs an introspective loss function. The policy generation network is optimized by calculating the difference between the generated policy distribution and the real labels. The loss function is defined by the following formula: ; in, The value represents the supervised loss of the introspection strategy; CrossEntropy represents the cross-entropy loss function. The distribution of predicted policies; This corresponds to the actual pixel-level label image. By minimizing... The system forces the generated scanning strategy to give higher priority or specific arrangement patterns to pixels in the crack region, so that the subsequent sequence modeling module can process key features earlier and more centrally.

[0057] During the specific execution of inference or forward propagation, the system distributes the strategy. A deterministic pixel permutation index is obtained by performing either a maximum value indexing (argmax) operation or a probability-based sampling operation. This permutation index defines the scan permutation strategy. This method flattens and rearranges a two-dimensional image feature map into a one-dimensional token sequence. This content-based dynamic scanning approach makes adjacent elements in the one-dimensional sequence more semantically related, rather than merely geometrically adjacent, thus enhancing the sequence model's ability to capture long-distance dependencies. For the specific mathematical properties and code implementation of Gumbel-Softmax and the cross-entropy loss function, those skilled in the art can refer to relevant deep learning algorithm library documentation.

[0058] In the introspective perception unit, the spatial fusion module is responsible for capturing long-range dependencies in the spatial dimension of features. To achieve a global receptive field while maintaining linear computational complexity, this invention employs the SI-WKV (Self-Insight RWKV) algorithm, which incorporates an introspective strategy, to replace the traditional self-attention mechanism.

[0059] Spatial Hybrid Module Receives Input Features ,in The total number of spatial locations after flattening (i.e. ), This represents the number of feature channels. Before entering the core computation, the system performs a gated modulation operation (ReGate). This operation utilizes the globally reconstructed features output by the reconstructed feature generator 100. As a contextual condition, for input features Element-wise weighted modulation is performed to inject global semantic information into local features. The modulated features are then passed through three parallel linear projection layers, each mapping to an acceptance vector. Key vector Sum value vector The linear projection layer preferably adopts the aforementioned block codebook linear layer structure to reduce the number of parameters.

[0060] To optimize the sequence modeling results, the system applies a scan permutation generated by an introspective strategy. The feature sequence is rearranged. The generated key vector... Sum value vector The system is based on the sorted index. Adjust their order in the one-dimensional sequence so that semantically related pixels are adjacent to each other in the sequence.

[0061] On the rearranged sequence, the system performs SI-WKV recursive computation. This computation process employs a time-mixing mechanism with linear complexity, effectively aggregating historical information of the sequence. For the first... The Token (i.e., the rearranged Token) (a spatial location), its output state It is obtained by weighted summation of information from the current moment and all previous moments. This recursive calculation process is defined by the following formula: ; in, Indicates the first The aggregated output vector at each position; and Indicates according to the scanning strategy The mapped index of the actual physical location; It is a channel-level spatial attenuation parameter used to control the range of influence of historical information on the current moment, and is a learnable positive vector. This is a bias parameter used to independently weight the information at the current time step, ensuring that the current feature retains sufficient significance in the aggregation. The exponentiation operation in the formula... All operations are element-wise; , Indicates in the mapping index or The standard key vector at that location; Indicates in The standard value vector (Value) at that location; This represents a specific state feature vector (derived from a variant of the bond vector or a specific storage state) that participates in the molecular accumulation calculation at a historical time step. This represents a specific state feature vector (derived from a variant of the value vector or a specific storage state) that participates in the denominator normalization calculation at a historical time step.

[0062] The SI-WKV operation described above defines a mapping function from input key-value pairs to sequence output. To preserve the spatial correspondence of the feature maps, the system performs a reverse permutation operation after computation. This process restores a one-dimensional sequence to its original two-dimensional spatial layout. The overall operation flow of SI-WKV is described by the following formula: in, This indicates the output of the SI-WKV module; This represents the recursive calculation logic of the aforementioned formula; This represents a sequence permutation operation performed according to a strategy; This indicates a reverse permutation operation.

[0063] The final spatial mixing output is determined by the acceptance vector. Control. The receptivity vector, after being processed by the Sigmoid activation function, acts as a gating signal, determining the information transmission ratio at each spatial location. This gating signal is element-wise multiplied with the output of SI-WKV and then projected through the output projection layer. Perform the transformation. Final spatial blending features. It is calculated using the following formula: ; in, The output characteristics of the spatial mixing module; This is used to output a linear projection layer. Through the above mechanism, the spatial blending module can efficiently integrate global spatial information according to an adaptive scan path, capturing the slender topological structure of the crack.

[0064] In the introspective perception unit, the channel fusion module is configured to mine deep semantic attributes of feature channels through nonlinear transformations and channel attention mechanisms while preserving spatial correspondence. The channel fusion module receives input features from the preceding module. The input features have typically undergone layer normalization. The channel mixing module first processes the input features... Through linear projection layer The model is mapped to the hidden space, and a Squared-ReLU activation function is applied to introduce higher-order nonlinear features. This process aims to generate intermediate feature representations that are both sparse and highly expressive. Its calculation process is defined by the following formula: ; in, This represents the characteristics of the intermediate channel after activation; This represents a linear projection layer used for channel feature mapping, which preferably adopts the aforementioned block codebook linear layer structure to reduce the number of parameters; Squared-ReLU represents squaring the output after RelU activation.

[0065] To further enhance the interactivity of the channel dimension and filter key features, the channel blending module optimizes intermediate features. A channel attention mechanism is introduced. This mechanism first uses a global average pooling operation. Spatial dimensionality information is compressed, and then channel dependencies are captured through a bottleneck structure containing two one-dimensional convolutional layers. The first one-dimensional convolutional layer... It is used for dimensionality reduction to reduce computation, followed by ReLU activation, and then through a second one-dimensional convolutional layer. Dimensionality is restored, and finally the Sigmoid activation function is used. Generate channel attention weights in the range (0,1). The attention weight for this channel is defined by the following formula: ; in, Represents the channel attention weight vector; This indicates a global average pooling operation; and This represents a one-dimensional convolutional layer used for local interaction modeling in the channel dimension.

[0066] Obtaining channel attention weights Subsequently, the channel mixing module performs weighted modulation and normalization operations on the features. The system then processes the intermediate features... With weight Element-wise multiplication is performed to enhance the channel responses important for the crack segmentation task using an attention mechanism. Subsequently, to smooth the feature distribution and accelerate network convergence, the system performs a layer normalization (LayerNorm) operation on the weighted features to generate normalized modulated features. This intermediate processing step is represented as follows: .

[0067] Finally, the channel mixing module generates the final output through a gating mechanism. The system utilizes another set of linear projection layers. For the original input The transformation is performed, and an acceptability-gated signal is generated using the Sigmoid activation function. Simultaneously, a linear projection layer is utilized. Normalized modulation features Map back to the output dimension. Final channel blending output. It is obtained by element-wise multiplication of the acceptability-gated signal and the mapped modulation features, and its calculation is defined by the following formula: ; in, The output characteristics of the channel mixing module; This represents the linear projection layer used to generate the receptance vector; The linear projection layer used to map the modulated features into a value vector uses the above structure. The channel mixing module combines nonlinear activation, channel attention mechanism and gated linear unit to achieve refined extraction and recombination of multimodal crack features.

[0068] In the feature extraction subsystem 200, a dual-branch collaborative perceptron is constructed to organically combine the ability to capture long-range dependencies in the spatial dimension with the ability to mine deep semantics in the channel dimension. This perceptron, as a basic building block of the feature extraction network, is stacked in a cascaded manner to abstract image features layer by layer.

[0069] The dual-branch cooperative perceptron is configured to include two cooperating processing branches: a spatial mix branch and a channel mix branch. These two branches are structurally concatenated using a residual connection to jointly encode the input features. Define the... The input features of the two-branch collaborative perceptron of the layer are This feature first enters the spatial mixing branch.

[0070] In the spatial mixing branch, input features First, the features undergo layer normalization to unify the feature distribution and accelerate training convergence. The normalized features are then fed into the aforementioned spatial blending module. In this module, the system utilizes a dynamically generated scan path generated by an introspection strategy and aggregates global contextual information in the sequence using the SI-WKV algorithm. This process enables each pixel in the feature map to perceive semantic information from other locations in the image, effectively establishing end-to-end dependencies, especially for features with elongated topological structures like cracks. The output of the spatial blending module is then connected to the original input via residual connections. Add to generate intermediate features The computational logic of this process is defined by the following formula: ; in, Indicates the first Input features of the layer perceptron; LayerNorm represents the layer normalization operation; This represents the scan permutation index generated by the introspection strategy; SpatialMix represents the spatial mixing function containing SI-WKV operations. This represents the intermediate features after spatial context aggregation.

[0071] Subsequently, intermediate features The process then proceeds to the channel blending branch. In this branch, the features undergo layer normalization again before being fed into the aforementioned channel blending module. The channel blending module focuses on reorganizing and refining the channel dimensions of the features through nonlinear transformations and channel attention mechanisms, while preserving their spatial position. This process helps extract local texture details and edge sharpness information from cracks. The output of the channel blending module is also connected to the intermediate features via residual connections. The features are added together to generate the final output features of the current dual-branch cooperative perceptron. The computational logic of this process is defined by the following formula: ; Wherein, ChannelMix represents a channel mixing function that includes a squared-corrected linear unit and a channel attention mechanism; Indicates the first The final output feature of the layer will be used as the input to the next perceptron layer.

[0072] Through the aforementioned dual-branch collaborative structure, the system implements a feature processing logic of first global localization and then local refinement. The spatial hybrid branch solves the problem of limited receptive field by utilizing linear complexity sequence modeling, while the channel hybrid branch enhances the nonlinear expressive power of features. Both work collaboratively through residual paths, ensuring effective gradient propagation in the deep network, enabling the system to learn high-quality crack feature representations that combine global topological structure and local detailed texture without increasing computational overhead. For the specific implementation of layer normalization and residual connections, those skilled in the art can use standard operators in deep learning frameworks for construction.

[0073] After the dual-branch collaborative perceptron extracts multi-scale features of different modalities, the system performs multi-modal feature aggregation and hierarchical decoding operations to comprehensively utilize the complementary information of each modality and restore the spatial resolution of the image. This process aims to form an information-rich fusion representation and restore spatial details in a coarse-to-fine order.

[0074] First, regarding those from Features from different modalities are aggregated. For the first... Each feature scale is used by the system to acquire each modality. Input features To process within a unified feature space, the system first uses an embedding function. Features from each modality are mapped. Then, an average aggregation strategy is used to fuse the features from all modalities, forming preliminary hybrid features. To further optimize inter-modal interactions and enhance the representation of key features, these preliminary hybrid features are input into a cross-modal attention module. Internally, it integrates lightweight spatial convolution and channel attention mechanisms. Spatial convolution is used to extract local spatial features, while the channel attention mechanism dynamically reweights channels based on feature importance, ensuring that the model adaptively focuses on modal feature channels that contribute more to crack segmentation, thereby achieving efficient intermodal interaction and fusion. The final generated fused feature representation The calculation process is defined by the following formula: ; in, Indicates the first Scale fusion characteristics; Indicates the total number of input modes; Indicates the first The mode in the th ... Scale of input features; This represents a feature embedding or projection function; Indicates the first A scale-based cross-modal attention module, which includes a channel attention mechanism for dynamic weighting and a spatial convolution structure.

[0075] In obtaining a fusion feature set containing rich semantic information Then, the system uses a hierarchical decoder to gradually recover spatial details. This decoding process follows a logic of progressing from deep, coarse-grained features to shallow, fine-grained features. For the current decoding scale... The system receives data from the previous level (i.e., deeper levels). Decoding features of ) And perform an upsampling operation on it. This allows its spatial resolution to be integrated with the current scale. To maintain consistency, the upsampled features are then concatenated with the fused features at the current scale along the channel dimension. The concatenated features are then processed through an update module. This module typically contains convolutional layers or residual blocks to fuse semantic and detail information from different levels, thereby generating decoded features at the current scale. The above layered decoding process is defined by the following formula: ; in, Indicates the first Decoding output features at scale; Indicates the first Scale-based feature update module; This indicates an upsampling operation (such as bilinear interpolation or deconvolution). This indicates a feature concatenation operation along the channel dimension; For the fusion features corresponding to the scale; This represents the total number of layers in the feature pyramid. Through this iterative decoding process, the network ultimately outputs a segmentation prediction map containing fine-grained edge information. For the specific code implementation of the cross-modal attention module and the upsampling operation, those skilled in the art can utilize standard neural network algorithms within deep learning frameworks.

[0076] The multimodal crack segmentation method begins with the data acquisition step. The system receives multimodal image sequences from an acquisition terminal (such as a drone, crawling robot, or handheld inspection device). These multimodal image sequences include, but are not limited to, visible light RGB images, infrared thermal imaging images, and depth / LiDAR images. The system performs preprocessing operations on the input raw images, including resolution alignment, denoising, and normalization, mapping pixel values ​​to standard intervals to form the input tensor.

[0077] To address the modality missingness issue in the input data, the system performs a global consistency reconstruction step. For any given input sample, the system first detects the currently available modality set. For each available modality, its features are extracted using the aforementioned reconstruction feature generator 100, and noise is filtered out using a channel calibration mechanism. Subsequently, an adaptive gated fusion mechanism dynamically adjusts the weights of the original features and context features based on the local characteristics of the image content. The system non-parametrically aggregates all processed available modality features and generates a global reconstruction feature map through projection transformation. This feature map serves as a global prior for subsequent processing, implicitly compensating for missing modality information in the feature space.

[0078] After acquiring the global reconstructed feature map, the system proceeds to the feature extraction and encoding steps. This step utilizes multiple stacked dual-branch cooperative perceptrons (TBCPs) to construct a hierarchical encoder. In each layer, the system first generates a dynamic scan permutation index tailored to the current image topology based on the global reconstructed feature map using an introspection strategy (SIS). Using this index, the spatial blending module performs a linear complexity SI-WKV operation in the serialized one-dimensional space to capture the long-range dependencies of the cracks. Next, the channel blending module utilizes a grouped codebook linear layer and a squared ReLU activation function to mine deep semantic features while preserving the spatial structure. This process progressively reduces the spatial resolution of the feature map and increases the number of channels, ultimately generating a multi-scale feature pyramid representation.

[0079] Subsequently, the system performs feature decoding and segmentation map generation steps. The decoder receives multi-scale features from the encoder and performs hierarchical decoding by combining global context information. At each decoding level, the system uses a cross-modal attention module (CMA) to dynamically weight and fuse features from different modalities (or reconstructed supplementary features), and calculates the fused features using a formula. The fused features are concatenated and upsampled with the output of the previous decoding stage to gradually restore the spatial details of the image. Finally, the decoder output is passed through a convolutional prediction head and mapped to a pixel-level crack probability map. During the training phase, the system calculates the total loss, including binary cross-entropy loss, Dice loss, and introspection policy supervision loss, based on the predicted map and the ground truth labels, and updates the network parameters using the backpropagation algorithm; during the inference phase, the system directly outputs a binarized crack segmentation mask.

Claims

1. A lightweight self-insight fusion RWKV crack segmentation method for modal missing features, characterized in that, Includes the following steps: The available modality index of the input data is identified, the available modality data is input into the reconstruction feature generator, feature extraction and channel calibration are performed on the available modality data, and a global reconstruction feature map is generated; The feature extraction subsystem generates a strategy distribution for sequence scanning based on the global reconstructed feature map, controls the internal introspective perception unit to encode the image feature sequence, and outputs a multi-level feature map. The cross-modal interaction fusion module receives the multi-level feature map, performs cross-modal interaction and integration of features from different modalities, and generates fused features; The decoder module receives the fused features, performs upsampling and reconstruction of the features step by step, and outputs a pixel-level crack segmentation prediction map.

2. The lightweight self-insight fusion RWKV crack segmentation method for modal missing features as described in claim 1, characterized in that, The process by which the reconstruction feature generator generates the global reconstruction feature map includes: For each available modality, intermediate features are extracted using depthwise separable convolution; Channel calibration is performed on the intermediate features to generate enhanced contextual features; Calculate the gating coefficients of the original input features, and use an adaptive gating fusion mechanism to weightedly combine the original input features with the enhanced context features to obtain the enhanced features; Non-parametric element-wise aggregation is performed on all the enhanced features, and the global reconstructed feature map is generated by projection transformation.

3. The lightweight self-insight fusion RWKV crack segmentation method for modal missing features as described in claim 2, characterized in that, The channel calibration process includes: Perform a global average pooling operation on the intermediate features to generate channel descriptors; The channel descriptor is processed using one-dimensional convolution; Channel weight vectors are generated using the Sigmoid activation function, and then the channel weight vectors are multiplied element-wise with the intermediate features.

4. The lightweight self-insight fusion RWKV crack segmentation method for modal missing features according to claim 1, characterized in that, The process by which the feature extraction subsystem generates the policy distribution includes: The global reconstructed feature map is processed using a convolution classifier to extract unnormalized prediction scores; The predicted scores are sampled using the Gumbel-Softmax reparameterization technique to generate the policy distribution; Perform a maximum value indexing operation or a probability-based sampling operation on the strategy distribution to obtain a determined scan permutation index, which is used to rearrange two-dimensional image features into a one-dimensional sequence.

5. The lightweight self-insight fusion RWKV crack segmentation method for modal missing features according to claim 4, characterized in that, The introspective sensing unit includes a spatial blending module, which performs the following steps: The input features of the spatial mixing module are gated and modulated using the global reconstructed feature map; The modulated features are mapped into an acceptance vector, a key vector, and a value vector using a linear projection layer. The key vector and the value vector are rearranged according to the scan sort index; Perform SI-WKV recursive computation on the rearranged sequence to generate an aggregated output vector; Perform an inverse permutation operation to restore the spatial correspondence of features, multiply the aggregated output vector and the acceptance vector element-wise, and then output the spatial hybrid features after transformation by the output linear projection layer.

6. The lightweight self-insight fusion RWKV crack segmentation method for modal missing features according to claim 1, characterized in that, The introspective sensing unit includes a channel mixing module, which performs the following steps: The input features of the channel mixing module are mapped to the hidden layer space, and the squared-corrected linear unit activation function is applied to generate intermediate features. The intermediate features are processed using a structure containing two one-dimensional convolutional layers to generate channel attention weights, which are then multiplied element-wise with the intermediate features. The multiplied features are normalized at the layer level, and a channel-mixed output is generated through a gating mechanism.

7. The lightweight self-insight fusion RWKV crack segmentation method for modal missing features according to claim 1, characterized in that, The feature extraction subsystem employs a block codebook linear layer structure when performing linear transformation operations. The block codebook linear layer structure includes learnable codebook vectors and a static block graph; The codebook vector stores the unique weight values ​​that are allowed to exist, and the static grouping graph stores integer indices pointing to the codebook vector; During the calculation process, the static grouping graph is used as an address index to retrieve values ​​from the codebook vector to dynamically synthesize a weight matrix, or the retrieved values ​​are directly used to perform calculations with the input features.

8. The lightweight self-insight fusion RWKV crack segmentation method for modal missing features according to claim 1, characterized in that, The process by which the cross-modal interaction fusion module generates the fusion feature includes: The features of each modality are mapped to a unified feature space by an embedding function, and average aggregation is performed to form preliminary mixed features; The preliminary mixed features are processed using a cross-modal attention module, which integrates spatial convolution and channel attention mechanisms; The feature channels are reweighted using the channel attention mechanism to output the fused features.

9. The lightweight self-insight fusion RWKV crack segmentation method for modal missing features according to claim 1, characterized in that, The decoder module adopts a hierarchical cascaded structure; For the current decoding scale, receive the decoding features from the previous level and perform an upsampling operation; The upsampled features are then concatenated with the fused features of the same scale along the channel dimension. The updated module fuses and splices the features to generate the decoded features for the current scale. The final decoded output is mapped to a binarized crack segmentation result by a convolutional prediction head.

10. A lightweight self-insight fusion RWKV crack segmentation system for modal missing features, characterized in that, The lightweight self-insight fusion RWKV crack segmentation method for modal missing features, applied to any one of claims 1-9, includes: The reconstruction feature generator identifies the available modality index of the input data, receives the available modality data and performs feature extraction and channel calibration to generate a global reconstruction feature map; The feature extraction subsystem receives the global reconstructed feature map, generates a strategy distribution for sequence scanning based on the global reconstructed feature map, and controls the internal introspective perception unit to encode the image feature sequence and output a multi-level feature map. The cross-modal interaction fusion module receives the multi-level feature map, performs cross-modal interaction and integration of features from different modalities, and generates fused features; The decoder module receives the fused features, performs upsampling and reconstruction of the features step by step, and outputs a pixel-level crack segmentation prediction map.

Images