High-speed rail anomaly detection method based on spatial and semantic enhanced visual-language learning

Abstract

The application discloses a high-speed rail anomaly detection method based on space and semantic enhanced visual-language learning, and relates to the technical field of high-speed rail anomaly detection. The high-speed rail anomaly detection method comprises the following steps: S1, collecting data of each component inside the high-speed rail; S2, classifying and labeling the data, labeling each pixel point on the image to form a labeled image, and performing text description on the images in the data set; S3, S4, constructing a space and semantic enhanced visual-language learning network; S5, inputting a test image into the optimized space and semantic enhanced visual-language learning network model, and outputting an abnormal segmentation image. The application breaks through the limitation of traditional single visual mode, and has more excellent performance and generalization in processing small area anomalies, multi-scale irregular anomalies and diversified environmental changes, thereby providing a high-robustness detection algorithm with high precision and strong generalization ability for high-speed rail anomaly detection.

Description

Technical Field

[0001] This invention relates to the field of high-speed rail anomaly detection technology, specifically to a high-speed rail anomaly detection method based on spatial and semantic augmented visual-language learning. Background Technology

[0002] Trains operate at high speeds for extended periods, and their undercarriages and running gear face extremely complex internal conditions and external natural environments. This makes critical components highly susceptible to wear and even failure under the combined effects of multiple factors. Under the harsh conditions of high-speed operation, even minor component abnormalities can escalate into major traffic safety accidents, posing a significant threat to the lives and property of passengers. Therefore, achieving timely and accurate detection of abnormalities in critical components of high-speed trains is of paramount practical importance for strengthening railway safety and ensuring the safe travel of passengers.

[0003] Common anomalies in high-speed rail operation include oil leaks, foreign object intrusion, and plaster damage. Detecting these defects is extremely challenging: on the one hand, anomaly samples are scarce and exhibit significant intra-class variability, with irregular shapes and large scale ranges; on the other hand, there is a severe imbalance in the distribution between complex backgrounds and anomaly areas, further increasing the difficulty of identification. Early detection relied primarily on manual visual inspection, a method with inherent limitations: not only are labor costs high and work efficiency low, but inspectors are also susceptible to subjective fatigue and objective field-of-view limitations, making it difficult to continuously and comprehensively interpret high-resolution images, leading to a high risk of missed and false detections. In recent years, breakthroughs in deep learning technology have provided a new paradigm for addressing this challenge. Data-driven automated identification algorithms can efficiently and accurately extract anomaly features from complex images, significantly reducing labor and economic costs while significantly improving detection efficiency and reliability, and have become a core technical means to ensure the safety of high-speed rail operation and maintenance.

[0004] Currently, research on anomaly detection in high-speed rail is still in its early stages, with existing literature largely limited to data augmentation strategies or single-visual-modality detection paradigms. However, the inherent scarcity of anomaly samples on high-speed rail severely restricts the feature representation capabilities of single-modality methods, and insufficient contextual feature interaction fails to capture more representative features, resulting in low detection accuracy. Therefore, there is an urgent need to explore a more efficient and robust new detection framework to overcome existing performance bottlenecks and provide a solid technical guarantee for the safe and stable operation of high-speed rail. Summary of the Invention

[0005] The purpose of this invention is to provide a high-speed rail anomaly detection method based on spatial and semantic augmented visual-language learning. This method involves data collection, image annotation and text design, image preprocessing, construction of a spatial and semantic augmented visual-language learning network model, training and parameter optimization to obtain the optimal model, and inference using the model to output segmented images of anomaly distributions. This addresses the problems of limited feature extraction from existing single visual modalities and insufficient interaction of contextual features.

[0006] The technical solution of this invention to solve the above-mentioned technical problems is as follows: A method for detecting anomalies in high-speed trains based on spatial and semantic enhanced visual-language learning is provided, comprising the following steps: S1. Data Collection: Collect data on various components inside the high-speed rail, including images from different environments; S2. Image annotation and text design: Classify and annotate the data, annotate each pixel on the image to form an annotated image, and provide text descriptions for the images in the dataset; S3. Data Preprocessing: Preprocess the image and scale it to the same size; preprocessing includes data augmentation and normalization. S4. Construct a spatial and semantic augmented visual-language learning network: Input the preprocessed image and text description into the spatial and semantic augmented visual-language learning network model, extract multi-level visual and linguistic features, use a spatial semantic enhancer to enhance high-level visual features, use a multi-scale low-level visual feature enhancer to enhance low-level visual features, use a query-driven visual-language cross-modal fusion module to bridge the visual and linguistic modalities, train the model and optimize the parameters. S5. Output anomaly segmentation image: Input the test image into the optimized spatial and semantic enhancement vision-language learning network model, and output anomaly segmentation image.

[0007] Furthermore, in step S1, the data includes normal data and abnormal data; The abnormal data included oil stains, foreign objects, and damaged putty.

[0008] Furthermore, in step S2, the pixels on each abnormal image are labeled using LabelMe software to generate labeled images corresponding to the abnormalities; The pixel values ​​in the labeled image represent the category: 0 represents the background, 1 represents oil stains, 2 represents foreign objects, and 3 represents damaged putty.

[0009] Furthermore, in step S3, data augmentation includes random horizontal flipping, image brightness and saturation adjustment; Normalization involves normalizing the images using the mean and standard deviation from the ImageNet dataset.

[0010] Furthermore, the text encoder extracts language features, and the weights used during training are frozen and do not participate in weight adjustment.

[0011] Furthermore, in step S4, the spatial and semantic enhanced vision-language learning network model includes a spatial semantic enhancer, a multi-scale low-level feature enhancer, and a query-driven vision-language cross-modal fusion module.

[0012] Furthermore, in step S4, multi-level visual and linguistic features are extracted, including the following sub-steps: S401. Extracting multi-level visual features using a visual encoder. , Extracting language features using a text encoder ; S402. Based on the CC module, perform dimensionality reduction and remove redundant features from the multi-level visual features to obtain optimized multi-level features. , .

[0013] Furthermore, in step S4, a spatial semantic enhancer is used to enhance high-level visual features containing rich semantic information. This enhancer then interacts with other hierarchical features to obtain a spatial attention map, which is multiplied by the high-level visual features to highlight salient features. Finally, the enhanced features are added to the original high-level visual features. This process includes the following sub-steps: S403, Input Multi-level Features , ; It is a high-level visual feature that interacts with features at other levels; S404, will and To enable interaction, adaptive mean pooling will be used to... Scale to At the same scale, using convolution operations to The number of channels and Alignment obtained ,right Perform deformation and channel transformation operations to obtain ; S405 and The interaction is performed using matrix multiplication. The maximum value is taken in the last dimension of the interaction result, and then it is transformed into a matrix... Using the same dimensions, and obtaining spatial attention maps based on the Sigmoid function. ; S406 and , and The interaction process and and The interaction process is similar, and the same logic applies. and ; and with Multiply and then add Enhanced high-level visual features .

[0014] Furthermore, in step S4, a multi-scale low-level feature enhancer is used to enhance low-level visual features containing spatial detail information. Using low-level visual features as the main component, a cascaded interaction method is employed to sequentially fuse them with features from multiple scales. Specifically, this includes the following sub-steps: S407, Input Multi-level Features , ; and enhanced high-level visual features , These are low-level visual features; S408, Low-level visual features and enhanced high-level visual features The concatenated vectors are then transformed and input into a Swin Transformer block for long-range context dependency modeling to obtain refined features. ; S409, Applying multi-scale pooling attention modules Obtain the pooled attention map, and Multiply and then add to get ; S410, in and The above-mentioned interactions are used to obtain , and Enhanced low-level visual features of the interaction .

[0015] Furthermore, the multi-scale pooling attention module consists of three parallel global adaptive mean pooling operations, which compress the input feature map to different scales. , and Then, after upsampling, the samples are added together through a convolution operation and the Sigmoid function.

[0016] Furthermore, in step S4, a query-driven vision-language cross-modal fusion module is used to bridge the visual and language modalities. Learnable query embeddings are introduced as a semantic bridge to achieve multi-level information exchange and fine-grained semantic alignment, promoting deep fusion of vision and semantics and unifying them into a shared feature space. This specifically includes the following sub-steps: S411, Input Enhanced Low-Level Visual Features Enhanced high-level visual features Learnable query embeddings and and linguistic features ,right The deformation and transformation channels make it compatible with the scale of cross-modal interaction, which is similar to... The updated query representation is obtained through interaction with Cross Transformer. Two independent multilayer perceptrons are applied to A generated characteristic and Multiply to obtain the candidate segmentation mask Another generated vector and Multiplication yields the candidate class vector ; S412, , and The interaction between them uses the method in S411 to obtain the candidate segmentation mask. and candidate category vectors ; S413, Apply candidate segmentation mask and candidate segmentation mask After concatenation, the data is fused using a Swin Transformer block to obtain a comprehensive segmentation mask representation, which includes the candidate class vectors. and candidate category vectors After concatenation, the data is fused using a Transformer block to obtain a comprehensive category vector representation.

[0017] Furthermore, in step S4, the loss between the model's predicted value and the labeled image is calculated based on cross-entropy loss and Dice loss. The model parameters are optimized using Adamw, and samples are repeatedly extracted for iterative training to obtain the optimal parameter model. When training a model, loss calculation is required, which includes the following steps: S414. During training, the candidate segmentation mask and candidate class vector are matched with the labeled samples to calculate the loss and optimize the model parameters.

[0018] Furthermore, in step S5, the fused segmentation mask and the category vector are multiplied to obtain the final segmentation result.

[0019] This invention employs data collection, image annotation and text design, image preprocessing, and the construction of a spatial and semantically enhanced visual-language learning network model. The model is trained and its parameters optimized to obtain the optimal model, which is then used during inference to output segmented images with anomalous distributions. The spatial and semantically enhanced visual-language learning model includes a spatial semantic enhancer, a multi-scale low-level feature enhancer, and a query-driven visual-language cross-modal fusion module. For high-level visual features, a spatial semantic enhancer is proposed to strengthen them, enabling cross-level interaction between high-level visual features and other layer features to obtain spatial attention, thus restoring spatial detail while preserving rich semantics. For low-level visual features, a multi-scale low-level feature enhancer is proposed to enhance them, integrating them with other layer features to capture contextual dependencies and mitigate semantic deficiencies. The query-driven visual-language cross-modal fusion module utilizes learnable query vectors to promote interaction between visual and linguistic modalities, bridging the modal gap between them. This method is suitable for automated detection of anomalies in high-speed rail and exhibits strong generalization and robustness, significantly improving detection performance. This invention breaks through the limitations of traditional single visual modality and exhibits superior performance and generalization in handling small-area anomalies, multi-scale irregular anomalies, and diverse environmental changes. It provides a robust detection algorithm for high-speed rail anomaly detection that combines accuracy and strong generalization ability.

[0020] The present invention has the following beneficial effects: 1. This invention provides a vision-language bidirectional interactive anomaly detection framework. This framework breaks through the limitations of traditional single-modal or simple multimodal splicing. By constructing a cross-modal semantic alignment and dynamic interaction mechanism, it realizes deep coupling and complementary enhancement of visual features and language knowledge, thereby improving the performance of high-speed rail anomaly detection.

[0021] 2. This invention employs a spatial semantic enhancer to strengthen high-level visual features, compensating for their rich global semantics but lack of spatial detail; and a multi-scale low-level feature enhancer to enhance low-level visual features, compensating for their rich local detail but lack of global semantics. The enhanced high- and low-level visual features are then fully integrated to help the model better perceive anomalies, improve its detection capabilities, and enhance its generalization and robustness. Attached Figure Description

[0022] Figure 1 This is a flowchart of a high-speed rail anomaly detection method based on spatial and semantic augmented vision-language learning; Figure 2 Here are some example images of anomalies on high-speed trains; Figure 3 A schematic diagram of a spatial and semantically enhanced visual-language learning network model; Figure 4A schematic diagram of a spatial semantic enhancer; Figure 5 A schematic diagram of a multi-scale low-level feature enhancer; Figure 6 This is a schematic diagram of a query-driven vision-language cross-modal fusion module. Detailed Implementation

[0023] The present invention will be further illustrated below with reference to specific embodiments, but the embodiments do not limit the present invention in any way. Unless otherwise specified, the reagents, methods, and equipment used in the present invention are conventional reagents, methods, and equipment in this technical field.

[0024] Example 1:

[0025] A high-speed rail anomaly detection method based on spatial and semantic augmented vision-language learning (e.g.) Figure 1 (As shown), including the following steps: S1. Data (normal and abnormal data) of various components inside the high-speed train is collected using cameras from different perspectives. The data includes images from different environments; abnormal data includes oil stains, foreign objects, and damaged putty; examples of abnormal data are shown below. Figure 2 As shown; S2. Using LabelMe software, the pixels on each abnormal image are labeled to generate labeled images corresponding to the abnormality. The pixel values ​​in the labeled images represent the category: 0 represents the background, 1 represents oil stains, 2 represents foreign objects, and 3 represents putty damage. Text descriptions are also provided for the images in the dataset. S3. Preprocess the image and scale it to the same size; preprocessing includes data augmentation and normalization; data augmentation includes random horizontal flipping, image brightness and saturation adjustment; normalization includes normalizing the image using the mean and standard deviation from the ImageNet dataset. S4. Construct a spatial and semantically enhanced visual-language learning network model, such as Figure 3 As shown: The image and text descriptions are input into a spatial and semantically enhanced visual-language learning network model. Multi-level visual and linguistic features are extracted, and a spatial semantic enhancer is used to strengthen high-level visual features (such as...). Figure 4 As shown), a multi-scale low-level feature enhancer is used to enhance low-level visual features (such as...). Figure 5 As shown), a query-driven visual-language cross-modal fusion module is used to bridge the visual and language modalities (e.g. Figure 6 As shown in the figure, the loss between the model's predicted value and the labeled image is calculated based on cross-entropy loss and Dice loss. The model parameters are optimized using Adamw, and the optimal parameter model is obtained by repeatedly sampling and iteratively training. The specific steps include: S401. Extracting multi-level visual features using a visual encoder. , Extracting language features using a text encoder ; S402. Based on the CC module, perform dimensionality reduction and remove redundant features from the multi-level visual features to obtain optimized multi-level features. , ; S403, Input Multi-level Features , ; It is a high-level visual feature that interacts with features at other levels; S404, will and To enable interaction, adaptive mean pooling will be used to... Scale to At the same scale, using convolution operations to The number of channels and Alignment obtained ,right Perform deformation and channel transformation operations to obtain ; S405 and The interaction is performed using matrix multiplication. The maximum value is taken in the last dimension of the interaction result, and then it is transformed into a matrix... Using the same dimensions, and obtaining spatial attention maps based on the Sigmoid function. ; S406 and , and The interaction process and and The interaction process is similar, and the same logic applies. and ; and with Multiply and then add Enhanced high-level visual features ; S407, Input Multi-level Features , ; and enhanced high-level visual features , These are low-level visual features; S408, Low-level visual features and enhanced high-level visual features The concatenated vectors are then transformed and input into a Swin Transformer block for long-range context dependency modeling to obtain refined features. ; S409, Applying multi-scale pooling attention modules Obtain the pooled attention map, and Multiply and then add to get ; S410, in and The above-mentioned interactions are used to obtain , and Enhanced low-level visual features of the interaction ; S411, Input Enhanced Low-Level Visual Features Enhanced high-level visual features Learnable query embeddings and and linguistic features ,right The deformation and transformation channels make it compatible with the scale of cross-modal interaction, which is similar to... The updated query representation is obtained through interaction with Cross Transformer. Two independent multilayer perceptrons are applied to A generated characteristic and Multiply to obtain the candidate segmentation mask Another generated vector and Multiplication yields the candidate class vector ; S412, , and The interaction between them uses the method in S411 to obtain the candidate segmentation mask. and candidate category vectors ; S413, Apply candidate segmentation mask and candidate segmentation mask After concatenation, the data is fused using a Swing Transformer block to obtain a comprehensive segmentation mask representation, which includes the candidate class vectors. and candidate category vectors After concatenation, the comprehensive category vector representation is obtained by fusion through a Transformer block; S414, during training, the candidate segmentation mask and candidate category vector are matched with the labeled samples to calculate the loss and optimize the model parameters; S5. Input the test image into the optimized spatial and semantic enhancement visual-language learning network model, and multiply the fused segmentation mask and category vector to obtain the final segmentation result.

[0026] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A high-speed rail anomaly detection method based on spatial and semantic augmented vision-language learning, characterized in that, Includes the following steps: S1. Data collection: Collect data on various components inside the high-speed rail, including images under different environments; S2. Image annotation and text design: Classify and annotate the data, annotate each pixel on the image to form an annotated image, and provide text descriptions for the images in the dataset; S3. Data preprocessing: Preprocess the image and scale it to the same size; the preprocessing includes data augmentation and normalization. S4. Construct a spatial and semantic augmented visual-language learning network: Input the preprocessed image and text description into the spatial and semantic augmented visual-language learning network model, extract multi-level visual and linguistic features, use a spatial semantic enhancer to enhance high-level visual features, use a multi-scale low-level feature enhancer to enhance low-level visual features, use a query-driven visual-language cross-modal fusion module to bridge the visual and linguistic modalities, train the model and optimize the parameters. S5. Output anomaly segmentation image: Input the test image into the optimized spatial and semantic enhancement vision-language learning network model, and output anomaly segmentation image.

2. The high-speed rail anomaly detection method based on spatial and semantic augmented visual-language learning according to claim 1, characterized in that, In step S1, the data includes normal data and abnormal data; The abnormal data includes oil stains, foreign objects, and damaged putty.

3. The high-speed rail anomaly detection method based on spatial and semantic augmented visual-language learning according to claim 1, characterized in that, In step S2, the pixels on each abnormal image are labeled using LabelMe software to generate labeled images corresponding to the abnormalities. The pixel values ​​in the labeled image represent categories: 0 represents the background, 1 represents oil stains, 2 represents foreign objects, and 3 represents damaged putty.

4. The high-speed rail anomaly detection method based on spatial and semantic enhanced visual-language learning according to claim 1, characterized in that, In step S3, the data augmentation includes random horizontal flipping, image brightness and saturation adjustment; The normalization process involves normalizing the images using the mean and standard deviation from the ImageNet dataset.

5. The high-speed rail anomaly detection method based on spatial and semantic augmented visual-language learning according to claim 1, characterized in that, In step S4, multi-level visual features and linguistic features are extracted. It includes the following steps: S401. Extracting multi-level visual features using a visual encoder. , ; Extracting language features using a text encoder ; S402. Based on the CC module, perform dimensionality reduction and remove redundant features from the multi-level visual features to obtain optimized multi-level features. , .

6. The high-speed rail anomaly detection method based on spatial and semantic augmented visual-language learning according to claim 1, characterized in that, In step S4, a spatial semantic enhancer is used to enhance high-level visual features containing rich semantic information. This enhancer then interacts with other hierarchical features to obtain a spatial attention map. This attention map is multiplied with the high-level visual features to highlight salient features. Finally, the enhanced features are added back to the original high-level features. Specifically... include The following are the steps: S403, Input Multi-level Features , ; It is a high-level visual feature that interacts with features at other levels; S404, will and To enable interaction, adaptive mean pooling will be used to... Scale to At the same scale, using convolution operations to The number of channels and Alignment obtained ,right Perform deformation and channel transformation operations to obtain ; S405 and The interaction is performed using matrix multiplication. The maximum value is taken in the last dimension of the interaction result, and then it is transformed into a matrix... Using the same dimensions, and obtaining spatial attention maps based on the Sigmoid function. ; S406 and , and The interaction process and and The interaction process is similar, and the same logic applies. and ; and with Multiply and then add Enhanced high-level visual features .

7. The high-speed rail anomaly detection method based on spatial and semantic enhanced visual-language learning according to claim 1, characterized in that, In step S4, a multi-scale low-level visual feature enhancer is used to enhance low-level visual features containing spatial detail information. Using low-level visual features as the main component, a cascaded interactive approach is employed to sequentially fuse them with features from multiple scales. Specifically... It includes the following steps: S407, Input Multi-level Features , ; and enhanced high-level visual features , These are low-level visual features; S408, Low-level visual features and enhanced high-level visual features The concatenated vectors are then transformed and input into a Swin Transformer block for long-range context dependency modeling to obtain refined features. ; S409, Applying multi-scale pooling attention modules Obtain the pooled attention map, and Multiply and then add to get The multi-scale pooling attention module consists of three parallel global adaptive mean pooling operations, which compress the input feature map to... , and Then, after upsampling, the samples are added together through a convolution operation and a sigmoid function; S410, in and The above-mentioned interactions are used to obtain , and Enhanced low-level visual features of the interaction .

8. The high-speed rail anomaly detection method based on spatial and semantic enhanced visual-language learning according to claim 1, characterized in that, In step S4, a query-driven vision-language cross-modal fusion module is used to bridge the visual and language modalities. Learnable query embeddings are introduced as a semantic bridge to achieve multi-level information exchange and fine-grained semantic alignment, promoting deep fusion of vision and semantics and unifying them into a shared feature space. Specifically, this includes the following sub-steps: S411, Input Enhanced Low-Level Visual Features Enhanced high-level visual features Learnable query embeddings and and linguistic features ,right The deformation and transformation channels make it compatible with the scale of cross-modal interaction, which is similar to... The updated query representation is obtained through interaction with Cross Transformer. Two independent multilayer perceptrons are applied to A generated characteristic and Multiply to obtain the candidate segmentation mask Another generated vector and Multiplication yields the candidate class vector ; S412, , and The interaction between them uses the method in S411 to obtain the candidate segmentation mask. and candidate category vectors ; S413, Apply candidate segmentation mask and candidate segmentation mask After concatenation, the data is fused using a Swin Transformer block to obtain a comprehensive segmentation mask representation, which includes the candidate class vectors. and candidate category vectors After concatenation, the data is fused using a Transformer block to obtain a comprehensive category vector representation.

9. The high-speed rail anomaly detection method based on spatial and semantic enhanced visual-language learning according to claim 8, characterized in that, In step S4, the loss between the model prediction and the labeled image is calculated based on cross-entropy loss and Dice loss. The model parameters are optimized using Adamw, and samples are repeatedly extracted for iterative training to obtain the optimal parameter model. When training a model, loss calculation is required, which includes the following steps: S414. During training, the candidate segmentation mask and candidate class vector are matched with the labeled samples to calculate the loss and optimize the model parameters.

10. The high-speed rail anomaly detection method based on spatial and semantic enhanced visual-language learning according to claim 8, characterized in that, In step S5, the fused segmentation mask and the category vector are multiplied to obtain the final segmentation result.

Images