An anomaly detection system and method based on a large pre-trained model for contrastive language images

Abstract

This invention provides an anomaly detection system and method based on a contrastive language image pre-trained model, belonging to the field of track structure safety detection technology. Based on the advanced contrastive language image pre-trained CLIP model architecture, a multimodal model system for detecting railway fastener anomalies is constructed, successfully achieving accurate type determination and pixel-level localization of fastener anomalies. This system effectively utilizes the zero-shot recognition capability and powerful semantic representation characteristics of the CLIP model, eliminating the need to reconstruct or train models for untrained fastener anomaly types, significantly improving the generalization and practicality of the detection task, and overcoming the bottleneck of traditional methods requiring repeated annotation and data collection when dealing with unknown anomalies. By introducing an insertable adapter fine-tuning mechanism and a self-attention mechanism, while maintaining the original multimodal alignment capability of CLIP, the system significantly improves the extraction and perception capabilities of local anomaly features of fasteners, achieving excellent transfer and adaptation performance.

Description

Technical Field

[0001] This invention belongs to the field of track structure safety detection technology, specifically relating to an anomaly detection system and method based on a large pre-trained model of contrastive language images. Background Technology

[0002] In modern railway transportation systems, rail fasteners are indispensable core components of the track structure, and their health directly affects the safety and stability of railway transportation. During long-term service, fasteners may experience fractures, displacements, or missing parts due to factors such as cyclical load impacts, environmental corrosion, material aging, or improper maintenance. This significantly weakens the track's ability to maintain geometric position, affects train stability, and can lead to catastrophic accidents such as train derailments. Therefore, developing efficient and precise intelligent anomaly detection technology for rail fasteners is of great significance for achieving intelligent track maintenance, preventing major train accidents, and ensuring safe railway transportation.

[0003] Chinese patent publication CN119090839A discloses an intelligent detection method for abnormal rail fasteners based on multi-source information fusion. This method involves acquiring fastener images, electromagnetic data, and location data, and using deep learning to process the images to achieve intelligent detection through multi-source data fusion, enabling real-time early warning of rail fastener issues to improve railway traffic safety. Chinese patent application CN119313959A discloses a method, device, medium, and product for detecting abnormal railway fasteners. This method involves acquiring target rail image data, inputting the data into a fastener detection model to obtain fastener location information, and using the location information and the original image as input to an image classification neural network model to achieve abnormal fastener detection. Chinese patent application CN114612833A discloses a method and system for identifying abnormal rail fasteners based on deep learning. This method involves acquiring rail fastener images, locating the fastener position using a fastener detection model, generating a predicted bounding box image of the fastener, and inputting the predicted bounding box image and its basic information into a fastener abnormality state recognition model to obtain the final abnormality category of the fastener. Chinese patent application CN120298307A discloses a railway track defect detection method based on the MFGA-YOLO model, which includes inputting a dataset of track defect images into the MFGA model based on the YOLO11N architecture. The YOLO model is used to train railway track defects and output the detection results, which improves the detection efficiency and greatly reduces the cost of manual inspection.

[0004] The aforementioned technologies disclose fastener detection methods based on deep learning. However, existing methods only detect abnormal fastener types trained on the model, failing to detect abnormal fasteners not trained on, leading to misjudgments or missed detections when the detection model encounters unknown anomalies. Traditional deep learning assigns a dedicated fastener detection model to each anomaly detection task, requiring the development of different fastener detection models for different circuits. Summary of the Invention

[0005] The purpose of this invention is to overcome the above-mentioned shortcomings and provide an anomaly detection system and method based on a contrastive language image pre-trained large model. The aim is to construct a multimodal model for fastener anomaly detection capable of detecting unknown anomaly types without requiring repeated model building, significantly improving the detection accuracy of rail fasteners. This is achieved by introducing advanced instruments and equipment to acquire raw image data of rail fasteners, utilizing a vision-based large model tool for preliminary segmentation of anomaly samples, and introducing a pre-trained multimodal model for fastener anomaly detection along with fine-tuning techniques to achieve fastener anomaly detection for both trained and untrained anomaly categories.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides an anomaly detection method based on a large pre-trained model of contrastive language images, comprising the following steps: The original image data of the rail fastener is acquired and preprocessed to obtain the preprocessed image data; A standardized fastener dataset was constructed based on the preprocessed image data; The normalized fastener dataset is input into a pre-trained multimodal model for fastener anomaly detection to obtain anomaly detection results; The fastener anomaly detection multimodal model is constructed based on the CLIP model and the self-attention mechanism. The constructed fastener anomaly detection multimodal model is trained in two stages to obtain a pre-trained fastener anomaly detection multimodal model.

[0007] The method for constructing a normalized fastener dataset based on preprocessed image data is as follows: The preprocessed image data, including normal samples and abnormal samples, is randomly selected. The abnormal samples are initially segmented using a visual large model tool to generate an initial pixel-level binary mask for the abnormal samples, while an all-zero mask is automatically generated for the normal samples. The visual large model tool is an interactive segmentation tool. The initial pixel-level binary mask of the abnormal sample is corrected and the region is reconstructed to generate a pixel-level abnormal mask; The pixel-level anomaly mask, all-zero mask, and original image data are classified according to anomaly type, and a storage structure with an image-mask association index is established. Based on the storage structure of the image-mask association index, a corresponding natural language description is automatically generated for each image, forming a multimodal data sample with image-mask-text alignment; The multimodal data samples are normalized in size, converted in format, and verified to obtain a standardized fastener dataset.

[0008] In the step of inputting the normalized fastener dataset into the pre-trained multimodal model for fastener anomaly detection to obtain the anomaly detection results, the construction and training method of the multimodal model for fastener anomaly detection is as follows: By embedding trainable Adapter modules into the visual encoder and text encoder of the CLIP model, the updated CLIP model is obtained. A self-attention mechanism based on value vector similarity is constructed in the updated CLIP model to obtain a multimodal model for fastener anomaly detection; Based on the multimodal model for fastener anomaly detection, global features and local block features of fastener images in the normalized fastener dataset are extracted. The cosine similarity between the global features and local block features and the predefined normal state text embedding and abnormal state text embedding are calculated to obtain image-level fastener anomaly classification prediction and pixel-level anomaly region segmentation prediction. Image-level fastener anomaly classification prediction is optimized using the binary cross-entropy loss function, while pixel-level anomaly region segmentation prediction is optimized using the Dice loss function and the Focal loss function. The fastener anomaly detection multimodal model is trained in two stages based on the total loss function to obtain a pre-trained fastener anomaly detection multimodal model; The total loss function is a weighted sum of the binary cross-entropy loss function, the Dice loss function, and the Focal loss function.

[0009] In the step of embedding trainable Adapter modules into the visual encoder and text encoder of the CLIP model respectively to obtain the updated CLIP model, image features are extracted using the CLIP model. The formula for extracting the image features is as follows: (1) in, For the CLIP model, a visual encoder; A text encoder for the CLIP model; It is the set of real numbers; These represent image features and text features, respectively. For the input image, For the first Category names; It is the feature dimension. It represents the number of categories.

[0010] The Adapter module performs feature fusion with the image features extracted by the CLIP model through residual connections. The specific method is as follows: First, the Adapter module is transformed, and its formula is expressed as follows:

[0011] in, A transformation function that acts on image features. This is a transformation function that operates on text features; Let T be the activation function, and let T represent the transpose transformation. These are the visual trainable parameters of the Adapter module; These are the text-trainable parameters of the Adapter module. As a bottleneck dimension; Next, the transformed Adapter module is fused with the image features extracted by the CLIP model, and the formula is expressed as follows: (3) in, It is a hyperparameter or a learnable scalar. For the updated image features, This refers to the updated text features.

[0012] In the step of constructing a self-attention mechanism based on value vector similarity in the updated CLIP model to obtain a multimodal model for fastener anomaly detection, the query matrix in the self-attention mechanism is... Bond matrix Replace with value matrix A self-attention mechanism for value vector similarity is constructed, and its formula is expressed as follows: (5) in This represents the value of the self-attention mechanism. Indicates the first Row vectors at positions, Indicates the first A column vector at position , The length of the input sequence. Represents the dimension of a vector. This represents the probability distribution function.

[0013] The step of extracting global and local block features of fastener images from a standardized fastener dataset based on a multimodal model for fastener anomaly detection, and calculating cosine similarity between the global and local block features and predefined normal and abnormal state text embeddings to obtain image-level fastener anomaly classification prediction and pixel-level anomaly region segmentation prediction, is expressed by the following formula: (6) in: This represents the predefined text embeddings for abnormal and normal states. This represents the global feature vector of the fastener image. This represents the feature vector of a local block in the image of the fastener. This indicates the prediction results for fastener anomaly classification. This indicates the prediction results for anomaly region segmentation. The number of local blocks into which the image is segmented. This indicates the calculation of cosine similarity.

[0014] The steps of optimizing image-level fastener anomaly classification prediction using the binary cross-entropy loss function and optimizing pixel-level anomaly region segmentation prediction using the Dice and Focal loss functions are specifically expressed as follows:

[0015]

[0016] in: This represents the binary cross-entropy loss function. Indicates the true label, This indicates the probability that the fastener is normal or abnormal as predicted by the model; Represents the Dice loss function. Represents the first in the true segmentation mask A label of 1 pixel, The model represents the first The predicted probability that each pixel belongs to the target category. For smoothing coefficients; Represents the Focal loss function. As a balance factor, For modulation factor, This indicates the height and width of the segmentation mask.

[0017] The method for two-stage training of the multimodal model for fastener anomaly detection based on the total loss function is as follows: The first stage freezes the backbone parameters of the CLIP model and fine-tunes the Adapter module using the total loss function to establish the foundation for text-image semantic alignment. The second stage freezes all parameters from the first stage and fine-tunes the Adapter module for newly added fastener images using the total loss function to improve the sensitivity of the multimodal fastener anomaly detection model to detect unknown fastener anomalies that have not appeared before.

[0018] Secondly, the present invention provides an anomaly detection system based on a large pre-trained model of contrastive language images, comprising: The fastener image acquisition module is used to acquire the raw image data of the rail fastener and preprocess it to obtain the preprocessed image data; The normalized dataset construction module is used to construct a normalized fastener dataset based on preprocessed image data; The multimodal fastener anomaly detection and localization module is used to input the standardized fastener dataset into a pre-trained multimodal fastener anomaly detection model to obtain anomaly detection results; The fastener anomaly detection multimodal model is constructed based on the CLIP model and the self-attention mechanism. The constructed fastener anomaly detection multimodal model is trained in two stages to obtain a pre-trained fastener anomaly detection multimodal model.

[0019] Compared with the prior art, the present invention has the following beneficial effects: This invention, based on an advanced Contrastive Language–Image Pre-training (CLIP) model architecture, constructs a multimodal model for railway fastener anomaly detection, successfully achieving accurate type determination and pixel-level localization of fastener anomalies. This invention effectively utilizes the zero-shot recognition capability and powerful semantic representation characteristics of the CLIP model, eliminating the need to rebuild or train the model for untrained fastener anomaly types. This significantly improves the generalization and practicality of the detection task, overcoming the bottleneck of traditional methods requiring repeated annotation and data collection when dealing with unknown anomalies.

[0020] This invention integrates three core modules: a fastener image acquisition module, a standardized dataset construction module, and a multimodal fastener anomaly detection and localization module, forming a comprehensive detection solution. At the data level, automated mask annotation is achieved by introducing a large visual model, and multimodal data samples including image-mask-text alignment are constructed, providing a solid foundation for model training. At the algorithm level, by introducing a trainable Adapter module and a self-attention mechanism structure, the extraction and perception capabilities of local anomaly features of fasteners are significantly improved while maintaining the original multimodal alignment capabilities of the CLIP model, achieving excellent transfer and adaptation performance.

[0021] This invention uses the CLIP model as the base model and employs a fine-tuning method that can be inserted into the CLIP model to achieve fastener anomaly detection. This method offers high accuracy and strong generalization in fastener anomaly detection, fully utilizing the zero-shot recognition capability of the CLIP model to accurately detect unknown fastener anomaly types during the training phase. Through the fine-tuning strategy, only a few parameters need to be adjusted to quickly adapt to new tasks, avoiding the tedious process of repeatedly building models for different fastener anomaly types.

[0022] This invention fully utilizes fastener images and their corresponding natural language descriptions to form multimodal data samples aligned with image-mask-text. Through a multimodal model for fastener anomaly detection, it achieves accurate identification of both known and unknown types of fastener anomalies. Furthermore, this invention significantly enhances the CLIP model's ability to extract local features at the pixel-level anomaly information, thereby further improving the robustness and generalization performance of rail fastener anomaly detection.

[0023] The method proposed in this invention not only has high detection accuracy and strong anti-interference ability, but also has high system compatibility and engineering deployability. It can effectively adapt to changes in different line conditions, ambient lighting and fastener morphology, meet the urgent need for efficient, accurate and universal anomaly detection technology in railway safety operation and maintenance, and provide reliable technical support and innovative path for smart railway inspection. Attached Figure Description

[0024] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the dataset for railway rail fasteners. Figure 3 This is a diagram showing the detection results of the multimodal model for fastener anomaly detection in Embodiment 2 of the present invention. Detailed Implementation

[0025] To further understand the content of this invention, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.

[0026] Example 1 like Figure 1 As shown, an anomaly detection method based on a large pre-trained model of contrastive language images includes the following steps: S1: Obtain the original image data of the rail fastener and preprocess it to obtain the preprocessed image data; S2: Construct a normalized fastener dataset based on the preprocessed image data; S3: Input the normalized fastener dataset into the pre-trained fastener anomaly detection multimodal model to obtain the anomaly detection results; The fastener anomaly detection multimodal model is constructed based on the CLIP model and the self-attention mechanism. The constructed fastener anomaly detection multimodal model is trained in two stages to obtain a pre-trained fastener anomaly detection multimodal model.

[0027] Specifically, in step S1, the original image data of the rail fastener is acquired and preprocessed to obtain preprocessed image data. The details are as follows: Based on the need for automated inspection of railway fasteners, a high-performance industrial camera, a track robot trolley, a power module, a data storage module, and a synchronization control module are integrated to build an automated image acquisition system that can move along the track. Based on standard fastener samples, the parameters of industrial cameras are calibrated to accurately obtain their intrinsic parameters (such as focal length and principal point coordinates) and extrinsic parameters (such as installation angle and position), laying the foundation for subsequent accurate measurement and analysis of images. Based on information such as the actual length of the railway line to be inspected, track curvature, and fastener spacing, the travel speed of the track robot trolley, the sampling frequency of the industrial camera, and the image resolution are set. The track robot trolley is controlled to move at a constant speed and the industrial camera is triggered synchronously to collect high-definition raw image data. Based on the requirements for data quality control and enhancement, the collected raw image data is standardized and converted into a standardized format, automatically numbered and stored according to the type of anomaly (including displacement, breakage, loss, etc.), and the data volume is expanded through operations such as cropping, contrast adjustment and noise addition to obtain preprocessed image data.

[0028] In this preferred embodiment, a TVI-LCM-01 industrial camera is selected as the image sensor, which is installed on the track robot car. A 24V power module is used to power the system, a built-in 1TB storage module is used to cache image data, and an embedded synchronous control module is equipped to form an integrated image acquisition platform that can move autonomously along the track. The Zhang Zhengyou calibration method was adopted. By taking pictures of the multi-angle checkerboard calibration board, the intrinsic parameter matrix and distortion coefficient of the industrial camera were calculated. Combined with high-precision measuring tools, the installation extrinsic parameters and position coordinates of the industrial camera relative to the track were determined. The overall calibration reprojection error was controlled within 0.5 pixels.

[0029] For the track to be inspected, the travel speed of the track robot is set to 0.5m / s, the sampling frequency of the industrial camera is 10Hz, the image resolution is 2048×2048, and a single image covers two complete fastener areas. Through the synchronous control module, the movement of the track robot and the exposure time of the industrial camera are precisely coordinated, the track robot is controlled to move at a constant speed along the track and the industrial camera is triggered to acquire images in real time, so as to obtain high-definition original image data of the rail fasteners. The acquired raw image data underwent standardized preprocessing. OpenCV was used to convert all images to a uniform 518×518 pixel PNG format, and they were automatically named and categorized according to their labeled anomaly types (including displacement, breakage, and missing images). Data augmentation techniques such as random cropping, linear contrast adjustment, and adding Gaussian noise were employed to augment the acquired raw image data, ultimately resulting in a total of 2284 images. This yielded preprocessed image data with balanced categories and clear structure, providing a data foundation for subsequent training of a large-scale detection model.

[0030] S2: Based on the preprocessed image data, a normalized fastener dataset suitable for model training is constructed to provide training data for the multimodal anomaly detection model. Details are as follows: To balance efficiency and few-shot learning, preprocessed image data, including normal and abnormal samples, is randomly selected. Visual large model tools are used to perform preliminary segmentation on abnormal samples and generate initial pixel-level binary masks for abnormal samples. At the same time, all-zero masks are generated for normal samples to automate the annotation process. To address the problem of anomaly mask construction, the initial pixel-level binary mask of anomaly samples is modified and the region is reconstructed to produce a high-quality pixel-level anomaly mask that can distinguish multiple categories. Pixel-level anomaly masks, all-zero masks, and original image data are classified according to anomaly type, and a storage structure with an image-mask association index is established to ensure data retrieval. Based on the storage structure of the image-mask association index, a corresponding natural language description is automatically generated for each image, forming a multimodal data sample with image-mask-text alignment; Furthermore, the training set and validation set are divided, and the multimodal data samples are normalized in size, converted in format, and validated to obtain a standardized fastener dataset.

[0031] In a preferred embodiment, approximately 200 normal samples and 50 to 80 samples of each abnormal type are randomly selected from all preprocessed image data. An interactive segmentation tool (X-Anything) integrating the SAM model (Segment Anything Model) is used to identify and segment abnormal regions in the selected images, automatically generating initial pixel-level binary masks for abnormal samples. Normal samples are automatically generated with all pixels having a zero value to represent a state without abnormalities. The initial pixel-level binary mask of the abnormal sample is imported into image processing software (such as Photoshop, Knockout), and then compared with the original image data to obtain a pixel-level abnormal mask. The edge contour of the pixel-level abnormal mask is smoothed, and the discontinuous areas inside are filled to eliminate the noise generated by the initial segmentation and generate a high-quality pixel-level abnormal mask. The refined pixel-level anomaly masks and all-zero masks are paired with the corresponding original image data, and classified according to anomaly types such as "normal", "broken", and "displaced". The mapping relationship between images and masks is recorded through a JSON (JavaScript Object Notation) index file, and a storage structure for image-mask association index is established (e.g., "normal-000.png" corresponds to "normal-000_mask.png"). Using a multi-label text generation algorithm based on anomaly features, and based on the storage structure of the image-mask association index, the corresponding natural language description (e.g., "normal fastener", "broken fastener") is automatically generated, forming a multimodal data sample with image-mask-text alignment. The text, image, and mask are bound by a unique ID (Identity Document) and stored in JSON format to achieve semantic alignment among the three. Following the category distribution characteristics of the MVTec AD (MVTec Anomaly Detection Dataset), a training set (200 normal images, 50 images for each type of anomaly such as displacement, breakage, and loss) and a validation set (20 normal images and 20 images for each type of anomaly) were constructed from the labeled data. Both the training and validation sets underwent standardized processing including size normalization, format conversion, and data validation. A Python script was used to check file integrity (no corruption), mask-image size consistency, and text description-anomaly type matching. Finally, a standardized fastener dataset of 490 images (400 training images + 90 validation images) was formed. Some original fastener images and masks are shown below. Figure 2 As shown.

[0032] S3: Input the normalized fastener dataset into the pre-trained multimodal model for fastener anomaly detection to obtain the anomaly detection results. The specific construction and training method of the multimodal model for fastener anomaly detection is as follows: 1) Trainable Adapter modules are embedded in the visual encoder and text encoder of the CLIP model respectively to obtain the updated CLIP model; the Adapter module performs feature fusion with the image features extracted by the CLIP model through residual connection, so as to combine the original general semantic information of the CLIP model with the specific information required for the fastener anomaly detection task.

[0033] We adopted the CLIP "ViT-L-14-336" model released by OpenAI as the base model, and embedded a trainable Adapter module in each Transformer layer of its visual encoder and text encoder. Image features were extracted through the CLIP model, and the formula for image feature extraction is as follows: (1) in, For the CLIP model, a visual encoder; A text encoder for the CLIP model; It is the set of real numbers; These represent image features and text features, respectively. For the input image, For the first Category names; It is the feature dimension. This is the number of categories, Q=2.

[0034] The transformation of the Adapter module is expressed by the following formula: (2) in, A transformation function that acts on image features. Here, T represents the transformation function applied to text features; T denotes the transpose transformation. These are the visual trainable parameters of the Adapter module. These are the text-trainable parameters of the Adapter module. It is the bottleneck dimension, used for dimensionality reduction to reduce the number of parameters. This is the activation function.

[0035] The transformed Adapter module is then fused with the image features extracted by the CLIP model, and the formula is as follows: (3) in It is a hyperparameter or a learnable scalar. For the updated image features, This refers to the updated text features.

[0036] Furthermore, the Adapter module is used to adapt to downstream rail fastener anomaly detection tasks, with only the Adapter parameters optimized. The contrast loss is defined as follows: (4) in, For the set of trainable parameters, It is the number of training samples. For the first Each sample corresponds to the updated text features. For the first Each sample corresponds to the updated image features; It is the temperature coefficient.

[0037] 2) Construct a self-attention mechanism based on value vector similarity in the updated CLIP model to obtain a multimodal model for fastener anomaly detection.

[0038] The query matrix in the standard self-attention mechanism Bond matrix Replace with value matrix , build The self-attention structure enhances the CLIP model's ability to extract local anomaly information from fasteners. The self-attention mechanism is implemented as follows: (5) in This represents the value of the self-attention mechanism, which will be used to query the self-attention mechanism. ,key Replace all with , Indicates the first Row vectors at positions, Indicates the first A column vector at position , Represents the dimension of a vector. This represents the probability distribution function.

[0039] 3) Based on the constructed multimodal model for fastener anomaly detection, the CLIP visual encoder extracts global and local block features from fastener images in the normalized fastener dataset. The global and local block features are then compared with predefined normal and abnormal state text embeddings to calculate cosine similarity, resulting in image-level fastener anomaly classification prediction and pixel-level anomaly region segmentation prediction. The specific formulas are as follows: (6) in This represents the predefined text embeddings for abnormal and normal states. This represents the global feature vector of the fastener image. This represents the feature vector of a local block in the image of the fastener. This indicates the prediction results for fastener anomaly classification. This indicates the prediction results for anomaly region segmentation. The number of local blocks into which the image is segmented. This indicates the calculation of cosine similarity.

[0040] Furthermore, the anomaly region segmentation prediction results are upsampled to align with the original image size. The resulting image is obtained after alignment and upsampling. , This indicates the height and width of the segmentation mask.

[0041] 4) For image-level fastener anomaly classification prediction, a binary cross-entropy loss function is used for optimization to supervise the accuracy of global anomaly judgment. For pixel-level anomaly region segmentation prediction, a Dice loss function is used to optimize the overlap between the predicted region and the ground truth label. The Focal loss function is combined to alleviate the problem of foreground-background pixel imbalance. The total loss function is a weighted sum of the binary cross-entropy loss function, the Dice loss function, and the Focal loss function to achieve joint optimization. The specific implementation formula is as follows: (7) in This represents the binary cross-entropy loss function. Indicates the true label, This indicates the probability that the fastener is normal or abnormal as predicted by the model; (8) in Represents the Dice loss function. Represents the first in the true segmentation mask A label of 1 pixel, The model represents the first The predicted probability that each pixel belongs to the target category. For smoothing coefficients; Represents the Focal loss function. As a balance factor, For modulation factor, This indicates the height and width of the segmentation mask.

[0042] The total loss function is: (9) 5) The constructed fastener anomaly detection model undergoes two-stage training, specifically: Phase 1: Freeze the CLIP model backbone parameters, fine-tune the text adapter for text-image alignment in Phase 1, add trainable text adapter control coefficients, and use the total loss function for supervision to establish the foundation for text-image semantic alignment. Phase 2: Freeze all model parameters (including the adapter) trained in Phase 1, fix the text semantic embedding, fine-tune only the newly added fastener image adapter module, train using the total loss function, and add trainable image adapter control coefficients to enhance the fastener anomaly detection multimodal model's ability to generalize detection of unknown anomaly types.

[0043] Example 2 To verify the performance of the proposed anomaly detection method based on a large pre-trained model of contrastive language images for detecting novel (i.e., unseen during training) fastener anomaly types, a systematic evaluation was conducted based on the constructed normalized fastener dataset. This invention designed an anomaly cross-experiment: a specific anomaly type (e.g., "fracture") was hidden in the training set, and this type was treated as an unseen anomaly during testing to verify the model's generalization ability. Evaluation was conducted at both the image and pixel levels: image-level anomaly classification performance was measured using the Area Under the Receiver Operating Characteristic Curve (AUROC) and Average Precision (AP); pixel-level anomaly localization performance was evaluated using the AUROC metric. All experiments were performed on a GPU (Graphics Processing Unit) computing platform equipped with an NVIDIA RTX 4090 (48GB VRAM). Specific metrics are shown in Table 1 below. Table 1 Fastener Abnormality Detection Indicators

[0044] Experimental results demonstrate that this method achieves excellent detection and localization accuracy for both known and unknown anomaly types, proving its good generalization ability. For visualization of some anomaly detection results, please refer to [link to visualization]. Figure 3 .

[0045] Example 3 An anomaly detection system based on a contrastive language image pre-trained model includes: The fastener image acquisition module is used to acquire the raw image data of the rail fastener and preprocess it to obtain the preprocessed image data; The normalized dataset construction module is used to construct a normalized fastener dataset based on preprocessed image data; The multimodal fastener anomaly detection and localization module is used to input the standardized fastener dataset into a pre-trained multimodal fastener anomaly detection model to obtain anomaly detection results; The fastener anomaly detection multimodal model is constructed based on the CLIP model and the self-attention mechanism. The constructed fastener anomaly detection multimodal model is trained in two stages to obtain a pre-trained fastener anomaly detection multimodal model.

[0046] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An anomaly detection method based on a large pre-trained model of contrastive language images, characterized in that, Includes the following steps: The original image data of the rail fastener is acquired and preprocessed to obtain the preprocessed image data; A standardized fastener dataset was constructed based on the preprocessed image data; The normalized fastener dataset is input into a pre-trained multimodal model for fastener anomaly detection to obtain anomaly detection results; The fastener anomaly detection multimodal model is constructed based on the CLIP model and the self-attention mechanism. The constructed fastener anomaly detection multimodal model is trained in two stages to obtain a pre-trained fastener anomaly detection multimodal model.

2. The anomaly detection method based on a large pre-trained model of contrastive language images according to claim 1, characterized in that, The method for constructing a normalized fastener dataset based on preprocessed image data is as follows: The preprocessed image data, including normal samples and abnormal samples, is randomly selected. The abnormal samples are initially segmented using a visual large model tool to generate an initial pixel-level binary mask for the abnormal samples, while an all-zero mask is automatically generated for the normal samples. The visual large model tool is an interactive segmentation tool. The initial pixel-level binary mask of the abnormal sample is corrected and the region is reconstructed to generate a pixel-level abnormal mask; The pixel-level anomaly mask, all-zero mask, and original image data are classified according to anomaly type, and a storage structure with an image-mask association index is established. Based on the storage structure of the image-mask association index, a corresponding natural language description is automatically generated for each image, forming a multimodal data sample with image-mask-text alignment; The multimodal data samples are normalized in size, converted in format, and verified to obtain a standardized fastener dataset.

3. The anomaly detection method based on a large pre-trained model of contrastive language images according to claim 1, characterized in that, In the step of inputting the normalized fastener dataset into the pre-trained multimodal model for fastener anomaly detection to obtain the anomaly detection results, the construction and training method of the multimodal model for fastener anomaly detection is as follows: By embedding trainable Adapter modules into the visual encoder and text encoder of the CLIP model, the updated CLIP model is obtained. A self-attention mechanism based on value vector similarity is constructed in the updated CLIP model to obtain a multimodal model for fastener anomaly detection; Based on the multimodal model for fastener anomaly detection, global features and local block features of fastener images in the normalized fastener dataset are extracted. The cosine similarity between the global features and local block features and the predefined normal state text embedding and abnormal state text embedding are calculated to obtain image-level fastener anomaly classification prediction and pixel-level anomaly region segmentation prediction. Image-level fastener anomaly classification prediction is optimized using the binary cross-entropy loss function, while pixel-level anomaly region segmentation prediction is optimized using the Dice loss function and the Focal loss function. The fastener anomaly detection multimodal model is trained in two stages based on the total loss function to obtain a pre-trained fastener anomaly detection multimodal model; The total loss function is a weighted sum of the binary cross-entropy loss function, the Dice loss function, and the Focal loss function.

4. The anomaly detection method based on a large pre-trained model of contrastive language images according to claim 3, characterized in that, In the step of embedding trainable Adapter modules into the visual encoder and text encoder of the CLIP model respectively to obtain the updated CLIP model, image features are extracted using the CLIP model. The formula for extracting the image features is as follows: (1) in, For the CLIP model, a visual encoder; A text encoder for the CLIP model; It is the set of real numbers; These represent image features and text features, respectively. For the input image, For the first Category names; It is the feature dimension. It represents the number of categories.

5. The anomaly detection method based on a large pre-trained model of contrastive language images according to claim 4, characterized in that, The Adapter module performs feature fusion with the image features extracted by the CLIP model through residual connections. The specific method is as follows: First, the Adapter module is transformed, and its formula is expressed as follows: in, A transformation function that acts on image features. This is a transformation function that operates on text features; Let T be the activation function, and let T represent the transpose transformation. These are the visual trainable parameters of the Adapter module; These are the text-trainable parameters of the Adapter module. As a bottleneck dimension; Next, the transformed Adapter module is fused with the image features extracted by the CLIP model, and the formula is expressed as follows: (3) in, It is a hyperparameter or a learnable scalar. For the updated image features, This refers to the updated text features.

6. The anomaly detection method based on a large pre-trained model of contrastive language images according to claim 5, characterized in that, In the step of constructing a self-attention mechanism based on value vector similarity in the updated CLIP model to obtain a multimodal model for fastener anomaly detection, the query matrix in the self-attention mechanism is... Bond matrix Replace with value matrix A self-attention mechanism for value vector similarity is constructed, and its formula is expressed as follows: (5) in This represents the value of the self-attention mechanism. Indicates the first Row vectors at positions, Indicates the first A column vector at each position, The length of the input sequence. Represents the dimension of a vector. This represents the probability distribution function.

7. The anomaly detection method based on a large pre-trained model of contrastive language images according to claim 6, characterized in that, The step of extracting global and local block features of fastener images from a standardized fastener dataset based on a multimodal model for fastener anomaly detection, and calculating cosine similarity between the global and local block features and predefined normal and abnormal state text embeddings to obtain image-level fastener anomaly classification prediction and pixel-level anomaly region segmentation prediction, is expressed by the following formula: (6) in: This represents the predefined text embeddings for abnormal and normal states. This represents the global feature vector of the fastener image. This represents the feature vector of a local block in the image of the fastener. This indicates the prediction results for fastener anomaly classification. This indicates the prediction results for anomaly region segmentation. The number of local blocks into which the image is segmented. This indicates the calculation of cosine similarity.

8. The anomaly detection method based on a large pre-trained model of contrastive language images according to claim 7, characterized in that, The steps of optimizing image-level fastener anomaly classification prediction using the binary cross-entropy loss function and optimizing pixel-level anomaly region segmentation prediction using the Dice and Focal loss functions are specifically expressed as follows: in: This represents the binary cross-entropy loss function. Indicates the true label, This indicates the probability that the fastener is normal or abnormal as predicted by the model; Represents the Dice loss function. Represents the first in the true segmentation mask A label of 1 pixel, The model represents the first The predicted probability that each pixel belongs to the target category. For smoothing coefficients; Represents the Focal loss function. As a balance factor, For modulation factor, This indicates the height and width of the segmentation mask.

9. The anomaly detection method based on a large pre-trained model of contrastive language images according to claim 8, characterized in that, The method for two-stage training of the multimodal model for fastener anomaly detection based on the total loss function is as follows: The first stage freezes the backbone parameters of the CLIP model and fine-tunes the Adapter module using the total loss function to establish the foundation for text-image semantic alignment. The second stage freezes all parameters from the first stage and fine-tunes the Adapter module for newly added fastener images using the total loss function to improve the sensitivity of the multimodal fastener anomaly detection model to detect unknown fastener anomalies that have not appeared before.

10. An anomaly detection system based on a large pre-trained model of contrastive language images, characterized in that, include: The fastener image acquisition module is used to acquire the raw image data of the rail fastener and preprocess it to obtain the preprocessed image data; The normalized dataset construction module is used to construct a normalized fastener dataset based on preprocessed image data; The multimodal fastener anomaly detection and localization module is used to input the standardized fastener dataset into a pre-trained multimodal fastener anomaly detection model to obtain anomaly detection results; The fastener anomaly detection multimodal model is constructed based on the CLIP model and the self-attention mechanism. The constructed fastener anomaly detection multimodal model is trained in two stages to obtain a pre-trained fastener anomaly detection multimodal model.

Images