Railway infrastructure detection large model method and device

Abstract

The application discloses a railway infrastructure detection large model method and device, the method comprises the following steps: acquiring real-time multi-modal data of railway infrastructure; the real-time multi-modal data comprises image data and / or time series data; inputting the real-time multi-modal data into a pre-trained railway infrastructure anomaly detection model, and outputting an anomaly detection result; the railway infrastructure anomaly detection model is obtained by training a railway infrastructure anomaly detection large model network by using historical multi-modal data and a preconfigured detection knowledge base, and the application can improve the accuracy and efficiency of railway infrastructure detection, reduce the cost of railway infrastructure detection, and provide comprehensive and operable data support for subsequent maintenance and decision-making.

Description

Technical Field

[0001] This invention relates to the field of railway information technology, and in particular to a method and apparatus for large-scale railway infrastructure inspection. Background Technology

[0002] Anomaly detection in railway infrastructure is a crucial aspect of ensuring the safe operation of rail transit, but existing technologies still present challenges in actual infrastructure inspections.

[0003] Traditional manual inspections in complex environments have a high rate of missed detections. Rule-driven algorithms experience a significant drop in accuracy under non-ideal conditions and lack robustness, failing to meet the real-time inspection needs of thousands of kilometers of high-speed railways daily. Single visual models can only output defect location coordinates and lack semantic description capabilities for anomaly types, severity, and cause analysis. Furthermore, they cannot integrate prior knowledge such as railway industry standards, historical fault cases, and time-series waveform characteristics, resulting in a lack of multimodal semantic association capabilities. Consequently, the inspection results lack industry standardization and decision support for maintenance recommendations. Summary of the Invention

[0004] This invention provides a large-scale model method for railway infrastructure inspection, which can improve the accuracy and efficiency of railway infrastructure inspection, reduce the cost of railway infrastructure inspection, and provide comprehensive and operable data support for subsequent maintenance and decision-making. The method includes:

[0005] Acquire real-time multimodal data of railway infrastructure; real-time multimodal data includes image data and / or time-series data;

[0006] Real-time multimodal data is input into a pre-trained railway infrastructure anomaly detection model, which outputs anomaly detection results. The railway infrastructure anomaly detection model is obtained by training a large-scale railway infrastructure anomaly detection model network using historical multimodal data and a pre-configured detection knowledge base. The large-scale railway infrastructure anomaly detection model network includes a visual encoder, an adapter, and a large language model connected in sequence. The visual encoder is used to divide the image data in the received input data into blocks using a dynamic block-splitting strategy. The block-splitting image data and other data in the input data are processed through a ViT (Vision Transformer) layer to obtain data features. The adapter is used to perform pixel shuffling, add labels, and project the data features output by the visual encoder into the embedding space of the large language model in sequence. The large language model is used to concatenate the output of the adapter with the labeled text annotated using the pre-configured detection knowledge base for historical multimodal data. The concatenation result is input into the MoE (Mixture of Experts) architecture, and the processed result is output. The pre-configured detection knowledge base includes descriptive information for various facility data.

[0007] This invention provides a large-scale model device for railway infrastructure inspection, which can improve the accuracy and efficiency of railway infrastructure inspection, reduce the cost of railway infrastructure inspection, and provide comprehensive and operable data support for subsequent maintenance and decision-making. The device includes:

[0008] The acquisition module is used to acquire real-time multimodal data of railway infrastructure; the real-time multimodal data includes image data and / or time-series data;

[0009] The output module is used to input real-time multimodal data into a pre-trained railway infrastructure anomaly detection model and output anomaly detection results. The railway infrastructure anomaly detection model is obtained by training a large-scale railway infrastructure anomaly detection model network using historical multimodal data and a pre-configured detection knowledge base. The large-scale railway infrastructure anomaly detection model network includes a visual encoder, an adapter, and a large language model connected in sequence. The visual encoder is used to divide the image data in the received input data into blocks using a dynamic block-splitting strategy. The block-splitting image data and other data in the input data are processed through a ViT layer to obtain data features. The adapter is used to perform pixel shuffling, add labels, and project the data features output by the visual encoder into the embedding space of the large language model in sequence. The large language model is used to concatenate the output of the adapter with the labeled text annotated using the pre-configured detection knowledge base for historical multimodal data. The concatenated result is input into the MoE architecture, and the processed result is output. The pre-configured detection knowledge base includes descriptive information for various facility data.

[0010] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method for detecting large-scale railway infrastructure.

[0011] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for detecting large-scale railway infrastructure.

[0012] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described method for detecting large-scale railway infrastructure.

[0013] Compared with existing railway infrastructure detection schemes, this invention acquires real-time multimodal data of railway infrastructure. This real-time multimodal data includes image data and / or time-series data. The real-time multimodal data is input into a pre-trained railway infrastructure anomaly detection model, which outputs anomaly detection results. The railway infrastructure anomaly detection model is obtained by training a large-scale railway infrastructure anomaly detection model network using historical multimodal data and a pre-configured detection knowledge base. The large-scale railway infrastructure anomaly detection model network includes a visual encoder, an adapter, and a large language model connected in sequence. The visual encoder is used to divide the image data in the received input data into blocks using a dynamic block-segmentation strategy. The image data, after being segmented, and other data from the input data are processed by the ViT layer to obtain data features. The adapter is used to sequentially perform pixel shuffling, add labels, and project the data features output by the visual encoder onto the embedding space of the large language model. The large language model is used to concatenate the output of the adapter with the labeled text that uses a pre-configured detection knowledge base to annotate historical multimodal data. The concatenation result is input into the MoE architecture, and the processing result is output. The pre-configured detection knowledge base includes descriptive information of various facility data, which can improve the accuracy and efficiency of railway infrastructure detection, reduce the cost of railway infrastructure detection, and provide comprehensive and operable data support for subsequent maintenance and decision-making. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0015] Figure 1 This is a flowchart of the large-scale model method for railway infrastructure detection in an embodiment of the present invention;

[0016] Figure 2 This is a schematic diagram of the structure of the large-scale model network for anomaly detection of railway infrastructure in an embodiment of the present invention;

[0017] Figure 3 This is a schematic diagram of the structure of the large language model in an embodiment of the present invention;

[0018] Figure 4 This is a logical structure diagram of the large-scale model network for anomaly detection of railway infrastructure in this embodiment of the invention;

[0019] Figure 5 This is a schematic diagram of the anomaly detection results in an embodiment of the present invention;

[0020] Figure 6 This is a schematic diagram of the structure of the large-scale railway infrastructure detection system in an embodiment of the present invention;

[0021] Figure 7 This is a schematic diagram of the large-scale railway infrastructure inspection device in an embodiment of the present invention;

[0022] Figure 8 This is a schematic diagram of the computer device structure in an embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0024] In recent years, with the development of artificial intelligence technology, large-scale visual models, large-scale temporal models, and large-scale language models have shown significant advantages in image understanding, trend prediction, semantic recognition, and cross-modal association. These models, trained jointly with image, waveform, and text data, possess stronger semantic understanding and scene generalization capabilities, and can identify and describe target anomalies without requiring a large amount of labeled data.

[0025] To address the aforementioned issues, it is urgent to introduce large-scale models into railway infrastructure anomaly detection, combining visual information, temporal features, and semantic knowledge to improve the accuracy of anomaly detection and provide foundational support for the automated classification of anomaly types and the generation of textual information. This will help achieve intelligent, efficient, and low-cost railway infrastructure operation and maintenance management.

[0026] The technical problem to be solved by the embodiments of the present invention is how to improve the accuracy and efficiency of railway infrastructure inspection, reduce the cost of railway infrastructure inspection, provide comprehensive and operable data support for subsequent maintenance and decision-making, and provide a large model architecture and system suitable for intelligent inspection of railway power supply infrastructure. Figure 1 This is a flowchart of the large-scale model method for railway infrastructure detection in an embodiment of the present invention, such as... Figure 1 As shown, the method may include:

[0027] Step 101: Obtain real-time multimodal data of railway infrastructure; real-time multimodal data includes image data and / or time-series data;

[0028] Step 102: Input real-time multimodal data into a pre-trained railway infrastructure anomaly detection model and output anomaly detection results. The railway infrastructure anomaly detection model is obtained by training a large-scale railway infrastructure anomaly detection model network using historical multimodal data and a pre-configured detection knowledge base. The large-scale railway infrastructure anomaly detection model network includes a visual encoder, an adapter, and a large language model connected in sequence. The visual encoder is used to divide the image data in the received input data into blocks using a dynamic block-splitting strategy. The block-splitting image data and other data in the input data are processed by the ViT layer to obtain data features. The adapter is used to perform pixel shuffling, add labels, and project the data features output by the visual encoder into the embedding space of the large language model in sequence. The large language model is used to concatenate the output of the adapter with the labeled text annotated using the pre-configured detection knowledge base for historical multimodal data. The concatenation result is input into the MoE architecture, and the processing result is output. The pre-configured detection knowledge base includes descriptive information for various facility data.

[0029] The method provided in this embodiment of the invention can improve the accuracy and efficiency of railway infrastructure inspection through the above steps, reduce the cost of railway infrastructure inspection, and provide comprehensive and operable data support for subsequent maintenance and decision-making.

[0030] This invention addresses the technical problems in railway infrastructure inspection, such as the susceptibility of images and time-series data to noise interference in complex environments, the significant gap between structured prompts and feature semantics (e.g., deviation in mapping defect types to spatial locations), and the fragmentation of domain prior knowledge (e.g., lack of systematic modeling of standards / fault cases). It proposes a large-scale intelligent inspection model architecture and system for railway infrastructure. This invention innovatively proposes: (1) a large-scale intelligent inspection model architecture for railway infrastructure encompassing a visual model, a time-series model, and a large language model; (2) efficient methods for domain adaptation, knowledge base access, and parameter fine-tuning, training, and optimization of the large-scale intelligent inspection model for railway infrastructure; and (3) application methods for the large-scale intelligent inspection model system for railway infrastructure.

[0031] The following six parts will be introduced: Part 1: Data Preparation; Part 2: Large Model Architecture Design; Part 3: Domain Adaptation; Part 4: Efficient Fine-tuning of Large Model Parameters; Part 5: Large Model Training and Optimization; Part 6: Model Output and System Application.

[0032] Part 1: Data Preparation

[0033] In one embodiment, acquiring real-time multimodal data of railway infrastructure may include: using a multispectral optical sensor array, multi-source vibration sensor, multi-source pressure sensor and / or multi-source displacement sensor installed on a detection vehicle to acquire real-time multimodal data of railway infrastructure.

[0034] In one embodiment, real-time multimodal data of railway infrastructure is acquired using a multispectral optical sensor array, multi-source vibration sensor, multi-source pressure sensor, and / or multi-source displacement sensor installed on an inspection vehicle. This includes: acquiring panoramic image data and / or time-series data of railway infrastructure under various environments using the multispectral optical sensor array, multi-source vibration sensor, multi-source pressure sensor, and / or multi-source displacement sensor installed on the inspection vehicle; the various environments include various lighting conditions and / or various weather conditions.

[0035] First, multimodal data acquisition, including image data, time-frequency data, and time-series data, is conducted: A multispectral optical sensor array and multi-source vibration, pressure, and displacement sensors installed on specialized inspection vehicles are used to systematically collect panoramic images and time-series data of railway infrastructure. The data collection scope covers key railway facilities in the track maintenance, signaling, and power supply systems. To ensure data diversity and representativeness, the data includes different lighting conditions and various weather environments, forming a large-scale multimodal database that comprehensively reflects the actual operating environment of railway facilities.

[0036] In one embodiment, before inputting real-time multimodal data into a pre-trained railway infrastructure anomaly detection model and outputting anomaly detection results and a detection report, the method may further include: acquiring historical multimodal data; classifying the historical multimodal data to determine normal data and anomaly data; determining normal descriptive information of the railway infrastructure based on the normal data; the normal descriptive information includes one or any combination of the railway infrastructure's normal operating state characteristics, standard parameter range, and normal wear tolerance; determining anomaly descriptive information of the railway infrastructure based on the anomaly data; the anomaly descriptive information includes one or any combination of anomaly type, anomaly coordinates, anomaly severity, safety risk, and anomaly cause; organizing the normal and anomaly descriptive information into structured prompt text format; and configuring a detection knowledge base based on the structured prompt text format of the normal and anomaly descriptive information.

[0037] The configuration involves building a detection knowledge base (structured cue text dataset): Detailed analysis of the collected multimodal data is conducted manually or using artificial intelligence. For normal (facility) data: A detailed description of the equipment's normal operating status characteristics, standard parameter ranges, and normal wear tolerances is provided. For abnormal (facility) data: Accurate labeling of the abnormality type, abnormal location coordinates, abnormality severity, potential safety risks, and an analysis of the abnormality's causes are provided. This descriptive information is then organized into structured cue text format. A large-scale structured cue text dataset is constructed for use as input for text modality information in subsequent large-scale model training.

[0038] Part Two: Large Model Architecture Design

[0039] Figure 2 This is a schematic diagram of the structure of the large-scale model network for anomaly detection of railway infrastructure in an embodiment of the present invention, as shown below. Figure 2 As shown, in one embodiment, the railway infrastructure anomaly detection large model network includes a visual encoder, an adapter, and a large language model connected in sequence.

[0040] A high-resolution image input visual encoder for railway infrastructure is used to segment image data from received input data into blocks using a dynamic segmentation strategy. This dynamic segmentation strategy adaptively segments the image data, intelligently allocating different numbers of local blocks based on image content complexity and regions of interest.

[0041] The segmented image data and other data from the input data are processed through the ViT layer to obtain data features and extract rich features.

[0042] The ViT layer can include a Patch Embedding layer, an N-layer VisionTransfommer layer for visual attention, and an Output Projection layer.

[0043] The visual attention mechanism layer and the output feature projection layer are connected in sequence by: multi-head self-attention, layer normalization, and feedforward network.

[0044] Output features of the projection layer: visual feature output, global features, and local features.

[0045] The adapter is used to sequentially perform pixel shuffling, add labels, and project the data features output by the visual encoder into the embedding space of the large language model. The large language model is used to concatenate the output of the adapter with the labeled text that uses a pre-configured detection knowledge base to annotate historical multimodal data, input the concatenation result into the MoE architecture, and output the processing result.

[0046] Pixel shuffling is used to reduce sequence length while retaining information. Adding tags can add special tags and markers. Projection projects visual features into the LLM (Large Language Model) embedding space. The output is: visual tag sequence global features + local features + special tags.

[0047] like Figure 2As shown, in one embodiment, the visual encoder is specifically used to divide the image data in the received input data into blocks using a dynamic block division strategy, which are divided into a global thumbnail and multiple local images; the MoE architecture includes a multi-layer attention mechanism network and a multi-head latent attention mechanism layer.

[0048] The adapter is a vision-language adapter, which implements feature transformation and alignment.

[0049] like Figure 2 As shown, the input processing layer in the Large Language Model (MoE) processes the input, concatenating the visual tags with the annotated text from historical multimodal data, which is then labeled using a pre-configured detection knowledge base, as the text input. The MoE architecture includes an N-layer attention mechanism network, a Multi-head Latent Attention (MLA) layer, an expert routing mechanism, and an output processing layer. The expert routing mechanism employs a Top-2 expert selection strategy, with N experts per layer and dynamic load balancing. The output processing layer includes output projection and text generation. This entire process achieves a seamless conversion from multimodal information to language understanding and generation, enabling the model to efficiently handle various vision-language tasks.

[0050] Figure 3 This is a schematic diagram of the structure of the large language model in an embodiment of the present invention, such as... Figure 3 As shown, the text input in the large language model is a detection knowledge base obtained through standard specifications, historical faults, and historical experience.

[0051] Figure 4 This is a logical structure diagram of the large-scale model network for anomaly detection of railway infrastructure in an embodiment of the present invention, such as... Figure 4 As shown, basic data management is carried out through a railway infrastructure inspection and monitoring data platform with functions of data collection, data governance, computing power construction, and application deployment. This includes acquiring image data or time series data from professional inspection vehicles at the intelligent sensing layer, as well as constructing a multimodal data lake and an inspection knowledge base.

[0052] A multimodal large-scale model, a natural language large-scale model, a visual large-scale model, and a temporal series large-scale model were selected as the base models for the L0 layer-base layer of the large-scale model for railway infrastructure anomaly detection. The model exhibits strong capabilities in visual, temporal, and language processing, effectively handling the fusion of multimodal images, temporal signals, and textual information. It possesses efficient image and signal feature extraction modules, enabling in-depth feature mining of input railway infrastructure detection images and temporal signal data. Simultaneously, its language processing component can understand and process the textual descriptions of railway infrastructure within structured prompts.

[0053] The L1-domain layer empowerment model utilizes the L0-base layer basic model and basic data to perform equipment malfunction identification, change pattern analysis, development trend prediction, health status management, and auxiliary maintenance decision-making.

[0054] The model is applied at the L2-scenario layer to perform appearance anomaly identification, waveform anomaly identification, minute change analysis, seasonal pattern analysis, track settlement / bulging early warning, track quality index (TQI index) trend prediction, health status assessment, life cycle management, and intelligent attribution of defects and generation of maintenance solutions.

[0055] Through fine-tuning and adaptation of the aforementioned large model and user interaction verification, a large model architecture and system for intelligent detection of railway infrastructure were finally formed.

[0056] Part Three: Domain Adaptation

[0057] like Figure 3 As shown, an integrated detection knowledge base is constructed, incorporating vectorized standard specifications, historical fault cases, and expert experience. Domain-prior knowledge is injected through prompt engineering to guide the model in generating detection conclusions consistent with industry logic, achieving domain adaptation for the large model. When using structured prompt text retrieval for relevant information, the model can automatically retrieve relevant defect information, combining prompt information with input images and signals to achieve feature alignment, significantly improving defect recognition level and professionalism.

[0058] Figure 3 The detection knowledge base displayed in the right-hand area is a key enhancement module developed for a specific domain. It is directly connected to the text input part of the large language model via arrows, realizing efficient retrieval and generation enhancement technology.

[0059] This knowledge base cleverly integrates three key knowledge sources: standards and specifications, historical failure cases, and historical experience, forming a complete knowledge integration mechanism. This design enables the system to proactively retrieve relevant professional knowledge before generating a response, significantly improving the accuracy and reliability of the model in professional fields such as railway facility inspection. It successfully combines advanced visual recognition capabilities with in-depth professional domain knowledge, making it particularly suitable for railway infrastructure defect detection scenarios with extremely high requirements for professionalism and accuracy.

[0060] Part Four: Efficient Fine-Tuning of Large Model Parameters

[0061] In one embodiment, the railway infrastructure anomaly detection large model network also includes multiple fine-tuning modules for fine-tuning the parameters of the visual encoder, adapter, and large language model, respectively.

[0062] In one embodiment, a large-scale model network for railway infrastructure anomaly detection is trained using historical multimodal data and corresponding labeled text to generate a railway infrastructure anomaly detection model. This includes: pre-training the large-scale model network using historical multimodal data and corresponding labeled text; and then using a fine-tuning module with PEFT (Parameter-Efficient Fine-Tuning) technology to fine-tune the parameters of the pre-trained model according to the following formula to obtain the railway infrastructure anomaly detection model:

[0063] W = W0 + ΔW;

[0064] Where W represents the parameters of the model after fine-tuning, W0 represents the original parameters of the model, and ΔW represents the model fine-tuning update parameters.

[0065] The prepared multimodal data and corresponding structured cue text are combined to form training sample pairs, which are then input into the railway infrastructure anomaly detection model. Considering the complexity of domain knowledge and the efficient use of computational resources, the method employs efficient parameter fine-tuning techniques as its core strategy. These fine-tuning methods keep most of the weights of the pre-trained model unchanged, updating only a small number of key parameters. The calculation method for this type of method is as described in the formula above.

[0066] Through iterative training with a large number of training samples, the model gradually learns specific knowledge and anomaly detection patterns in the railway infrastructure field. For example, during training, the model can learn the correlation between image features of a normal track appearance and corresponding descriptive text, as well as the changing patterns of image features and descriptive text when anomalies occur.

[0067] As described in this embodiment of the invention, the PEFT fine-tuning method mainly functions in three core stages: First, a fine-tuning parameter module is added to the self-attention layer of the visual encoder to adjust the visual feature extraction capability; second, the fine-tuning parameter module is applied to the feature projection layer of the vision-language adapter to optimize the mapping of visual information to the language space; finally, the fine-tuning parameter module is selectively applied to expert routing and attention mechanisms in the MoE layer of the large language model. This maintains the model's high performance on specific vision-language tasks while updating only a small number of model parameters, enabling the model to better adapt to the data and task requirements of the railway infrastructure field.

[0068] Part 5: Large Model Training and Optimization

[0069] In one embodiment, before inputting real-time multimodal data into a pre-trained railway infrastructure anomaly detection model and outputting anomaly detection results and a detection report, the method may further include: acquiring historical multimodal data and the detection knowledge base; using the detection knowledge base to annotate the historical multimodal data with text; and using the historical multimodal data and the corresponding annotated text to train a large-scale railway infrastructure anomaly detection model network to generate a railway infrastructure anomaly detection model.

[0070] In one embodiment, the large-scale model network for railway infrastructure anomaly detection is pre-trained using historical multimodal data and corresponding labeled text. This includes: using historical multimodal data and corresponding labeled text, a cosine annealing learning rate adjustment strategy is adopted to dynamically adjust the learning rate so that the learning rate gradually decreases from a preset initial value; and the large-scale model network for railway infrastructure anomaly detection is pre-trained using gradient accumulation training technology, mixed precision training technology, weight decay algorithm, and LoRA (Low-Rank Adaptation) regularization algorithm.

[0071] In one embodiment, after obtaining the railway infrastructure anomaly detection model, the method may further include: validating the railway infrastructure anomaly detection model using a validation set; employing an early stopping mechanism, whereby training is stopped when the model performance data obtained from the validation set fails to improve for multiple consecutive rounds, and the currently trained model is determined to be a well-trained railway infrastructure anomaly detection model; the model performance data includes one or any combination of loss function, accuracy, and precision.

[0072] Data preprocessing:

[0073] During the model training and optimization phases, the historical multimodal data, namely the constructed railway facility anomaly detection dataset, is first rigorously divided into a training set for parameter updates, a validation set for monitoring the training process and preventing overfitting, and a test set for finally evaluating the model's generalization ability. To enhance the model's robustness and simulate real-world, variable environments, the training data undergoes multiple image enhancement processes, including multi-scale random cropping, small-angle rotation, and brightness / contrast adjustments. Simultaneously, a stratified sampling strategy is employed to ensure a balanced distribution of training samples across different railway facility categories and defect types, preventing the model from underlearning for a few categories. Regarding text labeling, defect descriptions are standardized and terminology is unified, and structured output templates are designed to lay the foundation for generating high-quality, standardized reports.

[0074] Training process:

[0075] The model's training process integrates multiple efficient strategies to improve efficiency and effectiveness. A cosine annealing learning rate scheduling curve is employed to dynamically adjust the learning rate, gradually decreasing it from its initial value. Gradient accumulation and mixed-precision training techniques are combined to effectively reduce GPU memory usage while expanding training batches, thus more stably approaching the global optimum and accelerating convergence. Furthermore, weight decay (typically 1e-2) is introduced, combined with LoRA-specific regularization methods, to suppress overfitting on the training set and improve the model's generalization ability. To prevent ineffective training and wasted computational resources, the system implements an early stopping mechanism: training stops when performance on the validation set (such as the loss function or key evaluation metrics) fails to show significant improvement for five consecutive rounds, locking in the current optimal model parameter state.

[0076] Part Six: Model Output and System Application

[0077] In one embodiment, the anomaly detection result is a detection report; the detection report includes one or any combination of the following: image data annotated with the anomaly location, anomaly type, textual description of the anomaly location, anomaly severity classification, potential security risks, maintenance recommendations, and maintenance time limit requirements.

[0078] In one embodiment, the railway infrastructure detection large model method may further include: acquiring user-input demand text data; inputting the demand text data into a pre-trained railway infrastructure anomaly detection model; and outputting a response text.

[0079] Figure 5 This is a schematic diagram of the anomaly detection results in an embodiment of the present invention. Output format: The trained and optimized model can detect multimodal data of the input railway infrastructure and generate multi-dimensional professional output. The core output format includes intuitive defect visualization and response text on the original image. By accurately drawing corresponding bounding boxes, the abnormal railway facility parts in the image are accurately defined, such as... Figure 5 As shown, Figure 5 The defect is a missing fastener in a certain engineering inspection data.

[0080] Simultaneously, the large language model generates a detailed and industry-standard professional inspection report. This report not only includes a textual description of the defect type and its precise spatial location but also classifies the severity of the defect according to railway industry standards. Furthermore, the report assesses potential operational safety risks and provides specific maintenance recommendations and clear timeframes (e.g., "replacement recommended within 48 hours"). When necessary, it also cites relevant standards and specifications as a basis for judgment, thus providing comprehensive and actionable intelligent support for subsequent maintenance and decision-making.

[0081] The embodiments of the present invention can be applied to sections of high-speed rail, conventional rail, heavy-haul rail and urban rail lines.

[0082] Figure 6 This is a schematic diagram of the structure of the large-scale railway infrastructure detection system in an embodiment of the present invention, as shown below. Figure 6 As shown, the railway infrastructure inspection large-scale model system can be divided into four parts: intelligent application, model interaction, system base, and infrastructure. The infrastructure part mainly completes the dynamic inspection of multi-source, multi-modal data acquisition, using high-performance servers + GPU (Graphics Processing Unit) computing power. The system base includes: inspection and monitoring data platform, large-scale model optimization, training, and deployment; the inspection and monitoring data platform includes: multi-source inspection data collection (ledger data, inspection data, knowledge base data, operation and maintenance data), and large-scale model processing (distributed storage, data cleaning, data standardization, and data visualization); large-scale model optimization includes data annotation, data augmentation, training data construction, model fine-tuning, and performance evaluation; training and deployment include performance evaluation, knowledge augmentation, and illusion elimination. The model interaction comprises a human-computer interaction component, a model component, and a physical system component. The human-computer interaction component includes: requirement input, client, and answer output. The model component includes a knowledge system for managing human experience reports, standards, regulations, and rules; a large language model; expert modules for providing prompts and translations to the large language model (providing visual models for detection data analysis, 1-D signal intelligent analysis, early warning, and multimodal models; statistical indicator libraries, trend analysis, comprehensive evaluation, and report regulation templates for data statistical analysis; and detection agent reasoning, data feature pattern extraction, data association mining, and heterogeneous data multimodal fusion); and decision support for dedicated training with the large language model, including data analysis reports and maintenance plan generation. The knowledge system performs retrieval-enhanced generation (RAG) with the large language model, and the physical system interacts with the model component via control commands. Intelligent applications provide detection knowledge Q&A, intelligent detection production assistance, Q&A intelligent assistance, and a detection knowledge training system, enabling rapid orchestration of business applications and providing flexible intelligent interactive applications. All of the above systems can be deployed and operated privately.

[0083] In summary, compared with existing railway infrastructure detection schemes, the embodiments of the present invention acquire real-time multimodal data of railway infrastructure; the real-time multimodal data includes image data and / or time-series data; the real-time multimodal data is input into a pre-trained railway infrastructure anomaly detection model, and anomaly detection results are output; the railway infrastructure anomaly detection model is obtained by training a large-scale railway infrastructure anomaly detection model network using historical multimodal data and a pre-configured detection knowledge base; the large-scale railway infrastructure anomaly detection model network includes a visual encoder, an adapter, and a large language model connected in sequence; the visual encoder is used to process the image data in the received input data using a dynamic block partitioning strategy. The image data is segmented and processed by the ViT layer along with other data from the input data to obtain data features. The adapter is used to sequentially perform pixel shuffling, add labels, and project the data features output by the visual encoder onto the embedding space of the large language model. The large language model is used to concatenate the output of the adapter with the labeled text that has been annotated with historical multimodal data using a pre-configured detection knowledge base. The concatenation result is input into the MoE architecture, and the processing result is output. The pre-configured detection knowledge base includes descriptive information of various facility data, which can improve the accuracy and efficiency of railway infrastructure detection, reduce the cost of railway infrastructure detection, and provide comprehensive and operable data support for subsequent maintenance and decision-making.

[0084] The key technical points of this invention are:

[0085] (1) Intelligent detection large model architecture and cross-modal fusion technology for railway infrastructure:

[0086] A four-level serial architecture of "visual encoder - temporal signal features - language adapter - large language model" is constructed to realize cross-modal fusion of railway facility images, signal features and structured prompt words.

[0087] By associating defect types with structured prompts and the coordinates of bounding boxes in images, a closed-loop reasoning chain of "text instructions - image features - signal features - spatial localization" is formed, breaking through the bottleneck of the separation of semantics and data information in traditional methods.

[0088] (2) Deep injection and adaptation technology of knowledge in the field of railway infrastructure:

[0089] By integrating standards and specifications, historical failure cases, and expert experience, a structured domain knowledge base is formed. This knowledge base is then directly integrated with the text input layer of the large language model through prompting engineering to achieve industry-specific logical constraints on the test results.

[0090] Domain-Adaptive Parameter Efficient Fine-Tuning (PEFT) proposes a PEFT fine-tuning strategy for railway scenarios, which updates only the parameters of key model modules to achieve adaptation to the railway infrastructure detection domain.

[0091] (3) Feature extraction and prediction techniques for large time series models:

[0092] A multi-scale feature fusion network based on a spatiotemporal attention mechanism captures the local dependencies and global trends of time-series data through dynamic weight allocation;

[0093] An adaptive hierarchical encoder utilizes a sparse self-attention module to reduce the computational complexity of sequences while preserving key temporal patterns.

[0094] A multi-task joint prediction framework, combining adversarial training and uncertainty quantification modules, enhances the robustness of time series prediction models.

[0095] By extracting features from time-series signal detection data of railway infrastructure, analyzing patterns such as minute changes and seasonal variations, we can support trend prediction of the Track Quality Index (TQI) and early warning of track settlement and track bulging.

[0096] (4) Multi-source integrated infrastructure testing, evaluation, and result output:

[0097] Image level: Defect locations are precisely marked using bounding boxes, supporting rapid localization;

[0098] Signal level: Defect mileage location and frequency are accurately marked through waveform time and frequency characteristics, supporting rapid positioning;

[0099] At the text level: Generate inspection reports that conform to industry standards, including defect type, severity and safety risk assessment, repair recommendations and time limits, and standard references.

[0100] Cross-system compatibility: Supports testing of facilities from multiple disciplines such as civil engineering, electrical engineering, and power supply. By expanding the prompt word tags, it can quickly adapt to the testing needs of different types of equipment.

[0101] The core technical points of this invention are:

[0102] (1) Large model architecture for cross-modal detection: a four-level serial structure including a visual encoder, a signal feature attempt-language adapter and a large language model, and its specific application in railway infrastructure detection.

[0103] (2) The method of combining domain knowledge base with prompting engineering: the construction method of railway facility defect knowledge base, the integration architecture with large language model, and the mechanism of retrieving knowledge through prompt words and guiding the conclusion of railway infrastructure inspection.

[0104] (3) Parameter Efficient Fine-Tuning (PEFT) Strategy: A method for updating key module fine-tuning parameters and optimizing computational efficiency in railway infrastructure inspection scenarios.

[0105] (4) Structured output format and report generation logic: based on <|ref|><det|> The format of railway infrastructure defect location results with labels such as "T" and "Professional report generation rules including industry standard references".

[0106] This invention addresses technical problems in railway infrastructure inspection, such as the susceptibility of images and time-series data to noise interference in complex environments, the significant gap between structured prompts and feature semantics (e.g., deviation in mapping defect type and spatial location), and the fragmentation of domain prior knowledge (e.g., lack of systematic modeling of standards / fault cases). It proposes a large-scale intelligent inspection model architecture and system for railway infrastructure.

[0107] The methods and systems of this invention can overcome the limitations of traditional single-modal detection (such as the lack of textual semantic understanding in pure vision, the lack of image spatial localization in pure text, and the lack of equipment graphic information in pure temporal series) and the low computational efficiency of full parameter fine-tuning of large-scale models. By constructing a multimodal large model architecture that integrates image and textual semantics and temporal feature trends, accessing a railway facility defect knowledge base to achieve domain adaptation, and using parameter efficient fine-tuning (PEFT) technology to optimize feature extraction and cross-modal mapping, this invention achieves high-precision positioning, multi-dimensional assessment, and intelligent report generation of railway infrastructure anomalies in complex scenarios, providing railway operation and maintenance with technical support that combines accuracy, robustness, and industry standardization.

[0108] This invention also proposes a large-scale railway infrastructure inspection device, the principle of which is similar to the large-scale railway infrastructure inspection method, and will not be described in detail here.

[0109] Figure 7 This is a schematic diagram of a large-scale railway infrastructure inspection device in an embodiment of the present invention, such as... Figure 7 As shown, the large-scale railway infrastructure inspection device includes:

[0110] The acquisition module 701 is used to acquire real-time multimodal data of railway infrastructure; the real-time multimodal data includes image data and / or time series data;

[0111] Output module 702 is used to input real-time multimodal data into a pre-trained railway infrastructure anomaly detection model and output anomaly detection results. The railway infrastructure anomaly detection model is obtained by training a large-scale railway infrastructure anomaly detection model network using historical multimodal data and a pre-configured detection knowledge base. The large-scale railway infrastructure anomaly detection model network includes a visual encoder, an adapter, and a large language model connected in sequence. The visual encoder is used to divide the image data in the received input data into blocks using a dynamic block-splitting strategy. The block-splitting image data and other data in the input data are processed by the ViT layer to obtain data features. The adapter is used to perform pixel shuffling, add labels, and project the data features output by the visual encoder into the embedding space of the large language model in sequence. The large language model is used to concatenate the output of the adapter with the labeled text annotated using the pre-configured detection knowledge base for historical multimodal data, input the concatenation result into the MoE architecture, and output the processing result. The pre-configured detection knowledge base includes descriptive information for various facility data.

[0112] In one embodiment, the acquisition module 701 is specifically used for:

[0113] Real-time multimodal data of railway infrastructure is acquired by using multispectral optical sensor arrays, multi-source vibration sensors, multi-source pressure sensors and / or multi-source displacement sensors installed on inspection vehicles.

[0114] In one embodiment, the acquisition module 701 is specifically used for:

[0115] Using a multispectral optical sensor array, multi-source vibration sensor, multi-source pressure sensor and / or multi-source displacement sensor installed on the inspection vehicle, panoramic image data and / or time-series data of railway infrastructure under various environments are collected; the various environments include various lighting conditions and / or various weather conditions.

[0116] In one embodiment, the large-scale railway infrastructure inspection device further includes: an inspection knowledge base configuration module, used for:

[0117] Acquire historical multimodal data;

[0118] Classify historical multimodal data to identify normal and abnormal data;

[0119] Based on normal data, determine the normal descriptive information of railway infrastructure; the normal descriptive information includes: one or any combination of the normal working condition characteristics of railway infrastructure, standard parameter range, and normal wear tolerance.

[0120] Based on the abnormal data, determine the abnormal description information of the railway infrastructure; the abnormal description information includes one or any combination of the following: abnormal type, abnormal coordinates, abnormal severity, safety risk and abnormal cause;

[0121] Organize normal and abnormal description information into a structured prompt text format;

[0122] Configure a detection knowledge base based on normal and abnormal description information in the form of structured prompt words.

[0123] In one embodiment, the large-scale railway infrastructure detection device further includes: a training module for:

[0124] Acquire historical multimodal data and the aforementioned detection knowledge base;

[0125] Text annotation of historical multimodal data was performed using a detection knowledge base;

[0126] By using historical multimodal data and corresponding labeled text, a large-scale model network for railway infrastructure anomaly detection is trained to generate a railway infrastructure anomaly detection model.

[0127] In one embodiment, the visual encoder is specifically used to divide the image data in the received input data into blocks using a dynamic block division strategy, which are divided into a global thumbnail and multiple local images; the MoE architecture includes a multi-layer attention mechanism network and a multi-head latent attention mechanism layer.

[0128] In one embodiment, the railway infrastructure anomaly detection large model network also includes multiple fine-tuning modules for fine-tuning the parameters of the visual encoder, adapter, and large language model, respectively.

[0129] In one embodiment, the training module is specifically used for:

[0130] We pre-trained a large-scale model network for anomaly detection in railway infrastructure using historical multimodal data and corresponding labeled text.

[0131] The fine-tuning module employs the PEFT technique for efficient parameter fine-tuning, and the pre-trained model is fine-tuned according to the following formula to obtain the railway infrastructure anomaly detection model:

[0132] W = W0 + ΔW;

[0133] Where W represents the parameters of the model after fine-tuning, W0 represents the original parameters of the model, and ΔW represents the model fine-tuning update parameters.

[0134] In one embodiment, the training module is specifically used for:

[0135] Using historical multimodal data and corresponding labeled text, a cosine annealing learning rate adjustment strategy is adopted to dynamically adjust the learning rate, so that the learning rate gradually decreases from the preset initial value. The large model network for railway infrastructure anomaly detection is pre-trained through gradient accumulation training technology, mixed precision training technology, weight decay algorithm and low-level adaptive LoRA regularization algorithm.

[0136] In one embodiment, the large-scale railway infrastructure inspection device further includes: a verification module, used for:

[0137] The railway infrastructure anomaly detection model was validated using a validation set;

[0138] An early stopping mechanism is adopted. When the model performance data obtained by validation using the validation set fails to improve for several consecutive rounds, training is stopped, and the model obtained by training is determined to be a well-trained railway infrastructure anomaly detection model. The model performance data includes one or any combination of loss function, accuracy and precision.

[0139] In one embodiment, the anomaly detection result is a detection report; the detection report includes one or any combination of the following: image data annotated with the anomaly location, anomaly type, textual description of the anomaly location, anomaly severity classification, potential security risks, maintenance recommendations, and maintenance time limit requirements.

[0140] In one embodiment, the large-scale railway infrastructure inspection device further includes: a response text output module, used for:

[0141] Obtain the user's input text data;

[0142] The required text data is input into a pre-trained railway infrastructure anomaly detection model, and the output is a response text.

[0143] In summary, in the apparatus proposed in this embodiment of the invention, the acquisition module is used to acquire real-time multimodal data of railway infrastructure; the real-time multimodal data includes image data and / or time-series data; the output module is used to input the real-time multimodal data into a pre-trained railway infrastructure anomaly detection model and output anomaly detection results; the railway infrastructure anomaly detection model is obtained by training a large-scale railway infrastructure anomaly detection model network using historical multimodal data and a pre-configured detection knowledge base; the large-scale railway infrastructure anomaly detection model network includes a visual encoder, an adapter, and a large language model connected in sequence; the visual encoder is used to process the image data in the received input data using a dynamic block partitioning strategy. The image data is segmented and processed by the ViT layer along with other data from the input data to obtain data features. The adapter is used to sequentially perform pixel shuffling, add labels, and project the data features output by the visual encoder onto the embedding space of the large language model. The large language model is used to concatenate the output of the adapter with the labeled text that has been annotated with historical multimodal data using a pre-configured detection knowledge base. The concatenation result is input into the MoE architecture, and the processing result is output. The pre-configured detection knowledge base includes descriptive information of various facility data, which can improve the accuracy and efficiency of railway infrastructure detection, reduce the cost of railway infrastructure detection, and provide comprehensive and operable data support for subsequent maintenance and decision-making.

[0144] The beneficial effects of this invention are as follows: it deeply integrates image visual features, temporal signal features, and textual semantic information, breaking through the technical bottleneck of traditional single-modal detection in railway infrastructure defect identification. Through cross-modal collaboration of visual encoder, temporal feature extraction, and large language model, it achieves deep feature mining and semantic understanding of structured prompts for multimodal infrastructure detection data in complex environments. It effectively solves the problems of feature blurring caused by illumination changes, occlusion, and strong background noise interference, as well as the text-image-signal-semantic mapping deviation, achieving accurate cross-modal information fusion for "defect localization-type identification-risk assessment".

[0145] The deep integration of the railway infrastructure defect knowledge base and the efficient parameter fine-tuning technology significantly improve the model's domain specialization capabilities and training efficiency: the former guides the model to generate detection conclusions that conform to industry logic by integrating standard specifications, historical cases and historical experience, solving the problem of fragmented domain knowledge in traditional methods; the latter improves the adaptability of large models to railway infrastructure anomaly identification problems by freezing the main parameters and updating only the key modules, thereby reducing training costs.

[0146] Through a fully automated analysis mechanism based on a large model, end-to-end processing is achieved, from image acquisition and signal feature extraction to multi-dimensional report generation. The model can not only accurately mark anomaly locations using bounding boxes, but also generate professional reports based on a knowledge base, including safety risk assessments, maintenance recommendations, and time limits, significantly improving the scientific rigor and efficiency of operational decisions.

[0147] Based on the aforementioned inventive concept, such as Figure 8 As shown, the present invention also proposes a computer device 800, including a memory 810, a processor 820, and a computer program 830 stored in the memory 810 and executable on the processor 820. When the processor 820 executes the computer program 830, it implements the aforementioned railway infrastructure detection large model method.

[0148] Based on the aforementioned inventive concept, this invention proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned large-scale model method for railway infrastructure detection.

[0149] Based on the aforementioned inventive concept, this invention proposes a computer program product, which includes a computer program that, when executed by a processor, implements a large-scale model method for railway infrastructure detection.

[0150] The acquisition, storage, use, and processing of data in this application comply with relevant laws and regulations.

[0151] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0153] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0154] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0155] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A large-scale model method for railway infrastructure inspection, characterized in that, include: Acquire real-time multimodal data of railway infrastructure; The real-time multimodal data includes image data and / or time-series data; The real-time multimodal data is input into a pre-trained railway infrastructure anomaly detection model, and the anomaly detection results are output. The railway infrastructure anomaly detection model is obtained by training a large-scale railway infrastructure anomaly detection model network using historical multimodal data and a pre-configured detection knowledge base. The large model network for anomaly detection in railway infrastructure includes a visual encoder, an adapter, and a large language model connected in sequence. The visual encoder is used to divide the image data in the received input data into blocks using a dynamic block division strategy. The block image data and other data in the input data are processed by the visual self-attention mechanism ViT layer to obtain data features. The adapter is used to sequentially perform pixel shuffling, add tags, and project the data features output by the visual encoder into the embedding space of the large language model. The large language model is used to concatenate the adapter's output with labeled text that uses a pre-configured detection knowledge base to annotate historical multimodal data. The concatenated result is input into the hybrid expert model (MoE) architecture, and the processed result is output. The pre-configured detection knowledge base includes descriptive information for various facility data. A dynamic block segmentation strategy adaptively segments the image data, intelligently allocating different numbers of local blocks based on the image content complexity and regions of interest. Pixel shuffling is used to reduce sequence length while retaining information. The projection is used to project visual features into the embedding space of the large language model. The MoE architecture includes an N-layer attention mechanism network, a multi-head latent attention mechanism layer, an expert routing mechanism, and an output processing layer. The expert routing mechanism includes a Top-2 expert selection strategy. Before inputting the real-time multimodal data into the pre-trained railway infrastructure anomaly detection model and outputting the anomaly detection results and report, the process also includes: Acquire historical multimodal data; classify the historical multimodal data to determine normal and abnormal data; based on the normal data, determine the normal description information of the railway infrastructure; the normal description information includes one or any combination of the normal working state characteristics, standard parameter range, and normal wear tolerance of the railway infrastructure; based on the abnormal data, determine the abnormal description information of the railway infrastructure; the abnormal description information includes one or any combination of the abnormal type, abnormal coordinates, abnormal severity, safety risk, and abnormal cause; organize the normal and abnormal description information into a structured prompt text format; configure a detection knowledge base based on the structured prompt text format of the normal and abnormal description information.

2. The method as described in claim 1, characterized in that, Acquire real-time multimodal data of railway infrastructure, including: Real-time multimodal data of railway infrastructure is acquired by using multispectral optical sensor arrays, multi-source vibration sensors, multi-source pressure sensors and / or multi-source displacement sensors installed on inspection vehicles.

3. The method as described in claim 2, characterized in that, Real-time multimodal data of railway infrastructure is acquired using multispectral optical sensor arrays, multi-source vibration sensors, multi-source pressure sensors, and / or multi-source displacement sensors installed on inspection vehicles, including: Using a multispectral optical sensor array, multi-source vibration sensor, multi-source pressure sensor and / or multi-source displacement sensor installed on the inspection vehicle, panoramic image data and / or time-series data of railway infrastructure under various environments are collected; the various environments include various lighting conditions and / or various weather conditions.

4. The method as described in claim 1, characterized in that, Before inputting the real-time multimodal data into the pre-trained railway infrastructure anomaly detection model and outputting the anomaly detection results and report, the process also includes: Acquire historical multimodal data and the aforementioned detection knowledge base; Text annotation of historical multimodal data was performed using a detection knowledge base; By using historical multimodal data and corresponding labeled text, a large-scale model network for railway infrastructure anomaly detection is trained to generate a railway infrastructure anomaly detection model.

5. The method as described in claim 4, characterized in that, The visual encoder is specifically used to divide the image data in the received input data into blocks using a dynamic block division strategy, resulting in a global thumbnail and multiple local images.

6. The method as described in claim 5, characterized in that, The large-scale model network for anomaly detection in railway infrastructure also includes multiple fine-tuning modules for fine-tuning the parameters of the visual encoder, adapter, and large language model.

7. The method as described in claim 6, characterized in that, Using historical multimodal data and corresponding labeled text, a large-scale model network for railway infrastructure anomaly detection is trained to generate a railway infrastructure anomaly detection model, including: We pre-trained a large-scale model network for anomaly detection in railway infrastructure using historical multimodal data and corresponding labeled text. The fine-tuning module employs the PEFT technique for efficient parameter fine-tuning, and the pre-trained model is fine-tuned according to the following formula to obtain the railway infrastructure anomaly detection model: W = W0 + ΔW; Where W represents the parameters of the model after fine-tuning, W0 represents the original parameters of the model, and ΔW represents the model fine-tuning update parameters.

8. The method as described in claim 7, characterized in that, Using historical multimodal data and corresponding labeled text, a large-scale model network for anomaly detection in railway infrastructure was pre-trained, including: Using historical multimodal data and corresponding labeled text, a cosine annealing learning rate adjustment strategy is adopted to dynamically adjust the learning rate, so that the learning rate gradually decreases from the preset initial value. The large model network for railway infrastructure anomaly detection is pre-trained through gradient accumulation training technology, mixed precision training technology, weight decay algorithm and low-level adaptive LoRA regularization algorithm.

9. The method as described in claim 7, characterized in that, After obtaining the railway infrastructure anomaly detection model, the following is also included: The railway infrastructure anomaly detection model was validated using a validation set; An early stopping mechanism is adopted. When the model performance data obtained by validation using the validation set fails to improve for several consecutive rounds, training is stopped, and the model obtained by training is determined to be a well-trained railway infrastructure anomaly detection model. The model performance data includes one or any combination of loss function, accuracy and precision.

10. The method as described in claim 1, characterized in that, The anomaly detection result is a detection report; the detection report includes one or any combination of the following: image data annotated with the anomaly location, anomaly type, text description of the anomaly location, anomaly severity classification, potential safety risks, maintenance recommendations, and maintenance time limit requirements.

11. The method as described in claim 1, characterized in that, Also includes: Obtain the user's input text data; The required text data is input into a pre-trained railway infrastructure anomaly detection model, and the output is a response text.

12. A large-scale model device for railway infrastructure inspection, characterized in that, include: The acquisition module is used to acquire real-time multimodal data of railway infrastructure; The real-time multimodal data includes image data and / or time-series data; The output module is used to input the real-time multimodal data into the pre-trained railway infrastructure anomaly detection model and output the anomaly detection results. The railway infrastructure anomaly detection model is obtained by training a large-scale railway infrastructure anomaly detection model network using historical multimodal data and a pre-configured detection knowledge base. The large model network for anomaly detection in railway infrastructure includes a visual encoder, an adapter, and a large language model connected in sequence. The visual encoder is used to divide the image data in the received input data into blocks using a dynamic block division strategy. The block image data and other data in the input data are processed by the visual self-attention mechanism ViT layer to obtain data features. The adapter is used to sequentially perform pixel shuffling, add tags, and project the data features output by the visual encoder into the embedding space of the large language model. The large language model is used to concatenate the adapter's output with labeled text that uses a pre-configured detection knowledge base to annotate historical multimodal data. The concatenated result is input into the hybrid expert model (MoE) architecture, and the processed result is output. The pre-configured detection knowledge base includes descriptive information for various facility data. A dynamic block segmentation strategy adaptively segments the image data, intelligently allocating different numbers of local blocks based on the image content complexity and regions of interest. Pixel shuffling is used to reduce sequence length while retaining information. The projection is used to project visual features into the embedding space of the large language model. The MoE architecture includes an N-layer attention mechanism network, a multi-head latent attention mechanism layer, an expert routing mechanism, and an output processing layer. The expert routing mechanism includes a Top-2 expert selection strategy. The knowledge base configuration module is used for: Acquire historical multimodal data; classify the historical multimodal data to determine normal and abnormal data; Based on normal data, determine the normal description information of the railway infrastructure; the normal description information includes one or any combination of the normal working status characteristics of the railway infrastructure, standard parameter range, and normal wear tolerance. Based on abnormal data, determine the abnormal description information of the railway infrastructure; the abnormal description information includes one or any combination of the abnormality type, abnormal coordinates, abnormality severity, safety risk, and abnormality cause. Organize the normal and abnormal description information into a structured prompt text format. Configure a detection knowledge base based on the structured prompt text format of the normal and abnormal description information.

13. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 11.

15. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 11.

Images