A TVDS Anomaly Detection Method and System Based on Self-Reinforcing Iteration

Abstract

This invention discloses a TVDS anomaly detection method and system based on self-reinforcing iteration. The method includes: performing structured cleaning, semantic segmentation, and multi-granularity vectorization on unstructured railway domain expert knowledge to construct a railway domain knowledge base adapted for real-time querying of a multimodal large language model; performing visual encoding and visual-text feature alignment and projection on the TVDS image to be detected; generating retrieval statements based on the multimodal large language model; obtaining enhanced context through hybrid retrieval; constructing conservative thought chain prompts and inputting them into the model for reasoning to obtain preliminary fault detection results; and then building a low-rank adaptive self-reinforcing iterative closed loop based on massive unlabeled TVDS images to achieve model self-evolution optimization. The optimization results are fed back into the knowledge base update and fault reasoning stages to form a closed-loop detection system. This application can effectively reduce the false alarm rate, ensure accurate and efficient detection, and is suitable for railway passenger car safety inspection scenarios.

Description

Technical Field

[0001] The technical field of this invention is the intersection of artificial intelligence and intelligent operation and maintenance of rail transit, and in particular, it relates to the application of multimodal large language models (MLLMs), retrieval augmentation generation (RAG), and parameter efficient fine-tuning (PEFT) technologies in industrial visual anomaly detection. Background Technology

[0002] The Train Vehicle Fault Trackside Image Detection System (TVDS) is a core system for detecting faults in the train running gear. It generates terabytes of high-definition images daily. The efficiency of traditional manual image interpretation has reached a bottleneck. While deep learning technology has been introduced into this field, existing solutions still face numerous unresolved issues when adapted to the safety-critical industrial scenario of railways. Both of the current mainstream technical approaches (dedicated small models based on supervised learning and zero-shot detection based on general multimodal large language models) have significant shortcomings.

[0003] Supervised learning-based small model detection: Current mainstream detection schemes mostly use CNN algorithms such as Faster R-CNN and YOLO series, which extract image features through convolutional neural networks and output fault category probabilities and bounding boxes, with the training objective of minimizing classification and regression losses. However, the limitations of this approach are quite prominent:

[0004] (1) There is a serious “data hunger” problem: Model training and detection are highly dependent on large-scale, class-balanced and high-quality manually labeled datasets. However, in actual operation of TVDS, the data presents an extreme “long-tail distribution” – the number of normal samples and common fault samples such as wear is huge, while the number of key serious fault samples such as breakage and foreign object impact is extremely scarce. The data imbalance makes it difficult for the model to fully learn the features of rare faults, and it is very easy to miss detection.

[0005] (2) Poor model flexibility and high maintenance cost: When faced with new fault modes and new vehicle parts that have never been seen before, the dedicated model trained on the closed set has no detection capability. It is necessary to re-collect massive image data and complete expensive manual annotation, and then retrain the model. The model iteration cycle is long and the maintenance cost is high.

[0006] Zero-shot detection based on general multimodal large language models: General multimodal large language models, represented by Qwen-VL, GPT-4V, and CLIP, rely on the Transformer architecture to achieve visual-text feature alignment and have powerful zero-shot / few-shot understanding and open-vocabulary detection capabilities. They can identify fault types not appearing in the training set. However, due to the generality of the training data and design logic, directly applying them to TVDS detection carries significant risks. The core issues are reflected in three aspects:

[0007] (1) Lack of "common sense" in the industrial field, high risk of hallucination: Although general multimodal large language models (such as GPT-4V, Qwen-VL) have strong open-domain visual understanding capabilities, their training data mainly comes from general text and images on the Internet, lacking professional knowledge and procedural constraints in the railway industry. The model cannot understand the strict quantitative scrapping standards in the railway passenger car maintenance procedures (e.g., "the limit is 95mm when the length of the brake disc thermal crack is less than 10mm from the edge"). This cognitive bias makes the model very prone to "hallucination", such as misjudging normal oil stains, water stains, reflections, machining textures or casting marks on the surface of parts as "oil leakage" or "cracks", or issuing false alarms for parts that are within the normal wear range but have not exceeded the limit. In the safety-critical railway field, a high false alarm rate will lead to unnecessary train detaining and transportation interruption, causing huge economic losses and scheduling chaos.

[0008] (2) Lack of interpretability and compliance in the decision-making process: Railway safety production requires that fault determination must be based on evidence. General-purpose models usually output ambiguous natural language descriptions (such as "it looks like there is a problem"), which cannot provide a precise and rigorous chain of judgment logic, nor can they cite specific regulations as support. In addition, the decision-making strategy of general-purpose models is usually more aggressive (tends to identify more objects), lacking a "conservative" decision-making mechanism for industrial scenarios, making it difficult to keep the false alarm rate at a very low level (such as <1%) while ensuring recall rate, which leads to front-line maintenance experts not being able to trust and adopt the model's auxiliary suggestions.

[0009] (3) Data value is not realized, and there is a lack of a low-cost model self-evolution mechanism: TVDS systems generate terabytes of image data daily, which contains rich information on scene changes (such as the state of components under different lighting, weather, and seasons). Existing technologies, whether for dedicated small models or general large models, typically have statically fixed parameters. Faced with constantly changing operating environments (concept drift), the models lack an automated closed-loop mechanism to utilize this massive amount of unlabeled data for self-updating. If fine-tuning is entirely dependent on manual annotation, the cost will be astronomical. Therefore, existing systems cannot achieve continuous and automated improvement in model performance as data accumulates, resulting in a huge waste of data assets.

[0010] The shortcomings of the two technical approaches mentioned above can be summarized as the three core problems that TVDS systems urgently need to solve during the intelligent upgrade process, which are also common bottlenecks at the industry level:

[0011] (1) High false alarm rate and low reliability: General multimodal large language models are prone to "illusion" false alarms in non-fault areas when there are no constraints, and special small models are prone to missing rare faults. Neither can meet the stringent requirements of railway safety production for low false alarms and high recall.

[0012] (2) Lack of domain expertise: All existing models lack professional knowledge support such as railway passenger car maintenance procedures, and cannot reason based on specific size limits and fault classification standards like human experts, resulting in inaccurate fault classification and lack of basis for judgment;

[0013] (3) Scarcity of high-quality labeled data: TVDS image data is massive but manual labeling is costly, and the long-tail distribution characteristics of fault samples are significant. Existing fully supervised learning methods are difficult to achieve continuous iterative optimization of the model, which restricts the progress of intelligent upgrading.

[0014] Therefore, directly applying existing deep learning technology to TVDS anomaly detection can no longer meet industry needs, and there is an urgent need to propose new detection methods and systems adapted to key railway safety scenarios. Summary of the Invention

[0015] The purpose of this invention is to provide a TVDS anomaly detection method and system based on self-enhancing iteration, which can improve the accuracy and efficiency of anomaly detection.

[0016] The technical solution provided in this application is as follows:

[0017] In a first aspect, this application provides a TVDS anomaly detection method based on self-enhancing iteration, comprising the following steps:

[0018] S1. Unstructured railway domain expert knowledge is structured, cleaned, semantically segmented, and multi-granularity feature vectorized to transform it into a railway domain knowledge base that can be queried in real time by a multimodal large language model; the railway domain knowledge base includes multiple knowledge blocks and corresponding dense semantic vectors and sparse keyword vectors;

[0019] S2. For the TVDS image to be detected, visual encoding is performed, followed by visual-text feature alignment and projection to obtain visual semantic features in the same semantic space as the input text features of the multimodal large language model. The visual semantic features are then transformed into natural language retrieval statements in the railway maintenance field through the multimodal large language model. A retrieval vector is generated based on the retrieval statement. A hybrid retrieval is performed in the railway domain knowledge base by combining dense semantic vectors and sparse keyword vectors to obtain enhanced context. A structured conservative thought chain prompt is constructed based on the enhanced context. The conservative thought chain prompt is input into the multimodal large language model for fault reasoning, and the preliminary fault detection results of the TVDS image are output.

[0020] S3. Based on massive unlabeled TVDS images, construct a low-rank adaptive self-enhancing iterative closed loop to achieve self-evolution optimization of the multimodal large language model; feed the model optimization results back to the dynamic update of the railway domain knowledge base in step S1 and the fault reasoning of the multimodal large language model in step S2 to achieve closed loop detection.

[0021] In some possible implementations, S1 includes the following sub-steps:

[0022] S11. Perform structured cleaning and semantic segmentation on railway expert knowledge documents to obtain multiple knowledge blocks;

[0023] S12. Perform multi-granularity feature vectorization embedding on the knowledge block, including dense semantic encoding and sparse keyword encoding, to obtain the dense semantic vector and sparse keyword vector of the knowledge block.

[0024] In some possible implementations, the multi-granularity feature vectorization in S1 includes dense semantic encoding and sparse keyword encoding. The dense semantic encoding process is as follows: a CLS marker is added to the beginning of the word sequence of the knowledge block, and initial features are obtained through word embedding and position encoding. The initial features are input into the Transformer encoder for multi-layer calculation. The features corresponding to the CLS marker in the output of the last layer are extracted and normalized by L2 to obtain the dense semantic vector of the knowledge block. The sparse keyword encoding process is as follows: an inverted index is constructed, the BM25 weight of each term w in the knowledge block is calculated, and a sparse keyword vector of the knowledge block is generated.

[0025] In some possible implementations, in step S2, the retrieval of the retrieval vector is performed in the railway domain knowledge base using a hybrid retrieval strategy to obtain enhanced context. This includes: calculating the dense retrieval score of each knowledge block using the retrieval vector and the dense semantic vector of each knowledge block; calculating the sparse retrieval score of each knowledge block using the retrieval vector and the sparse keyword vector of each knowledge block; calculating the inverse ranking fusion score of each knowledge block using an inverse ranking fusion algorithm; and selecting the top-scoring knowledge blocks based on the inverse ranking fusion scores. Each knowledge block serves as enhanced context; among which This is the default value.

[0026] In some possible implementations, the structured conservative thought chain prompts in S2, which are constructed in conjunction with the enhanced context, include: constructing structured conservative thought chain prompts by combining system setting information, enhanced context, visual semantic features, and conservative instruction information; and explicitly injecting the prior distribution of low false reporting direction for railway safety risk pre-control into the conservative instruction information.

[0027] In some possible implementations, in step S2, the conservative thought chain cue words are input into a multimodal large language model for fault reasoning, and the preliminary fault detection results of the TVDS image are output, including:

[0028] The conservative thought chain cue words are input into a multimodal large language model. The multimodal large language model generates an inference chain sequence from four explicit states: observation, comparison, reflection, and decision-making, and outputs a preliminary fault category determination result. , , where 0 and 1 represent normal and fault respectively;

[0029] The uncertainty-based entropy-based gating decision filters the preliminary fault category determination results to obtain preliminary fault detection results for the TVDS image.

[0030] In some possible implementations, entropy-based uncertainty gating decision filtering filters the preliminary fault category determination results, including: calculating the Shannon entropy of the inference chain sequence as an uncertainty measure. Through a double threshold decision function Preliminary determination of fault type Perform filtering, when Output alarm results in real time. In cases where manual review is required, a release result is output; otherwise, a preliminary fault detection result is generated. The low threshold of entropy, The high threshold for entropy is used, and both are set to preset values.

[0031] In some possible implementations, S3 includes the following sub-steps:

[0032] S31. Using a multimodal large language model, images in the unlabeled TVDS image dataset are used to generate image-text descriptions for railway maintenance and construct visual question-answer pairs related to fault determination. Dropout sampling inference and consistency filtering are applied to the same image to remove hallucination samples, retaining samples that meet the consistency constraints, forming a fine-tuned dataset. ;

[0033] S32, Based on fine-tuning dataset Low-rank adaptive LoRA technology was used to fine-tune the parameters of a multimodal large language model, resulting in model weight increments. ;

[0034] S33, Based on the model weight increment The iterative evolution of the multimodal large language model is completed, and the optimized multimodal large language model is fed back to the dynamic update of the railway domain knowledge base in step S1 and the fault reasoning stage in step S2.

[0035] In some possible implementations, S32 specifically involves: freezing the original weight matrix of the multimodal large language model. Two low-rank matrices are introduced, and the output features are obtained through forward propagation. A weighted causal language modeling loss is used as the loss function. The AdamW optimizer updates only the parameters of the two low-rank matrices to obtain the weight increment matrix. .

[0036] In some possible implementations, S33 specifically involves: incrementing the weight matrix. Model weights in the current round The models are then fused together to obtain the model weights for the next round. ,Will The multimodal large language model serves as the foundation for updating the railway domain knowledge base in step S1 and fault reasoning in step S2, enabling self-evolutionary optimization of the multimodal large language model.

[0037] Secondly, this application provides a TVDS anomaly detection system based on self-enhancing iteration, comprising:

[0038] The domain knowledge base construction module is used to perform structured cleaning, semantic segmentation, and multi-granularity feature vectorization of unstructured railway domain expert knowledge, transforming it into a railway domain knowledge base that can be queried in real time by a multimodal large language model; the railway domain knowledge base includes multiple knowledge blocks and corresponding dense semantic vectors and sparse keyword vectors;

[0039] The fault reasoning module, communicatively connected to the domain knowledge base construction module, performs visual encoding on the TVDS image to be detected, followed by visual-text feature alignment and projection to obtain visual semantic features in the same semantic space as the input text features of the multimodal large language model. The multimodal large language model then transforms these visual semantic features into natural language retrieval statements for the railway maintenance domain. Based on these retrieval statements, a retrieval vector is generated. This vector is combined with dense semantic vectors and sparse keyword vectors to perform a hybrid retrieval in the railway domain knowledge base to obtain enhanced context. The enhanced context is then used to construct structured conservative thought chain prompts. These prompts are input into the multimodal large language model for fault reasoning, and the module outputs preliminary fault detection results for the TVDS image.

[0040] The self-enhancing iterative optimization module is communicatively connected to the domain knowledge base construction module and the fault reasoning module, respectively. It is used to construct a low-rank adaptive self-enhancing iterative closed loop based on massive unlabeled TVDS images to achieve self-evolution optimization of the multimodal large language model. The model optimization results are fed back to the dynamic update of the railway domain knowledge base in step S1 and the fault reasoning of the multimodal large language model in step S2 to achieve closed loop detection.

[0041] Thirdly, this application provides an electronic device, including: a memory and a processor;

[0042] The memory is used to store computer programs;

[0043] The processor is used to invoke the computer program to execute the method described above.

[0044] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed on an electronic device, causes the electronic device to perform the method described above.

[0045] Fifthly, this application provides a computer program product, including a computer program that, when run on an electronic device, causes the electronic device to perform the method described above.

[0046] The specific implementation methods of the second to fifth aspects of this application can refer to the implementation methods of the first aspect, and will not be elaborated here.

[0047] Beneficial effects:

[0048] Compared with current mainstream zero-shot / few-shot detection techniques based on general multimodal large language models (such as GPT-4V and Qwen-VL) and CNN techniques, this invention has the following significant advantages:

[0049] (1) Extremely low false alarm rate and high security (solving the applicability problem of Safety-Critical scenarios): General large models usually tend to improve the recall rate while ignoring the false alarm rate. In the railway scenario, frequent false alarms can lead to unplanned train derailments, causing huge economic losses. The "conservative thinking chain" and "entropy uncertainty gating" introduced in this invention are like installing a "rigorous braking system" on the large model. Experiments show that, under the premise of ensuring that no critical faults are missed, this method can effectively filter more than 90% of false alarms caused by oil stains, water stains, and reflections, keeping the false alarm rate within a reasonable range, which is far better than the 5%-10% of general models, and meets the stringent requirements of railway safety production.

[0050] (2) Strong interpretability and compliance of fault determination (solving the "black box" trust problem): General models or traditional CNNs only output category labels or bounding boxes, lacking a basis for judgment. This invention utilizes RAG technology to ensure that each fault alarm generated by the model can clearly cite specific clauses in the railway passenger car maintenance regulations (such as "According to Article X of the regulations, the crack length exceeds the limit..."), and can demonstrate the reasoning process of "observing sharp crack edges and no oil stain reflection characteristics" through a thought chain. This "evidence-based" inspection report not only conforms to the operational specifications for railway fault identification, but also greatly enhances the trust of front-line maintenance experts in the AI ​​system.

[0051] (3) Zero-cost adaptation to new environments and faults (solving the challenges of model aging and iteration): The railway operating environment is complex and ever-changing (seasonal and lighting changes), and new train models and components are constantly being put into use. General model parameters are fixed and cannot adapt to this conceptual drift; while fully supervised models require the re-collection and labeling of a large amount of data. The self-enhancing iteration mechanism of this invention can utilize the massive amounts of unlabeled data generated daily by the TVDS system to automatically mine feature distributions in new scenarios through a "teacher model," and quickly update model parameters through LoRA fine-tuning. This enables the system to have the ability to "learn for life." Without increasing the cost of manual labeling, the model's detection performance and generalization ability will continue to improve over time, maximizing the value of data assets.

[0052] (4) Precise fault classification and quantitative analysis capabilities (solving the problem of coarse-grained detection): Traditional target detection can only locate the fault position and cannot perform fine classification. Thanks to the understanding of image details by the large model and the injection of procedural knowledge, this invention can classify faults according to quantitative standards in the procedures (such as length, depth, and area ratio) (e.g., distinguishing between minor faults that require "tracking and use" and serious faults that require "immediate vehicle impoundment"). This provides railway operation and maintenance departments with more refined decision support and avoids situations of "over-maintenance" or "neglect of maintenance". Attached Figure Description

[0053] Figure 1 This is a core technical architecture diagram of one embodiment of this application.

[0054] Figure 2 This is a system architecture and inference workflow diagram of one embodiment of this application.

[0055] Figure 3 This is a multimodal data generation diagram in one embodiment of this application.

[0056] Figure 4 This is a diagram of a self-enhancing iterative mechanism for a multimodal large language model in one embodiment of this application.

[0057] Figure 5This is a schematic diagram illustrating the principle of multimodal large language model fine-tuning and anomaly detection in one embodiment of this application. Detailed Implementation

[0058] To enable those skilled in the art to better understand the present application, the technical solution of the present application will be further described in detail below with reference to the embodiments and accompanying drawings.

[0059] This invention aims to construct a highly reliable, procedure-aware, and self-evolving intelligent TVDS detection system to achieve:

[0060] (1) Highly reliable conservative detection: By constructing a conservative thinking chain prompt word engineering system, designing multi-step confirmation, reverse questioning and confidence gating mechanism, actively constraining the divergent behavior of multimodal large language model, while ensuring the fault recall rate, the false alarm rate is reduced to an acceptable level for railway operation.

[0061] (2) Professional reasoning that integrates maintenance procedures: Through retrieval enhancement generation (RAG) technology, unstructured railway expert knowledge (such as fault maps) is transformed into a searchable vector knowledge base, enabling the multimodal large language model to "look up" and "understand" professional standards in real time during reasoning, and to achieve accurate fault classification and grading based on the provisions of the regulations.

[0062] (3) Low-cost self-enhancing iteration of multimodal large language models: A "data flywheel driven by multimodal large language models" mechanism is proposed. This mechanism utilizes the powerful zero-shot capability of multimodal large language models to automatically generate high-quality image and text descriptions and VQA data from massive unlabeled data. The model is then fed back into itself through parameter efficient fine-tuning (PEFT) technology. A self-enhancing iterative closed loop of "AI generating data → data optimizing AI" is established to solve the problem of scarcity of high-quality labeled data and to achieve continuous adaptation and performance improvement of multimodal large language models to new environments and new faults.

[0063] The specific embodiments of this application will now be described with reference to the accompanying drawings.

[0064] Example 1:

[0065] like Figure 1 As shown, this embodiment provides a TVDS anomaly detection method based on self-reinforcing iteration, employing a closed-loop detection framework of "knowledge vectorization injection—conservative Bayesian inference—low-rank adaptive evolution". Logically, this method is divided into three tightly coupled stages.

[0066] Phase 1: Building a Domain Knowledge Base Based on Semantic Sliding Window

[0067] The core of this stage is to map unstructured railway domain expert knowledge (such as railway passenger car maintenance procedures) into a high-dimensional dense vector space, construct a railway domain knowledge base that can be queried in real time by multimodal large language models, namely External Explicit Memory, and define a hybrid retrieval execution logic based on this knowledge base to provide complete support for knowledge invocation in the anomaly detection stage. Specifically, this includes:

[0068] Step 1. Structured cleaning and semantic segmentation of the procedure document, including:

[0069] Preprocessing: Perform OCR recognition on railway expert knowledge documents (PDF / Word) and define a cleaning function. Remove headers, footers, and garbled text / noise.

[0070] Semantic Sliding Window Segmentation: To preserve the contextual integrity of the fault determination criteria (preventing the separation of "fault phenomenon" and "limit value"), an overlapping sliding window strategy is employed. Let the cleaned document be a sequence of tokens. ,in For individual words after document segmentation, This represents the number of lexical units. It also sets the size of the semantic sliding window. (e.g., 512), the step size of the semantic sliding window (e.g., 256). The generated number Knowledge Chunk Represented as:

[0071] ,in The index value of the word element is used to define... The range of lexical terms.

[0072] In some embodiments, for each knowledge block, metadata (including chapter number and component name) can be appended to form key-value pairs, which can be represented as follows: ,in for The corresponding metadata.

[0073] Step 2. Multi-granularity feature vectorization embedding:

[0074] A dual-path encoding mechanism is adopted to extract semantic features and keyword features separately.

[0075] Dense semantic encoding includes:

[0076] Text preprocessing: dividing each knowledge block Each knowledge block is segmented into a word sequence, and a CLS (Classification) tag is added to the beginning of the word sequence of each knowledge block, resulting in... The lexical sequence (including the initial CLS marker).

[0077] Initial embedding: Word embedding and positional encoding are performed on the word sequence to obtain the initial features. .

[0078] Multi-layer coding: Utilizing a finely tuned Transformer encoder (such as BGE-M3) as the encoder. .Will enter The computation is performed by a layer-by-layer Transformer encoder. The computation of each layer-by-layer Transformer encoder includes multi-head self-attention (MSA) and a feedforward network (FFN):

[0079] ;

[0080] ;

[0081] In the formula, Here, Q, K, and V are the multi-head self-attention computation functions, respectively. ;in These are the linear projection weight matrices for query, key, and value, respectively, which are obtained from pre-training of a multimodal large language model and remain fixed during the fine-tuning phase. The dimensions of the Q and K vectors in self-attention are used for normalizing the attention score; and They are respectively In Transformer Layer and first Feature output of the layer encoder; This is a multi-head self-attention computation operation; For feedforward network computation operations; This is a layer normalization operation.

[0082] Feature extraction: Extract the last layer The features corresponding to the CLS labels are obtained by L2 normalization. Dense semantic vectors The formula is:

[0083] ;

[0084] In the formula, For multimodal large language model feature dimensions, such as ; This is an L2 normalization operation.

[0085] Sparse keyword encoding (SparseEmbedding): Constructing an inverted index (i.e., a reverse mapping table of keywords → knowledge blocks containing that keyword), and calculating the value of each term. exist The BM25 (Best Matching 25) weights are used to generate sparse keyword vectors. The formula is:

[0086] ;

[0087] ;

[0088] ;

[0089] In the formula, Indicates term w in BM25 weights; Indicates the inverse document frequency of term w. Indicates term w in word frequency in These are the adjustment parameters for the BM25 algorithm; The length adjustment parameter for the BM25 algorithm; for The text length (number of lexical units). This represents the average text length of all knowledge blocks. For example, . Indicates terms exist Sparse weights in; For term indexing, , This represents the total number of terms.

[0090] Phase Two: Highly Reliable Inference Based on Conservative Thought Chain Cue Words and Multimodal Fusion

[0091] This phase involves constructing an end-to-end detection network, introducing "conservatism" constraints and "uncertainty" quantification mechanisms. For example... Figure 2 As shown, the end-to-end high-reliability inference process in this stage can be combined with... Figure 2 understand, Figure 2This diagram illustrates the architecture and inference workflow of a high-confidence anomaly detection system according to an embodiment of this application. It showcases the entire process of a TVDS image from input to final detection result distribution (automatic archiving / manual review), intuitively demonstrating the interconnectedness of technical steps in visual feature extraction, RAG knowledge retrieval, cue word construction, model inference, and confidence level distribution. Specifically, it may include:

[0092] Step 3. Visual-Text Feature Alignment and Projection:

[0093] Visual Encoder: Input TVDS image Features are extracted using a VisionTransformer (ViT, such as ViT-BigHuge). The image is then segmented. Each image patch, plus positional encoding Input visual transformer:

[0094] ;

[0095] In the formula, The visual semantic features of the image extracted by ViT have a dimension of 1. ; The number of image blocks after image I is segmented; For visual semantic features; This is the feature extraction function for the visual transformer; The basic embedding features for image patches; This is a location-encoded feature.

[0096] Modal alignment projection (Projector): To adapt to the dimensionality of input features (text features) in a large language model (LLM). Feature mapping can be performed using a single-layer MLP or C-Former to ensure that visual semantic features and input text features of a large language model are in the same semantic space.

[0097] ;

[0098] In the formula, The projected visual semantic features have a dimension of . ; The feature dimension of the input text for the large language model; and These are the weight matrix and bias terms of the first layer of a single-layer MLP, respectively. and These are the weight matrix and bias terms of the second layer of a single-layer MLP, respectively. ; This is the activation function, used to introduce nonlinear feature transformations.

[0099] In the inference phase of anomaly detection, the input TVDS image is the TVDS image to be detected.

[0100] Step 4: Hybrid retrieval of retrieval vectors and RRF-based retrieval:

[0101] Based on projected visual semantic features Generate retrieval vectors with dimensions consistent with the knowledge base in the railway field. For example: Visual semantic features are transformed into natural language retrieval statements that conform to the description of railway maintenance procedures using a multimodal large language model. The same Transformer encoder used in step 2 of the first stage is used to encode the retrieval statements and perform L2 normalization to generate retrieval vectors. .

[0102] For the generated retrieval vector The process involves sequentially performing dense search, sparse search, and result fusion, as follows:

[0103] Dense Search Score Calculation: The dense search score is calculated using cosine similarity, using the following formula: In the formula, Indicates the dense search score;

[0104] Sparse retrieval score calculation: The sparse retrieval score is calculated using the BM25 algorithm, and the formula is as follows:

[0105] ;

[0106] In the formula, express With retrieval vector The sparse retrieval score.

[0107] Reciprocal Rank Fusion (RRF): To eliminate differences in score distribution, ranking fusion is used to obtain the final ranking.

[0108] ;

[0109] In the formula, for The bottom ranking combined with the score; For smoothing constants (e.g.) ), for Ranking in dense search for Ranking in sparse retrieval.

[0110] Knowledge Block Filtering: Selecting Top Scores ( ,like The knowledge blocks are used as enhanced context (e.g., “crack length limit is 5mm”).

[0111] In some embodiments, when the retrieval vector is generated based on the TVDS image When the system includes component names such as brake pads and wheelsets, it first filters out knowledge blocks Ci that are only associated with the component name in the metadata Mi. Then, it performs dense / sparse retrieval on the filtered knowledge blocks to avoid indiscriminate retrieval of the entire knowledge base (e.g., when detecting brake pad failure, there is no need to search for regulations and clauses of unrelated components such as bogies and couplers). This narrows the retrieval scope, improves retrieval efficiency, and avoids retrieval bias caused by interference from irrelevant knowledge.

[0112] After obtaining enhanced context through hybrid retrieval, the chapter number in the metadata can be directly associated with the specific clause of the railway passenger car maintenance regulations, allowing subsequent fault reasoning chains to clearly indicate "judged according to Chapter XX, Article XX of the regulations", thus meeting the core requirements of railway safety scenarios for fault judgment to be based on regulations and traceable.

[0113] Step 5: Constructing Conservative Mind Chain (CoT) Cue Words:

[0114] Constructing a structured, conservative thought process using prompts (Prompt) Explicitly inject the prior distribution of "presumption of innocence" (i.e., low false reporting in railway safety risk prevention and control):

[0115] ;

[0116] In the formula, Tips for constructing a complete, structured, conservative thought chain; The system setting information in the prompt (such as "You are a meticulous railway quality inspector..."); This indicates enhanced context; Conservative instructions in prompts, such as "For fuzzy features, prioritize negative samples (Negative / OilStain), only if visual semantic feature points or feature regions in the image are present." satisfy "And it is only marked as a fault when it fully conforms to the procedure description," express The probability of a fault characteristic. This is the threshold for the probability of failure.

[0117] In some embodiments, constructing conservative thought chain prompts In this case, metadata can be integrated into prompt words as semantic anchors to guide the multimodal large language model to focus on the specific procedure clauses of the currently detected component, and avoid the model from confusing the fault judgment rules of different components (such as distinguishing the crack length limit of brake pads and brake discs).

[0118] In addition, the maintenance procedures for different components have different threshold requirements for conservative judgment. The component name in the metadata can trigger the corresponding component's exclusive conservative instruction, so that the conservative logic of "presumption of innocence" can be adapted to the security level of different components.

[0119] Step 6: Autoregressive Reasoning and State Transition:

[0120] Multimodal large language models generate inference chain sequences in an autoregressive manner. ,in is the sequence length. (The...) The conditional probability of the step is:

[0121] ;

[0122] In the formula, This represents the output word at step t in the reasoning chain. This represents all output words before step t in the inference chain. This indicates that, given the preceding word and the overall prompt word, the generation in step t is... The conditional probability; This is the vocabulary weight matrix. This represents the hidden layer feature output of the multimodal large language model at step t. This is the normalization function.

[0123] The chain of thought consists of four explicit states:

[0124] (Observation): Extracting visual semantic feature vectors (e.g., "black linear traces").

[0125] Calculate distance (Compared with regulations) This represents the feature distance between the observed feature and the retrieved knowledge block.

[0126] (Reflection): Perform counterfactual reasoning, "If this is oil, what characteristics should it have?"

[0127] (Decision): Output preliminary fault category determination results For example, 0 indicates normal operation, and 1 indicates a fault.

[0128] Step 7: Entropy-based uncertainty gating decision:

[0129] To filter out low-confidence predictions, the Shannon entropy of the generated sequence is calculated as a measure of uncertainty. :

[0130] ;

[0131] In the formula, Represents the inference chain sequence Shannon entropy; The sequence length used in entropy calculation (consistent with the inference chain T). This represents the first t-1 lexical units (all lexical units before the t-th position) of a multimodal large language model. Under the condition that the word at position t is output, the word at position t is output. The probability of; For a single word element in the vocabulary, It is a vocabulary set for a multimodal large language model;

[0132] Define a dual-threshold decision function :

[0133] ;

[0134] In the formula, This represents a dual-threshold decision function for the input image I, outputting an alarm, review, or release result; The low threshold of entropy, For example, the high threshold of entropy; This mechanism ensures that alarms are triggered only for high-confidence faults, ambiguous samples are handled manually, and extremely low-confidence faults are filtered out directly.

[0135] Phase 3: Self-reinforcing closed loop based on low-rank adaptive (LoRA):

[0136] This phase addresses the challenge of data annotation by utilizing a multimodal large language model as a "teacher" to generate pseudo-labels, which then feed back into the multimodal large language model itself, creating a data flywheel. The automated multimodal data generation process described in this application can be combined with... Figure 3 understand, Figure 3 This multimodal data generation diagram, used in embodiments of this application, illustrates the generation logic from unlabeled TVDS images to high-quality image-text descriptions and visual question-answering pairs (VQA), which is a core step in constructing the fine-tuning dataset. Specifically, it may include:

[0137] Step 8: Build an automated data factory (DataFactory):

[0138] For massive unlabeled TVDS image datasets Where M is the total number of unlabeled TVDS images, Let j represent the j-th unlabeled image in the dataset.

[0139] Caption Generation: Utilizes the reasoning capabilities from steps 4-6 to generate detailed natural language descriptions for unlabeled TVDS images. .

[0140] Visual Question Answering Pair Construction (VQA Pair Construction): Based on Natural Language Description Constructing question-and-answer pairs , This represents the constructed Visual Question Answering (VQA) dataset, where q is the question and a is the answer.

[0141] Consistency Filtering: To remove illusions, the same TVDS image is processed... Conduct based on the second phase Subsequent Dropout sampling inference yields The result set of subsampling inference Retain the samples that satisfy the consistency constraints:

[0142] ;

[0143] In the formula, The mode (most frequent result) represents the set of inference results R. This is an indicator function; it returns 1 when the condition within the parentheses is true, and 0 otherwise. For consistency constraint thresholds, such as Samples are retained only when the consistency rate exceeds this value;

[0144] Among them, a sample refers to a complete set of multimodal labeled data generated from a single unlabeled TVDS image after processing by a multimodal large language model. It contains text descriptions and visual question-answer pairs and serves as the basic data unit for subsequent LoRA fine-tuning of the model.

[0145] This results in a high-quality model fine-tuning dataset. .

[0146] In some embodiments, a high-quality fine-tuning dataset is constructed. In this case, the component name in the metadata can be used as a tag to classify the text and image descriptions / visual question-and-answer pairs generated from unlabeled TVDS images, allowing LoRA fine-tuning to perform refined optimization for the fault judgment logic of different components.

[0147] Step 9: Efficient Parameter Fine-Tuning (PEFT-LoRA):

[0148] To update multimodal large language models at low cost, the LoRA technique is employed. This involves freezing most of the weights of the pre-trained multimodal large language model. Only two low-rank matrices are introduced. and (rank The forward propagation update formula is:

[0149] ;

[0150] In the formula, The output features of the forward propagation of the multimodal large language model, The input features for the forward propagation of a multimodal large language model, The original weight matrix for pre-training a multimodal large language model. The weight increment matrix introduced for LoRA ; Let be the first low-rank matrix of LoRA. Let be the second low-rank matrix of LoRA. This is the scaling factor for LoRA. Initially... Initialize using a Gaussian distribution. Initialize to 0.

[0151] Loss Function: A weighted causal language modeling loss is used, with a focus on the generated fault determination part.

[0152] ;

[0153] In the formula, For LoRA fine-tuning of weighted causal language modeling loss, The parameters to be updated for the multimodal large language model are (A, B). Let be the loss weight of the word at step t, when When it belongs to the key fault description words (such as "crack" or "loose"), otherwise ; For a sample pair in the training set, The input TVDS image, For the corresponding target text sequence (such as fault description, VQA question and answer pair); For the target text sequence The total length (i.e., the total number of lexical units). Let be the word index at position t in the text sequence; Indicates that in a given , and model parameters Under these conditions, the model output The probability of; For the target text sequence The real word at position t in the middle; For the target text sequence All words preceding the t-th position in the dictionary.

[0154] Parameter update rules (AdamW Optimizer):

[0155] ;

[0156] In the formula, The learning rate for the AdamW optimizer; The first moment estimate of the AdamW optimizer at step t; The second moment estimate of the AdamW optimizer at step t; and These are the parameter values ​​of the multimodal large language model at step t and step t+1, respectively.

[0157] Update matrix only With these parameters, memory usage is reduced to about 1% of the full fine-tuning.

[0158] The LoRA-based model self-enhancing iterative overall mechanism can be combined with... Figure 4 understand, Figure 4 This diagram illustrates the self-enhancing iterative mechanism of the multimodal large language model in this application, fully presenting the positive enhancement process of "massive unlabeled TVDS images → automated multimodal data generation → AI labeled dataset → LoRA parameter fine-tuning → enhancing the multimodal large language model → replacing the base model for iteration". It intuitively demonstrates the core logic of self-evolution of the model in this application, which generates data and optimizes the model based on data.

[0159] Step 10: Iterative Evolution:

[0160] ;

[0161] In the formula, The weights of the multimodal large language model in the (t+1)th iteration are... The weights of the multimodal large language model in the t-th iteration; The weight increments for the multimodal large language model obtained by LoRA fine-tuning.

[0162] The fine-tuned weights of the multimodal large language model are merged and used as the foundation for the next round of the data factory, forming a multimodal large language model. The positive reinforcement spiral enables the multimodal large language model to continuously become stronger as data accumulates.

[0163] The collaborative principle of fine-tuning and anomaly detection in the multimodal large language model proposed in this application can be combined with... Figure 5 understand, Figure 5 This is a schematic diagram illustrating the principle of multimodal large language model fine-tuning and anomaly detection in an embodiment of this application. It shows the fine-tuning process of the model using multimodal data such as component images and text descriptions, as well as the core principle of the model's final anomaly / normal determination, demonstrating the role of multimodal data fusion in improving detection accuracy.

[0164] In some embodiments, when the iteratively optimized model feeds back into the first stage, it can update the dense semantic vector and sparse keyword vector of such knowledge blocks based on the metadata statistics of frequently retrieved but low-accuracy knowledge blocks, instead of updating the entire knowledge base, thereby reducing iteration costs.

[0165] It should be understood that the step and stage numbers in this application, such as S1 to S3, are only used to distinguish and facilitate the expression of different steps and stages, and do not necessarily constitute a restriction on the execution order between the steps and stages.

[0166] This application has the following features and advantages:

[0167] (1) To address the extreme sensitivity of TVDS systems to false alarm rates, a decision-making system based on entropy reduction and conservative instructions, called "Conservative Thinking Chain Cue Engineering," was constructed. Traditional cue engineering often aims to stimulate the model's divergent thinking to obtain more information, which often leads to misreporting of interference items such as oil stains and water stains as faults in industrial quality inspection. This invention innovatively designs system instructions that include a "presumption of innocence" prior distribution and combines it with an explicit thinking chain (CoT) of "observation-comparison-reflection-decision" to force the model to perform reverse questioning (e.g., "Does this trace have the glossy characteristics of oil stains?"). Furthermore, an uncertainty gating mechanism based on Shannon entropy is introduced to quantify the degree of hesitation in the model's reasoning process and automatically intercept or divert low-confidence results for review. This improvement fundamentally changes the "aggressive" reasoning mode of general large models, making them suitable for Safety-Critical scenarios and solving the core pain point of high false alarm rates in industrial applications.

[0168] (2) Given the lack of industry-specific expertise in general large-scale models, a multi-granularity hybrid retrieval enhancement (RAG) framework integrating railway expert knowledge was constructed. Existing technologies either rely on pure visual semantic features or general semantics, failing to understand complex quantitative and qualitative judgment criteria such as "the limit is 95mm when the length of the brake disc thermal crack is less than 10mm from the edge." This invention proposes a document segmentation strategy based on a semantic sliding window, ensuring the integrity of the context of the regulations and designing a hybrid retrieval mechanism combining dense vector retrieval (semantic matching) and BM25 sparse retrieval (keyword exact matching). By fusing the retrieval results through the RRF algorithm, the unstructured regulations text is transformed into an "external explicit memory" for model reasoning in real time, enabling the model to accurately cite regulations as the basis for judgment during reasoning, thus achieving a leap from "perception-based detection" to "rule-based cognitive reasoning."

[0169] (3) To address the issues of long-tail distribution of railway fault samples and high annotation costs, a large-scale self-enhancing iterative closed-loop mechanism based on a "data flywheel" was established. Existing fully supervised methods are limited by the quantity and quality of labeled data, resulting in long model iteration cycles and high costs. This invention proposes using a "teacher model" with high-reliability reasoning capabilities to mine massive amounts of unlabeled TVDS images, automatically generating detailed image and text descriptions (Caption) and visual question-answering pairs (VQA). By introducing a "consistency check" filtering mechanism, noisy data generated by model illusions is eliminated, constructing a high-quality synthetic training set. Subsequently, using the parameter efficient fine-tuning (LoRA) technique, the domain knowledge contained in these synthetic data is internalized into the model parameters, forming a positive enhancement spiral of "model generating data → data optimizing model". This mechanism can achieve model adaptation to new vehicle types and new faults without continuous manual intervention, significantly reducing data annotation and model maintenance costs.

[0170] Example 2:

[0171] This embodiment provides a TVDS anomaly detection system based on self-reinforcing iteration, including:

[0172] The domain knowledge base construction module is used to perform structured cleaning, semantic segmentation, and multi-granularity feature vectorization of unstructured railway domain expert knowledge, transforming it into a railway domain knowledge base that can be queried in real time by a multimodal large language model; the railway domain knowledge base includes multiple knowledge blocks and corresponding dense semantic vectors and sparse keyword vectors;

[0173] The fault reasoning module, communicatively connected to the domain knowledge base construction module, performs visual encoding on the TVDS image to be detected, followed by visual-text feature alignment and projection to obtain visual semantic features in the same semantic space as the input text features of the multimodal large language model. The multimodal large language model then transforms these visual semantic features into natural language retrieval statements for the railway maintenance domain. Based on these retrieval statements, a retrieval vector is generated. This vector is combined with dense semantic vectors and sparse keyword vectors to perform a hybrid retrieval in the railway domain knowledge base to obtain enhanced context. The enhanced context is then used to construct structured conservative thought chain prompts. These prompts are input into the multimodal large language model for fault reasoning, and the module outputs preliminary fault detection results for the TVDS image.

[0174] The self-enhancing iterative optimization module is communicatively connected to the domain knowledge base construction module and the fault reasoning module, respectively. It is used to construct a low-rank adaptive self-enhancing iterative closed loop based on massive unlabeled TVDS images to achieve self-evolution optimization of the multimodal large language model. The model optimization results are fed back to the dynamic update of the railway domain knowledge base in the first stage and the fault reasoning of the multimodal large language model in the second stage to achieve closed loop detection.

[0175] Example 3:

[0176] This embodiment provides an electronic device, including: a memory and a processor;

[0177] The memory is used to store computer programs;

[0178] The processor is configured to invoke the computer program to execute the method as described in Embodiment 1.

[0179] Example 4:

[0180] This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is run on an electronic device, it causes the electronic device to perform the method described in Embodiment 1.

[0181] Example 5:

[0182] This embodiment provides a computer program product, including a computer program that, when run on an electronic device, causes the electronic device to perform the method described in Embodiment 1.

[0183] The specific implementation of the system, electronic device, computer-readable storage medium, and computer program product provided in this application can be referred to the specific embodiments of the above methods, and will not be repeated here.

[0184] Obviously, those skilled in the art should understand that the various units or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps into a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0185] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A self-enhancing iteration based TVDS anomaly detection method, characterized in that, Includes the following steps: S1. The unstructured railway domain expert knowledge is structured, cleaned, semantically segmented, and multi-granular feature vectorized, and transformed into a railway domain knowledge base that can be queried in real time by a multimodal large language model. The railway domain knowledge base includes multiple knowledge blocks and corresponding dense semantic vectors and sparse keyword vectors; S2. For the TVDS image to be detected, visual encoding is performed, followed by visual-text feature alignment and projection to obtain visual semantic features in the same semantic space as the input text features of the multimodal large language model. The visual semantic features are then transformed into natural language retrieval statements in the railway maintenance field through the multimodal large language model. Based on the retrieval statements, retrieval vectors are generated. Combined with dense semantic vectors and sparse keyword vectors, a hybrid retrieval is performed in the railway domain knowledge base to obtain enhanced context. A structured conservative thought chain prompt is constructed by combining system setting information, enhanced context, visual semantic features, and conservative instruction information. The conservative instruction information explicitly injects a priori distribution of low-false-reporting direction for railway safety risk prevention and control. The conservative thought chain prompt is input into a multimodal large language model, which generates an inference chain sequence based on four explicit states: observation, comparison, reflection, and decision-making, and outputs a preliminary fault category determination result. , , where 0 and 1 represent normal and fault respectively; the preliminary fault category judgment result is filtered by the uncertainty gating decision based on entropy to obtain the preliminary fault detection result of TVDS image; S3. Based on massive unlabeled TVDS images, construct a low-rank adaptive self-enhancing iterative closed loop to achieve self-evolution optimization of the multimodal large language model; feed the model optimization results back to the dynamic update of the railway domain knowledge base in step S1 and the fault reasoning of the multimodal large language model in step S2 to achieve closed loop detection.

2. The method according to claim 1, characterized in that, The multi-granularity feature vectorization in S1 includes dense semantic encoding and sparse keyword encoding. The dense semantic encoding process is as follows: a CLS marker is added to the beginning of the word sequence of the knowledge block, and the initial features are obtained through word embedding and position encoding. The initial features are input into the Transformer encoder for multi-layer calculation. The features corresponding to the CLS marker in the output of the last layer are extracted and normalized by L2 to obtain the dense semantic vector of the knowledge block. The sparse keyword encoding process is as follows: an inverted index is constructed, the BM25 weight of each term w in the knowledge block is calculated, and the sparse keyword vector of the knowledge block is generated.

3. The method according to claim 1, characterized in that, In step S2, a hybrid retrieval strategy is used to perform a retrieval in the railway domain knowledge base using the retrieval vector to obtain enhanced context. This includes: calculating the dense retrieval score of each knowledge block using the retrieval vector and the dense semantic vector of each knowledge block; calculating the sparse retrieval score of each knowledge block using the retrieval vector and the sparse keyword vector of each knowledge block; calculating the inverse ranking fusion score of each knowledge block using an inverse ranking fusion algorithm; and selecting the top-scoring knowledge blocks based on their inverse ranking fusion scores. Each knowledge block serves as enhanced context; among which This is the default value.

4. The method according to claim 1, characterized in that, Entropy-based uncertainty gating decision filtering filters the preliminary fault category determination results, including: calculating the Shannon entropy of the inference chain sequence as an uncertainty measure. Through a double threshold decision function Preliminary determination of fault type Perform filtering, when Output alarm results in real time. In cases where manual review is required, a release result is output; otherwise, a preliminary fault detection result is generated. The low threshold of entropy, The high threshold for entropy is used, and both are set to preset values.

5. The method according to claim 1, characterized in that, S3 includes the following sub-steps: S31. Using a multimodal large language model, images in the unlabeled TVDS image dataset are used to generate image-text descriptions for railway maintenance and construct visual question-answer pairs related to fault determination. Dropout sampling inference and consistency filtering are applied to the same image to remove hallucination samples, retaining samples that meet the consistency constraints, forming a fine-tuned dataset. ; S32, Based on fine-tuning dataset Low-rank adaptive LoRA technology was used to fine-tune the parameters of a multimodal large language model, resulting in model weight increments. ; S33, Based on the model weight increment The iterative evolution of the multimodal large language model is completed, and the optimized multimodal large language model is fed back to the dynamic update of the railway domain knowledge base in step S1 and the fault reasoning stage in step S2.

6. The method according to claim 5, characterized in that, Specifically, S32 involves freezing the original weight matrix of the multimodal large language model. Two low-rank matrices are introduced, and the output features are obtained through forward propagation. A weighted causal language modeling loss is used as the loss function. The AdamW optimizer updates only the parameters of the two low-rank matrices to obtain the weight increment matrix. .

7. The method according to claim 5, characterized in that, Specifically, S33 involves: adjusting the weight increment matrix. Model weights in the current round The models are then fused together to obtain the model weights for the next round. ,Will The multimodal large language model serves as the foundation for updating the railway domain knowledge base in step S1 and fault reasoning in step S2, enabling self-evolutionary optimization of the multimodal large language model.

8. A TVDS anomaly detection system based on self-reinforcing iteration, characterized in that, include: The domain knowledge base construction module is used to perform structured cleaning, semantic segmentation, and multi-granularity feature vectorization of unstructured railway domain expert knowledge, transforming it into a railway domain knowledge base that can be queried in real time by a multimodal large language model. The railway domain knowledge base includes multiple knowledge blocks and corresponding dense semantic vectors and sparse keyword vectors; The fault reasoning module, which is communicatively connected to the domain knowledge base construction module, is used to perform visual encoding on the TVDS image to be detected, and then perform visual-text feature alignment and projection to obtain visual semantic features in the same semantic space as the input text features of the multimodal large language model. The multimodal large language model is used to transform the visual semantic features into natural language retrieval statements in the field of railway maintenance. Based on the retrieval statements, a retrieval vector is generated. The dense semantic vector and sparse keyword vector are combined to perform a hybrid retrieval in the railway domain knowledge base to obtain enhanced context. A structured conservative thought chain prompt is constructed by combining system setting information, enhanced context, visual semantic features, and conservative instruction information. The conservative instruction information explicitly injects a priori distribution of low-false-reporting direction for railway safety risk prevention and control. The conservative thought chain prompt is input into a multimodal large language model, which generates an inference chain sequence based on four explicit states: observation, comparison, reflection, and decision-making, and outputs a preliminary fault category determination result. , , where 0 and 1 represent normal and fault respectively; The preliminary fault category determination results are filtered based on entropy-based uncertainty gating decision to obtain preliminary fault detection results for TVDS images. The self-enhancing iterative optimization module is communicatively connected to the domain knowledge base construction module and the fault reasoning module, respectively. It is used to construct a low-rank adaptive self-enhancing iterative closed loop based on massive unlabeled TVDS images to achieve self-evolution optimization of the multimodal large language model. The model optimization results are fed back to the domain knowledge base construction module for dynamic updating of the railway domain knowledge base and to the fault reasoning module for fault reasoning of the multimodal large language model to achieve closed loop detection.

Images