Construction method of railway multimodal knowledge base question answering system based on hybrid RAG architecture

Abstract

The present invention relates to the field of intelligent railway technology and provides a method for constructing a railway multimodal knowledge base question-answering system with a hybrid RAG architecture. The method comprises: 1. server local model deployment and fine-tuning; 2. data processing and database construction; establishing a railway specification knowledge base database management system based on MongoDB content storage and PostgreSQL vector search library; optimizing and merging search results for multiple sets of data based on hybrid search and RRF algorithms, and finally performing re-ranking and search filtering; 3. constructing a multimodal knowledge base; using multimodal embedding to embed images and text into a vector database, while simultaneously storing the corresponding original images and text in a document store; obtaining images from the document store during hybrid similarity search, and passing the original images and text blocks to a large model to generate answers; and 4. multi-platform expansion. The present invention can preferably construct a railway multimodal knowledge base question-answering system.

Description

Technical Field

[0001] The present invention relates to the field of intelligent railway technology, and in particular to a method for constructing a railway multimodal knowledge base question-answering system with a hybrid RAG architecture. Background Art

[0002] In recent years, technologies related to large language models (LLMs) have developed rapidly, showing great application potential in multiple transportation infrastructure fields such as traffic safety and autonomous driving. For example, the MT-GPT framework can be applied to the planning and design, operation and maintenance, and multi-faceted decision-making of transportation infrastructure. Some scholars have also built question-answering systems based on LLMs for traffic safety and BIM forward design, constructed local knowledge bases, and designed evaluation indicators to evaluate system performance. Some scholars have attempted to use LLMs for predictive health management of railway systems, analyzing and revealing potential fault hazards and providing scientific evidence. It is expected that LLMs will be used to generate fault diagnosis reports, maintenance planning, and scheduling documents for railway systems. However, in the field of intelligent railways, there are still many gaps in LLM research.

[0003] Retrieval-augmented generation (RAG) has been extensively applied in various intelligent question-answering scenarios for LLM, and some research has focused on improving its retrieval quality and enhancing generation robustness and confidence.

[0004] However, the existing RAG technology still has the following disadvantages:

[0005] 1. Monomodality: Existing RAG systems mostly process only unstructured text data or have very limited and rigid support for multimodality (especially image + text + structured tables). They struggle to effectively integrate and process information like standardized charts and tables, as well as structured data hidden within tables. This prevents them from fully utilizing industry-specific, multi-source, heterogeneous data.

[0006] 2. Weak structured data processing: Traditional RAGs typically treat tabular data simply as "text blocks" for vectorization, which easily destroys the table's row-column relationships, hierarchical structure, and inherent semantic relationships. This makes it difficult to accurately locate specific table cells during searches and understand cross-table data relationships, hindering the ability to extract accurate answers from complex tables such as those in industry regulations.

[0007] 3. Inadequate retrieval quality optimization: Basic RAGs typically rely solely on simple vector similarity searches, resulting in results that may contain irrelevant or redundant information. There is a lack of effective retrieval quality optimization mechanisms to optimize retrieval results.

[0008] 4. Low adaptability to specialized fields: Existing RAG technologies are mostly general-purpose and lack systematic verification in complex, demanding, and realistic vertical multi-scenarios such as railway engineering. Large models in general-purpose fields perform poorly in the professional terminology and standardized expressions of specialized vertical industries such as railways.

[0009] Therefore, a railway multimodal knowledge base question answering system construction method with a hybrid RAG architecture is needed to solve the above problems. Summary of the Invention

[0010] The present invention provides a method for constructing a railway multimodal knowledge base question-answering system with a hybrid RAG architecture, which can better construct a railway multimodal knowledge base question-answering system.

[0011] The method for constructing a railway multimodal knowledge base question-answering system based on a hybrid RAG architecture according to the present invention comprises the following steps:

[0012] Step 1: Server local model deployment and fine-tuning;

[0013] The deployed models include language models and text embedding models;

[0014] Fine-tuning is based on the DyLoRA fine-tuning framework, integrating dynamic rank allocation, hierarchical adaptation, and expert routing mechanisms to improve model performance while maintaining parameter efficiency;

[0015] Step 2: Data processing and database construction;

[0016] The data includes railway feasibility studies, track operation and maintenance reports, various industry design standards in docx and pdf formats, as well as design drawings, specification tables, and images in various formats;

[0017] After deploying the language model and text embedding model locally, we used open source projects to manage the model interfaces and established a railway specification knowledge base database management system based on MongoDB content storage and PostgreSQL vector retrieval libraries. PostgreSQL used vector fields to store vectors, while MongoDB was used to access the original vector data. During retrieval, the vectors were first retrieved, and then the original data content was searched in MongoDB based on the vector ID. For multiple sets of retrieved data, the search results were optimized and merged using a hybrid search and RRF algorithm, followed by re-ranking and search filtering.

[0018] Step 3: Multimodal knowledge base construction;

[0019] Using multimodal embedding, images and corresponding text are embedded in a vector database. The original images and text are stored in the document store. When performing hybrid similarity retrieval, the corresponding images in the document store are directly retrieved, and the original images and text blocks are passed to the large model to generate answers. For macro images, a text summary is generated for the image based on the multimodal large model, and the text summary is embedded and retrieved using a text embedding model.

[0020] Step 4: Multi-platform expansion.

[0021] Preferably, in step 1, the language model is an autoregressive language model of the Transformer architecture, which is pre-trained based on massive text data to learn the statistical laws, world knowledge, and complex patterns of language; and the core task of the text embedding model is to convert a piece of text into a fixed-length, dense vector for embedding.

[0022] Preferably, in step 1, the formula of the DyLoRA fine-tuning framework is as follows:

[0023] ;

[0024] in, The output of model fine-tuning; These are the pre-trained parameters that remain unchanged during fine-tuning; Represents the learnable parameters of each group of experts in the model expert routing system; and are two low-rank matrices trained when fine-tuning the model; represents rank, is the minimum rank, is the maximum value; Indicates the Sensitivity score of the layer network; is the sparse coefficient; represents transpose, Represents a natural constant.

[0025] As a preferred method, fine-tuning uses the instruction supervision fine-tuning dataset, which includes 10,000 railway industry corpora of industry specifications, feasibility studies, and operation and maintenance reports. The instruction supervision fine-tuning dataset is shown in the following formula:

[0026] ;

[0027] in, Indicated by A dataset consisting of independent samples; Represents each triplet sample; Indicates the Instructions for samples to guide model behavior; Indicates the The input of samples provides the specific context or problem of the task; Indicates the The output of each sample is the expected response or answer;

[0028] Core training parameters include learning rate, batch size, number of training rounds, maximum number of samples, and gradient clipping, which can be flexibly adjusted according to model parameters and dataset size.

[0029] As a preference, hybrid search includes semantic vector search and full-text sparse search. Semantic vector search calculates the distance between spatial vectors to obtain similarity, and full-text sparse search adopts a keyword search method based on a sparse algorithm.

[0030] After obtaining the results of both searches, the relevant content is input as prompt words into the large model through normalization, re-ranking and search filtering as a reference for the output; re-ranking is a method of combining multiple result sets with different relevance indicators into a single result set; search filtering uses the citation upper limit and minimum relevance to improve the quality of the search;

[0031] Among them, semantic vector search calculates index items With candidate documents The semantic vector similarity score between , the formula is as follows:

[0032] ;

[0033] Where, is the query vector, For each candidate document vector; full-text sparse retrieval calculates each candidate document Sparse retrieval score , whose formula is as follows:

[0034] ;

[0035] Where, For index items a term of ; For terms In the documentation word frequency in ; For Documents length; is the average document length of the document collection; Control word frequency saturation, the larger the The greater the impact; Controls the strength of document length normalization; For terms The inverse document frequency of is the total number of documents in the document collection; is the number of documents containing the term;

[0036] After obtaining all document sparsity scores and vector similarity scores, the formula for score normalization is as follows:

[0037] ;

[0038] Where, is the index item; For all candidate documents; and Represent the mean of all document sparse scores and vector similarity scores in the current candidate pool respectively; and and Represent the standard deviation of the sparsity scores and vector similarity scores of all documents in the current candidate pool.

[0039] Preferably, in step 2, the text is segmented and sliced ​​according to the rules and then converted into a slice format that can be semantically searched, and each slice data in the database is individually fine-tuned and corrected; the method for optimizing the knowledge database structure includes merging and organizing similar related content, processing situations where default questions cannot be matched, and optimizing table name and table body matching errors.

[0040] As a preference, in step 2, the table data is optimized by simplifying the table and converting the table into text; for large tables, a large table model is used to first encode the table and then embed it;

[0041] Table encoding first performs structural analysis, including identifying the row and column structure of the table, parsing merged cells, and extracting hierarchical relationships. Then, semantic encoding is performed, including linearization of the representation function and semantic embedding representation process. The formula is as follows:

[0042] ;

[0043] in, express Matrix, starting row index , starting column index ; and Respectively represent the total number of rows and columns in the logical area covered by table cells; and The number of cells merged horizontally and vertically respectively; represents the generating function of the linearized sequence; It is a cell text collection; is the string concatenation operator; is the column separator; is a unit separator; Represents the tensor product operation; Represents text generated by a pre-trained embedding model Embedding vector; Represents the word segmentation function for text segmentation; Indicates that the parameter is Embedding model encoder.

[0044] As a preference, in step 3, for input data image-text pairs, where the original image , The formulas generated for associating text, preprocessing and embedding are as follows:

[0045] ;

[0046] in, Indicates whether the image type is a macro image; Indicates the use of a pre-trained image classification model; represents the image that is judged as a macro image by the classification model; is the Softmax function; is the macro image threshold, if , then trigger summary generation ; Represents a multimodal large model; The maximum number of digest tokens; Represents the image embedding vector and aligned text embedding vectors The joint embedding vector of and Represent the embedding modality weights of image and text respectively; is the original text embedding vector that is not aligned with the image vector in the same dimension. is the projection matrix used to align the dimensions, and its parameters are obtained through linear regression training; and They are image embedding model and text embedding model respectively; and They are the graphic embedding dimension and the text embedding dimension respectively;

[0047] For CAD vector graphics format, the B-spline curve analytical method is adopted to construct the characteristic vector formula as follows:

[0048] ;

[0049] in, It means a Secondary B-spline curve, For the definition on Sub-B-spline basis functions ; The characteristic vector obtained by linear transformation is the curve parameters spliced ​​into a vector in a fixed order; represents the node vector; is the control point; is the weight factor corresponding to the B-spline.

[0050] In step 4, the local framework is expanded to web pages, WeChat official accounts, mini-programs, and third-party software. The core components include a protocol-adaptive API gateway and asynchronous messaging middleware.

[0051] A unified API gateway architecture is used to achieve multi-terminal access. The gateway has a built-in protocol conversion layer that supports unified conversion of HTTP / WebSocket heterogeneous protocols to gRPC protocol. The protocol conversion process is performed by inputting a normalization function. As shown below:

[0052] ;

[0053] in, Represents input data; when When coming from HTTP protocol, that is , perform serialization conversion from JSON to ProtoBuf; when the input data comes from the WeChat ecosystem, that is, , perform XML to ProtoBuf conversion; For heterogeneous input data sets, the protocol conversion delay is controlled within 15ms. ;

[0054] For high-concurrency scenarios, message queues are introduced for asynchronous peak shaving. The message processing delay model is as follows:

[0055] ;

[0056] in, Indicates the total message processing time; Indicates the serialization time of converting the original message data into a standardized format for transmission or storage; Indicates the number of messages; is the number of partitions; This is the single message consumption latency.

[0057] The beneficial effects of the present invention are as follows:

[0058] This invention applies hybrid RAG technology to the railway industry. Through processes such as server construction and local model deployment, data cleaning and processing, multimodal database construction, and multi-deployment platform testing, it constructs an intelligent question-and-answer system that integrates multimodal data such as railway design specifications, engineering cases, research reports, and operation and maintenance reports. It improves the efficiency and accuracy of knowledge acquisition for designers, and at the same time deeply integrates and utilizes multimodal information to enable complex decision-making and design optimization, ensure compliance and reduce design errors, and ultimately promote the railway industry's transformation towards intelligence and digitalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is a flowchart of a method for constructing a railway multimodal knowledge base question-answering system with a hybrid RAG architecture in an embodiment;

[0060] Figure 2 This is a schematic diagram of the railway industry data processing system and database construction in the embodiment;

[0061] Figure 3 Schematic diagram of database retrieval principle in the embodiment;

[0062] Figure 4 Schematic diagram of hybrid search in the embodiment;

[0063] Figure 5 Schematic diagram of constructing a multimodal knowledge base in an embodiment. DETAILED DESCRIPTION

[0064] In order to further understand the content of the present invention, the present invention is described in detail with reference to the accompanying drawings and embodiments. It should be understood that the embodiments are merely for explaining the present invention and are not intended to limit the present invention.

[0065] Example 1:

[0066] like Figure 1 As shown, this embodiment provides a method for constructing a railway multimodal knowledge base question-answering system with a hybrid RAG architecture, which includes the following steps:

[0067] Step 1: Server local model deployment and fine-tuning;

[0068] The deployed models include a language model and a text embedding model. The relevant programs in this embodiment are written in Python and accelerated using CUDA (Compute Unified Device Architecture). Data generation and processing, model deployment, and inference are performed on a Linux server. The language model is a typical autoregressive language model based on the Transformer architecture. It is pre-trained on massive amounts of text data to learn the statistical laws of language, world knowledge, and complex patterns. The core task of the text embedding model is to convert a piece of text into a fixed-length, dense vector for embedding (semantically similar text will have similar distances in vector space, such as cosine similarity, between their corresponding vectors). This model is essential for building industry knowledge bases.

[0069] Model fine-tuning refers to the instruction fine-tuning training of a large language model to adapt to a specific task. It can be mainly divided into two methods: parameter-efficient fine-tuning and full parameter fine-tuning. Among them, parameter-efficient fine-tuning (PEFT) only updates some parameters in the model, minimizes the number of fine-tuning parameters and computational complexity, significantly reduces training time and cost, and achieves efficient transfer learning. Among them, based on the traditional LoRA fine-tuning method, the present invention proposes a DyLoRA (Dynamic Low-Rank Adaptation) fine-tuning framework, which integrates dynamic rank allocation, hierarchical adaptation and expert routing mechanism to improve model performance while maintaining parameter efficiency. The formula is as follows:

[0070] ;

[0071] in, The output of model fine-tuning; These are the pre-trained parameters that remain unchanged during fine-tuning; Represents the learnable parameters of each group of experts in the model expert routing system; and These are two low-rank matrices trained when fine-tuning the model. Generally, one is initialized to zero during the initialization phase, while the other is initialized using a random Gaussian distribution to ensure that the mapping of the original model is not affected in the early stages of fine-tuning. represents rank, The minimum rank is 8. Take 64 as the maximum value; Indicates the Sensitivity score of the layer network; is the sparse coefficient, which is 0.3; represents transpose, Represents a natural constant.

[0072] Fine-tuning uses the instruction supervision fine-tuning dataset, which includes 10,000 railway industry corpora, including industry specifications, feasibility studies, and operation and maintenance reports. The instruction supervision fine-tuning dataset is shown in the following formula:

[0073] ;

[0074] in, Indicated by A dataset consisting of independent samples; Represents each triplet sample; Indicates the Instructions for samples to guide model behavior; Indicates the The input of samples provides the specific context or problem of the task; Indicates the The output of each sample is the expected response or answer;

[0075] Core training parameters include learning rate, batch size, number of training rounds, maximum number of samples, and gradient clipping, which are flexibly adjusted based on model parameters and dataset size. For example, using a 70B parameter model as an example, dynamic rank allocation reduced the total number of training parameters from 0.161B to 0.102B, a reduction of approximately 37%, effectively reducing training energy consumption.

[0076] Step 2: Data processing and database construction;

[0077] The railway industry has a wide variety of data types and quantities, including railway feasibility reports in docx and pdf formats, track operation and maintenance reports, various industry design standards, as well as design drawings, standard tables and pictures in various formats. In order to build a standardized industry multimodal knowledge base, different formats of data need to be processed and optimized differently, such as Figure 2 As shown, specifically:

[0078] 2.1) Data collection;

[0079] The data includes pre-feasibility and feasibility study reports, operation and maintenance reports, industry design specifications, design drawings, specification tables and pictures, etc.

[0080] 2.2) Data processing;

[0081] Text is primarily semantically identified and split using a hybrid retrieval method combining semantic vector retrieval and full-text sparse retrieval, with similarity metrics quantified. Tables are converted to text or recognized by optical character recognition (OCR), or encoded using a large table model before embedding. Images are first subjected to text extraction and vectorized image embedding. Complex images or drawings are embedded into a vector database along with the corresponding text using multimodal embedding. Macro images are then embedded and retrieved after generating a text summary based on a large multimodal model. Vector graphics such as CAD are converted to vector or JSON formats using methods such as B-spline curve analysis, and specialized deep learning models can also be trained.

[0082] 2.3) Data fusion and storage;

[0083] The text, tables, images, vector diagrams, matrices and formulas are integrated and stored; the integrated data is input into the MongoDB content storage and PostgreSQL vector retrieval library; a multi-level railway industry data processing system is constructed, and a database is built.

[0084] After deploying the language model and text embedding model locally, we use open source projects to manage the model interface and establish a railway specification knowledge base database management system based on MongoDB content storage and PostgreSQL vector retrieval library. The database retrieval principle is as follows: Figure 3 As shown in the figure, PostgreSQL sets the vector field to store vectors, while MongoDB is used to access vector original data. For example, when retrieving vector data 1, index vector 1 is first recalled, and then original data content 1 is found in MongoDB based on the ID of index vector 1. For multiple sets of retrieved data, such as original data content 2 and 3, the vector data 2-4 corresponding to indexes 2-4 are searched, filtered, normalized, and reordered based on hybrid retrieval and the RRF algorithm, and finally optimized and merged.

[0085] This embodiment adopts hybrid search enhancement generation (hybird RAG) to use semantic vector search and full-text sparse search to make up for their respective shortcomings, making the search results richer and more accurate while reducing the possibility of hallucinations in large models. The principle is as follows Figure 4 shown.

[0086] Semantic vector search calculates similarity by calculating the distance between spatial vectors. Its advantages include close semantic understanding, cross-lingual understanding (for example, matching Chinese questions with English knowledge points), multimodal understanding and mapping, and tolerance for errors (such as spelling errors and ambiguous descriptions). However, it also has drawbacks such as reliance on model training and unstable accuracy. Full-text sparse search, on the other hand, uses a keyword search method based on a sparse algorithm, suitable for precise matching of small, low-frequency characters. Search filtering is required during search, using a citation limit (i.e., a maximum of n tokens of content referenced per search) and a minimum relevance (i.e., filtering out low-relevance search results) to improve search quality.

[0087] like Figure 4 In the example, semantic vector retrieval and search filtering yield slices 1, 2, ..., while full-text sparse retrieval and search filtering yield slices a, b, .... After obtaining the results of both searches, the document and vector similarity metrics are normalized and then re-ranked. This allows the relevant content to be input as prompts into the larger model, serving as a reference for the output. Re-ranking combines multiple result sets with different relevance metrics into a single result set. For example, slice b, obtained through sparse search, has a normalized similarity of 0.9, which is higher than slices 1 and 2 obtained through semantic search, and thus serves as the primary reference for the output.

[0088] Among them, semantic vector search calculates index items With candidate documents The semantic vector similarity score between , the formula is as follows:

[0089] ;

[0090] Where, is the query vector, For each candidate document vector; full-text sparse retrieval calculates each candidate document Sparse retrieval score , whose formula is as follows:

[0091] ;

[0092] Where, For index items a term of; For terms In the documentation word frequency in ; For Documents length; is the average document length of the document collection; Control word frequency saturation, usually in the range of 1.2-2.0, the larger the The greater the impact; Controls the strength of document length normalization, typically in the range of 0.5-0.8, with 1 indicating full normalization and 0 indicating no normalization at all; For terms The inverse document frequency of is the total number of documents in the document collection; is the number of documents containing the term.

[0093] After obtaining all document sparsity scores and vector similarity scores, the formula for score normalization is as follows:

[0094] ;

[0095] Where, is the index item; For all candidate documents; and Represent the mean of all document sparse scores and vector similarity scores in the current candidate pool respectively; and and Represent the standard deviation of the sparsity scores and vector similarity scores of all documents in the current candidate pool.

[0096] Text generally needs to be segmented and sliced ​​according to specific rules before being converted into a semantically searchable slice format. Each slice of the database can also be fine-tuned and corrected individually. Furthermore, methods for optimizing the structure of knowledge databases include merging and organizing similar and related content, addressing situations where default questions cannot be matched, and optimizing table name and table body matching errors. These optimizations can be tailored to specific situations during the testing phase.

[0097] Because tabular data appears frequently and is more difficult to parse than text, it often requires optimization through methods like table simplification (e.g., reducing cell splits and adding symbolic explanations for easier indexing) and table-to-text conversion (summarizing simple tables with text). For large tables, specialized table models can be used to encode the table before embedding it.

[0098] Table encoding first performs structural analysis, including identifying the row and column structure of the table, parsing merged cells, and extracting hierarchical relationships. Then, semantic encoding is performed, including linearization of the representation function and semantic embedding representation process. The formula is as follows:

[0099] ;

[0100] in, express Matrix, starting row index , starting column index ; and Respectively represent the total number of rows and columns in the logical area covered by table cells; and The number of cells merged horizontally and vertically respectively; represents the generating function of the linearized sequence; It is a cell text collection; is the string concatenation operator; is the column separator; is a unit separator; Represents the tensor product operation; Represents text generated by a pre-trained embedding model Embedding vector; Represents the word segmentation function for text segmentation; Indicates that the parameter is Embedding model encoder.

[0101] Step 3: Multimodal knowledge base construction;

[0102] Since the railway industry includes design drawings and pictures in various formats, and large specification tables are not suitable for output as tables when indexing, and are more suitable for output in the form of pictures, it is necessary to build a multimodal knowledge base that includes picture formats.

[0103] The principle of embedding multimodal data is as follows Figure 5 As shown, for complex pictures or drawings, a multimodal embedding method is used to embed the picture, the corresponding picture title and the introduction text into the vector database. At the same time, the corresponding original picture and text are stored in the same slice of the database document. When performing mixed similarity retrieval, the relevant pictures of the content in the database can be obtained after the relevant text is indexed, and the relevant pictures and related text blocks are passed to the large model to generate answers; for macro pictures, a text summary is generated for the picture based on the multimodal large model, and the text summary is embedded and retrieved using a text embedding model, and the relevant pictures and related text corresponding to the retrieved text summary are passed to the large model to generate answers.

[0104] For input data image-text pairs, where the original image , The formulas generated for associating text, preprocessing and embedding are as follows:

[0105] ;

[0106] in, Indicates whether the image type is a macro image; Indicates the use of a pre-trained image classification model; represents the image that is judged as a macro image by the classification model; is the Softmax function; is the macro image threshold, take 0.8, if , then trigger summary generation ; Represents a large multimodal model, such as Qwen2.5-VL; The maximum number of digest tokens is 1024. Represents the image embedding vector and aligned text embedding vectors The joint embedding vector of and Represent the embedding modality weights of image and text respectively; is the original text embedding vector that is not aligned with the image vector in the same dimension. is the projection matrix used to align the dimensions, and its parameters are obtained through linear regression training; and They are image embedding model and text embedding model respectively; and They are the graphic embedding dimension and the text embedding dimension respectively;

[0107] For CAD vector graphics format, the B-spline curve analytical method is adopted to construct the characteristic vector formula as follows:

[0108] ;

[0109] in, It means a Secondary B-spline curve, For the definition on Sub-B-spline basis functions ; The characteristic vector obtained by linear transformation is the curve parameters spliced ​​into a vector in a fixed order; represents the node vector; is the control point; is the weight factor corresponding to the B-spline.

[0110] Step 4: Multi-platform expansion.

[0111] In step 4, the local framework is expanded to web pages, WeChat official accounts, mini-programs, and third-party software. The core components include a protocol-adaptive API gateway and asynchronous messaging middleware.

[0112] A unified API gateway architecture is used to achieve multi-terminal access. The gateway has a built-in protocol conversion layer that supports unified conversion of HTTP / WebSocket heterogeneous protocols to gRPC protocol. The protocol conversion process is performed by inputting a normalization function. As shown below:

[0113] ;

[0114] in, Represents input data; when When coming from HTTP protocol, that is , perform serialization conversion from JSON to ProtoBuf; when the input data comes from the WeChat ecosystem, that is, , perform XML to ProtoBuf conversion; For heterogeneous input data sets, the protocol conversion delay is controlled within 15ms. ;

[0115] For high-concurrency scenarios, message queues are introduced for asynchronous peak shaving, which enables stable message throughput to be maintained even under peak loads. The message processing latency model is as follows:

[0116] ;

[0117] in, Indicates the total message processing time; Indicates the serialization time of converting the original message data into a standardized format for transmission or storage; Indicates the number of messages; The default configuration is 8 partitions. This is the single message consumption latency.

[0118] The advantages of this embodiment are:

[0119] (1) A hybrid RAG architecture that integrates semantic vector retrieval and full-text retrieval is proposed. Combining re-ranking algorithms such as RRF with search filtering mechanisms significantly improves retrieval accuracy and fault tolerance, effectively alleviating the large model hallucination problem. This architecture achieves unified embedding and cross-modal association of multimodal data (text, tables, and drawings), establishes an industry knowledge base framework that supports semantic understanding, and provides multi-dimensional information support for complex decision-making.

[0120] (2) Build a database system based on MongoDB (content repository) and PostgreSQL (vector retrieval library) to support the efficient management and dynamic updating of heterogeneous data such as railway design specifications, operation and maintenance reports, and engineering cases. Through optimization strategies such as table simplification and image summary generation, solve the problem of parsing and retrieving non-text data, and improve the practicality and coverage of the knowledge base.

[0121] (3) In scenarios such as railway line design, track maintenance, and train traction testing, the system accurately outputs regulatory clauses, defect treatment solutions, and energy-saving technology basis to ensure that answers meet industry standards and reduce human experience bias. It supports multi-platform deployment (web, mobile, and third-party software), enabling designers to query knowledge across scenarios in real time, improving design efficiency and compliance, and providing a scalable technical paradigm for design optimization, fault prediction, and operation and maintenance decision-making, helping the railway system upgrade to a digital, knowledge-driven model.

[0122] The above is a schematic description of the present invention and its embodiments, which is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. Therefore, if a person skilled in the art is inspired by this and, without departing from the purpose of the present invention, designs a structure and embodiment similar to this technical solution without inventiveness, they shall fall within the scope of protection of the present invention.

Claims

1. A method for constructing a railway multimodal knowledge base question-answering system based on a hybrid RAG architecture, characterized by: The following steps are involved: Step 1: Server local model deployment and fine-tuning; The deployed models include language models and text embedding models; Fine-tuning is based on the DyLoRA fine-tuning framework, integrating dynamic rank allocation, hierarchical adaptation, and expert routing mechanisms to improve model performance while maintaining parameter efficiency; Step 2: Data processing and database construction; The data includes railway feasibility studies, track operation and maintenance reports, various industry design standards in docx and pdf formats, as well as design drawings, specification tables, and images in various formats; After deploying the language model and text embedding model locally, we used open source projects to manage the model interfaces and established a railway specification knowledge base database management system based on MongoDB content storage and PostgreSQL vector retrieval libraries. PostgreSQL used vector fields to store vectors, while MongoDB was used to access the original vector data. During retrieval, the vectors were first retrieved, and then the original data content was searched in MongoDB based on the vector ID. For multiple sets of retrieved data, the search results were optimized and merged using a hybrid search and RRF algorithm, followed by re-ranking and search filtering. Step 3: Multimodal knowledge base construction; Using multimodal embedding, images and corresponding text are embedded in a vector database. The original images and text are stored in the document store. When performing hybrid similarity retrieval, the corresponding images in the document store are directly retrieved, and the original images and text blocks are passed to the large model to generate answers. For macro images, a text summary is generated for the image based on the multimodal large model, and the text summary is embedded and retrieved using a text embedding model. Step 4: Multi-platform expansion.

2. The method for constructing a railway multimodal knowledge base question-answering system based on a hybrid RAG architecture according to claim 1, characterized in that: In step 1, the language model is an autoregressive language model based on the Transformer architecture. It is pre-trained based on massive text data to learn the statistical laws, world knowledge, and complex patterns of language. The core task of the text embedding model is to convert a piece of text into a fixed-length, dense vector for embedding.

3. The method for constructing a railway multimodal knowledge base question-answering system based on a hybrid RAG architecture according to claim 2, characterized in that: In step 1, the formula of the DyLoRA fine-tuning framework is as follows: ； in, The output of model fine-tuning; These are the pre-trained parameters that remain unchanged during fine-tuning; Represents the learnable parameters of each group of experts in the model expert routing system; and are two low-rank matrices trained when fine-tuning the model; represents rank, is the minimum rank, is the maximum value; Indicates the Sensitivity score of the layer network; is the sparse coefficient; represents transpose, Represents a natural constant.

4. The method for constructing a railway multimodal knowledge base question-answering system based on a hybrid RAG architecture according to claim 3 is characterized by: In step 1, fine-tuning uses the instruction supervision fine-tuning dataset, which includes 10,000 railway industry corpora of industry specifications, feasibility studies, and operation and maintenance reports. The instruction supervision fine-tuning dataset is shown in the following formula: ； in, Indicated by A dataset consisting of independent samples; Represents each triplet sample; Indicates the Instructions for samples to guide model behavior; Indicates the The input of samples provides the specific context or problem of the task; Indicates the The output of each sample is the expected response or answer; Core training parameters include learning rate, batch size, number of training rounds, maximum number of samples, and gradient clipping, which can be flexibly adjusted according to model parameters and dataset size.

5. The method for constructing a railway multimodal knowledge base question-answering system based on a hybrid RAG architecture according to claim 4 is characterized by: Hybrid search includes semantic vector search and full-text sparse search. Semantic vector search calculates the distance between spatial vectors to obtain similarity, while full-text sparse search uses a keyword search method based on a sparse algorithm. After obtaining the results of the two searches, the relevant content is input as prompt words into the large model as a reference for the output through normalization, re-ranking and search filtering; Reranking is a method of combining multiple result sets with different relevance metrics into a single result set; Search filtering uses citation caps and minimum relevance to improve the quality of retrieval; Among them, semantic vector search calculates index items With candidate documents The semantic vector similarity score between , the formula is as follows: ； Where, is the query vector, For each candidate document vector; full-text sparse retrieval calculates each candidate document Sparse retrieval score , whose formula is as follows: ； Where, For index items a term of ; For terms In the documentation word frequency in ; For Documents length; is the average document length of the document collection; Control word frequency saturation, the larger the The greater the impact; Controls the strength of document length normalization; For terms The inverse document frequency of is the total number of documents in the document collection; is the number of documents containing the term; After obtaining all document sparsity scores and vector similarity scores, the formula for score normalization is as follows: ； Where, is the index item; For all candidate documents; and Represent the mean of all document sparse scores and vector similarity scores in the current candidate pool respectively; and and Represent the standard deviation of the sparsity scores and vector similarity scores of all documents in the current candidate pool.

6. The method for constructing a railway multimodal knowledge base question-answering system based on a hybrid RAG architecture according to claim 5, characterized in that: In step 2, the text is segmented and sliced ​​according to the rules and converted into a slice format that can be searched semantically. Each slice data in the database is fine-tuned and corrected individually. Methods for optimizing the knowledge database structure include merging and organizing similar related content, handling situations where default questions cannot be matched, and optimizing table name and table body matching errors.

7. The method for constructing a railway multimodal knowledge base question-answering system based on a hybrid RAG architecture according to claim 6, characterized in that: In step 2, the table data is optimized by simplifying the table and converting the table into text. For large tables, the table model is used to encode the table first and then embed it. Table encoding first performs structural analysis, including identifying the row and column structure of the table, parsing merged cells, and extracting hierarchical relationships. Then, semantic encoding is performed, including linearization of the representation function and semantic embedding representation process. The formula is as follows: ； in, express Matrix, starting row index , starting column index ; and Respectively represent the total number of rows and columns in the logical area covered by table cells; and The number of cells merged horizontally and vertically respectively; represents the generating function of the linearized sequence; It is a cell text collection; is the string concatenation operator; is the column separator; is a unit separator; Represents the tensor product operation; Represents text generated by a pre-trained embedding model Embedding vector; Represents the word segmentation function for text segmentation; Indicates that the parameter is Embedding model encoder.

8. The method for constructing a railway multimodal knowledge base question-answering system based on a hybrid RAG architecture according to claim 7, characterized in that: In step 3, the input data is image-text pairs, where the original image , The formulas generated for associating text, preprocessing and embedding are as follows: ； in, Indicates whether the image type is a macro image; Indicates the use of a pre-trained image classification model; represents an image that is identified as a macro image by the classification model; is the Softmax function; is the macro image threshold, if , then trigger summary generation ; Represents a multimodal large model; The maximum number of digest tokens; Represents the image embedding vector and aligned text embedding vectors The joint embedding vector of and Represent the embedding modality weights of image and text respectively; is the original text embedding vector that is not aligned with the image vector in the same dimension. is the projection matrix used to align the dimensions, and its parameters are obtained through linear regression training; and They are image embedding model and text embedding model respectively; and They are the graphic embedding dimension and the text embedding dimension respectively; For CAD vector graphics format, the B-spline curve analytical method is adopted to construct the characteristic vector formula as follows: ； in, It means a Secondary B-spline curve, For the definition on Sub-B-spline basis functions ; The characteristic vector obtained by linear transformation is the curve parameters spliced ​​into a vector in a fixed order; represents the node vector; is the control point; is the weight factor corresponding to the B-spline.

9. The method for constructing a railway multimodal knowledge base question-answering system based on a hybrid RAG architecture according to claim 8, characterized in that: In step 4, the local framework is expanded to web pages, WeChat official accounts, mini-programs, and third-party software. The core components include a protocol-adaptive API gateway and asynchronous messaging middleware. A unified API gateway architecture is used to achieve multi-terminal access. The gateway has a built-in protocol conversion layer that supports unified conversion of HTTP / WebSocket heterogeneous protocols to gRPC protocol. The protocol conversion process is done by inputting a normalization function As shown below: ； in, Represents input data; when When coming from HTTP protocol, that is , perform serialization conversion from JSON to ProtoBuf; when the input data comes from the WeChat ecosystem, that is, , perform XML to ProtoBuf conversion; For heterogeneous input data sets, the protocol conversion delay is controlled within 15ms. ; For high-concurrency scenarios, message queues are introduced for asynchronous peak shaving. The message processing delay model is as follows: ； in, Indicates the total message processing time; Indicates the serialization time of converting the original message data into a standardized format for transmission or storage; Indicates the number of messages; is the number of partitions; This is the single message consumption latency.

Images