A private fusion scheduling method and system of a multi-source heterogeneous large language model for a railway intranet

Abstract

The application discloses a railway intranet-oriented multi-source heterogeneous large language model privatization fusion scheduling method and system, which comprises a model registration center, a request access gateway, a task classification module, a dynamic routing module, an inference engine management module, a response aggregation module, a GPU resource scheduling module and an operation and maintenance monitoring module. It has the following advantages: 1) all large language models are privately deployed on railway intranet GPU servers, and inference requests and business data are kept within the intranet throughout the process, eliminating data leakage and compliance risks from the physical isolation level. 2) Multiple models are used to cover all railway scenarios. Compared with using a single model, the comprehensive task accuracy of the dynamic routing scheme is improved from 82.4% / 79.6% to 96.3%. 3) High-concurrency stable inference service is realized under limited GPU resources. 4) Support for model hot swapping to adapt to the continuous evolution of railway business.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and railway information technology, specifically to a private fusion scheduling method and system for multi-source heterogeneous large language models in railway intranets. Background Technology

[0002] In railway transportation scheduling, passenger service, material management, safety monitoring and other business scenarios, a large number of tasks such as unstructured text processing, multimodal understanding, knowledge question answering, and decision support urgently require the support of large language models (LLMs). Currently, the industry mainly calls large language model services through public cloud APIs (such as OpenAI API, Baidu Wenxin Yiyan API, etc.) or deploys open source models in general Internet environments.

[0003] However, existing large language model deployment and invocation schemes have the following significant drawbacks in practical applications within the railway industry:

[0004] 1) The fundamental contradiction between data security and cloud reliance: The railway industry involves highly sensitive information such as train timetables, dispatching instructions, passenger information, and material procurement data. According to the "Regulations on the Security Protection and Management of Critical Information Infrastructure in Railways" and the requirements of Level 3 Information Security Protection, business data is strictly prohibited from leaving the network. However, public cloud big data model API solutions must transmit business data to cloud inference servers via the internet, posing serious security risks such as data leakage and man-in-the-middle attacks. Even with VPN encrypted channels, it is still impossible to avoid compliance issues related to data storage on third-party cloud service providers. Currently, there is no mature and feasible solution that can both utilize big data model capabilities and fully meet the requirement of keeping railway intranet data within the network.

[0005] 2) A single model cannot cover the diverse business needs of the railway: Due to differences in pre-training data distribution, model architecture, and parameter scale, different large language models perform inconsistently across different task types. For example, Qwen2.5-72B performs excellently in Chinese text understanding and knowledge question answering, but is inferior to DeepSeek R1 in complex multi-step reasoning (such as transportation plan scheduling optimization); DeepSeek R1 has significant advantages in logical reasoning and mathematical computation, but does not support image input; Qwen2.5VL has visual-language multimodal understanding capabilities and can handle visual tasks such as ticket images and equipment inspection photos, but is less efficient than pure text models in long document summarization tasks. Railway business scenarios cover a variety of task types, including text question answering, document summarization, logical reasoning, and image understanding, and a single model cannot achieve optimal performance for all scenarios.

[0006] 3) Lack of effective resource management mechanisms for multiple models coexisting in an intranet environment: Given the limited GPU server resources within the railway intranet (typically 2-4 servers equipped with NVIDIA A100 / A800 GPUs), simultaneously loading multiple high-parameter models (e.g., Qwen2.5-72B requires approximately 140GB of VRAM, DeepSeek R1-70B requires approximately 140GB of VRAM) will lead to VRAM overflow or severe resource contention. Existing open-source inference frameworks (vLLM, TGI, etc.) only support inference services for single model instances, lacking cross-model VRAM scheduling, request routing, and elastic scaling capabilities. Simply allocating fixed GPU resources independently to each model will result in resource waste or request queuing timeouts during periods of fluctuating business load (e.g., a large number of text tasks during peak daytime periods, or batch image recognition tasks at night). Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a private fusion scheduling method and system for multi-source heterogeneous large language models in railway intranets, which can effectively solve the above-mentioned problems.

[0008] The technical solution adopted in this invention is as follows:

[0009] In a first aspect, the present invention provides a private fusion scheduling method for multi-source heterogeneous large language models in railway intranets, comprising:

[0010] Step S1: The model registry maintains the configuration and status data of deployed model instances, including model instance metadata, deployment status, and load status; the model instance metadata includes model instance identifier, competent task type tag, and inference engine type; the deployment status includes loaded status, unloaded status, and loading status; the competent task type tag includes at least one competent task type.

[0011] Step S2: When the request access gateway receives an inference request initiated by the railway intranet business system, it performs request authentication, request rate limiting, and request format standardization operations on the inference request to obtain a standardized inference request with a unified inference request format, and then passes it to the task classification module.

[0012] Step S3: The task classification module determines the task type of the standardized inference request, obtains a task type tag, appends the task type tag to the metadata of the standardized inference request, obtains a tagged inference request, and passes it to the dynamic routing module.

[0013] Step S4: The dynamic routing module reads the configuration and status data of each deployed model instance in the model registry center, executes the task-aware model routing strategy, routes out the target model instance, and sends the target model instance identifier and the tagged inference request to the inference engine management module.

[0014] The execution task-aware model routing strategy, which routes out the target model instance, includes:

[0015] The decision determines whether there exists a model instance that is task type compatible and whose deployment status is either loaded or loading; the task type compatibility refers to the task type tag in the tagged inference request being located in the model instance's proficient task type tag.

[0016] If it exists, all model instances that meet the conditions will be used as candidate model instances. Taking into account the load status and historical average response time, the best candidate model instance will be selected as the target model instance.

[0017] If it does not exist, the GPU resource scheduling module is triggered to execute the model instance hot loading process, load the model weight file of the model instance that is suitable for the task type and is in the unloaded state into the GPU memory, and update the corresponding model instance in the model registry to the loaded state. At this time, the model instance becomes the target model instance.

[0018] Step S5: The inference engine management module searches the model registry center based on the target model instance identifier to obtain the inference engine type that executes the target model instance, and then determines the corresponding target inference engine instance, and adds the tagged inference request to the request queue of the target inference engine instance.

[0019] When the labeled reasoning request is processed, the target reasoning engine instance performs forward reasoning on the labeled reasoning request based on the target model instance to obtain a reasoning result containing the generated text and reasoning meta-information, and sends it to the response aggregation module.

[0020] In step S6, the response aggregation module performs post-processing on the inference result to obtain a response message and returns it to the request access gateway, which then pushes the response message to the railway intranet business system that initiated the request.

[0021] Furthermore, the configuration and status data of the model instance also includes: the GPU device number currently occupied by the model instance and the amount of video memory required for operation, the current number of concurrent requests of the model instance, the historical average response time, the health check status, and the model name, model type, and model parameter scale of the large language model corresponding to the model instance.

[0022] Furthermore, in the deployment state, the loaded state means that the model weight file of the model instance has been loaded into the GPU memory; the unloaded state includes two types: one is that the model weight file has been unloaded from the GPU memory to the system memory, and the other is that the model weight file has been unloaded from the system memory to the disk; the loading state means that the model weight file is being loaded from the disk or the system memory into the GPU memory.

[0023] The health check status of the model instance includes a healthy status and an unhealthy status;

[0024] The model types include plain text models and multimodal graph-text models; the plain text models are used to handle plain text tasks that do not contain images, including one or more of TextQA, TextSummary, LogicReason, and CodeGen; the multimodal graph-text models are used to handle tasks that contain images; the inference engine type refers to the type of inference engine that runs the model, and the same type of inference engine corresponds to one or more models.

[0025] Furthermore, the inference request is subjected to request authentication, request rate limiting, and request format standardization operations to obtain a standardized inference request with a unified inference request format, including:

[0026] The inference request is authenticated, specifically by token verification based on the railway intranet unified identity authentication platform.

[0027] The inference request is subject to request rate limiting, specifically by using a token bucket algorithm to control the upper limit of concurrent requests;

[0028] The request format standardization operation is performed on the inference request, specifically converting heterogeneous inference requests from different business systems into a unified request operation that conforms to the inference request protocol.

[0029] The obtained standardized inference request with a unified inference request format has the following fields: request unique identifier request_id; input text input_text; input image list input_images, which can be empty; source business system identifier source_system; request priority and maximum number of output tokens max_tokens.

[0030] Furthermore, the task classification module determines the task type of the standardized inference request to obtain a task type label, including:

[0031] Step S31: Determine whether the standardized inference request contains image input: If the input image list input_images is not empty, it indicates that the request contains image input, and proceed to step S32; if the input image list input_images is empty, it indicates that the request does not contain image input, and proceed to step S33.

[0032] Step S32: Determine if the request includes image input, and directly determine the task type as MultiModal (Multi-Modal).

[0033] Step S33: If the request does not contain image input, then perform lightweight text classification on the input text input_text. The method is as follows: extract the TF-IDF feature vector of the input text input_text and feed it into a pre-trained SVM classifier; the SVM classifier outputs the probability distribution of each task type, and takes the category with the highest probability as the determined task type; where each task type includes one or more of the following: TextQA, TextSummary, LogicReason, and CodeGen.

[0034] Furthermore, the step of selecting all model instances that meet the conditions as candidate model instances, and comprehensively considering the load status and historical average response time to determine the best candidate model instance as the target model instance, includes:

[0035] The following formula is used to calculate the routing score for each candidate model instance, and the candidate model instance with the highest score is selected as the target model instance:

[0036] Score(m) = α × TaskMatch(m) + β × (1 - Load(m)) + γ × (1 / AvgLatency(m))

[0037] Where: Score(m) represents the routing score of candidate model instance m;

[0038] TaskMatch(m)∈ {0, 1} represents the matching degree between the candidate model instance m and the current task type. If the current task type label is in the good task type label of the candidate model instance m, it is 1; otherwise, it is 0.

[0039] Load(m) is the current load rate of candidate model instance m, with a value range of [0, 1]. Load(m) = current_concurrency(m) / max_concurrency(m), where current_concurrency(m) is the current number of concurrent requests of candidate model instance m, and max_concurrency(m) is the maximum number of concurrent requests supported by candidate model instance m.

[0040] AvgLatency(m) is the historical average response time of candidate model instance m;

[0041] α, β, and γ are configurable weighting coefficients, with the default values ​​being α=0.5, β=0.3, and γ=0.2.

[0042] Furthermore, when the target inference engine instance performs forward inference, for models with streaming output, the inference results are transmitted back token by token via the SSE protocol.

[0043] Furthermore, the GPU resource scheduling module is specifically used for:

[0044] Monitor the real-time resource usage status of all GPU devices, including memory usage, GPU utilization, and temperature. Combine this with the configuration and status data of model instances in the model registry center and business load trends to execute elastic allocation strategies for GPU memory resources.

[0045] Step A1, Memory Pool Management:

[0046] The video memory of all available GPU devices is abstracted into a unified GPU video memory resource pool, and the total video memory capacity V_total(g), the allocated video memory capacity V_alloc(g), and the available video memory capacity V_free(g) of each GPU device are recorded.

[0047] Step A2, Elastic Scaling Strategy:

[0048] Let the average load rate of all model instances corresponding to the same model within the time window T be Load_avg(model, T);

[0049] If Load_avg(model, T_scale) > H_high and the current GPU memory resource pool has remaining space, horizontal scaling of model instances is triggered. A copy of the same model is launched on an idle GPU device and registered with the model registry center to achieve load balancing distribution of requests. Here, Load_avg(model, T_scale) > H_high means that the average load rate of the model continuously exceeds the high watermark threshold H_high for the time window of T_scale.

[0050] If Load_avg(m, T_shrink) < H_low and there are more than one active model instance for the same model, the model instance is shrunk, the redundant model instances of the same model are closed, and all model instances are gracefully closed after the existing requests are processed, releasing GPU memory; where Load_avg(m, T_shrink) represents the time window during which the average load rate of the model is continuously lower than the low water level threshold H_low.

[0051] When switching business load modes, the scaling up and down of model instances is automatically triggered by changes in load rate, enabling dynamic migration of GPU resources between different model instances;

[0052] Step A3, Hot loading and cold unloading of model instances:

[0053] When the GPU resource scheduling module receives a loading request for the target model instance m, it compares the available memory capacity V_free(g) of the current GPU memory resource pool with the memory usage V_required(m) required by the target model instance m.

[0054] If V_free(g) ≥ V_required(m), perform hot loading of model instances: traverse GPU devices to select the best GPU combination that satisfies V_free(g) ≥ V_required(m). The best GPU combination is the combination with the largest available video memory capacity to reduce fragmentation; load the model weight file of the target model instance m into the best GPU combination, and update the target model instance m in the model registry to the loaded state;

[0055] If V_free < V_required, the LRU cold unloading strategy is executed. The currently loaded model instances are sorted by the least recently used time, and the low-priority and least recently used model instances are unloaded in turn until V_free ≥ V_required, and then the model instances are hot-loaded.

[0056] Specifically: When executing the LRU cold unloading strategy, a tiered unloading strategy is executed when the model instance is unloaded. The first level of unloading transfers the model weights from GPU memory to system memory, preserving the ability to quickly recover. When system memory is insufficient, the second level of unloading is executed, writing the model weights from memory back to disk, completely releasing resources.

[0057] Step A4, GPU memory fragmentation management:

[0058] Continuously monitor the GPU memory fragmentation metric Frag(g) = 1 - MaxContiguousFree(g) / V_free(g), where MaxContiguousFree(g) is the maximum value of continuous available space. When Frag(g) > 0.3, memory defragmentation is triggered, and a brief unload-reload operation is performed on low-load model instances to merge fragmented free memory blocks.

[0059] Secondly, this invention provides a private fusion scheduling system for multi-source heterogeneous large language models oriented towards railway intranets. The system is deployed within the railway intranet and includes:

[0060] The model registry center is used to maintain the configuration and status data of deployed model instances, including model instance metadata, deployment status, and load status. The model instance metadata includes model instance identifier, task proficiency type tag, and inference engine type. The deployment status includes loaded, unloaded, and loading status. The task proficiency type tag includes at least one task proficiency type.

[0061] The request access gateway is used to receive inference requests initiated by the railway intranet business system, perform request authentication, request rate limiting and request format standardization operations on the inference requests, obtain standardized inference requests with a unified inference request format, and pass them to the task classification module.

[0062] The task classification module is used to determine the task type of the standardized inference request, obtain a task type tag, attach the task type tag to the metadata of the standardized inference request, obtain a tagged inference request, and pass it to the dynamic routing module.

[0063] The dynamic routing module is used to read the configuration and status data of each deployed model instance in the model registry center, perform routing decisions, route the target model instance, and send the target model instance identifier and the tagged inference request to the inference engine management module.

[0064] The execution of routing decisions, routing out the target model instance, includes:

[0065] The decision determines whether there exists a model instance that is task type compatible and whose deployment status is either loaded or loading; the task type compatibility refers to the task type tag in the tagged inference request being located in the model instance's proficient task type tag.

[0066] If it exists, all model instances that meet the conditions will be used as candidate model instances. Taking into account the load status and historical average response time, the best candidate model instance will be selected as the target model instance.

[0067] If it does not exist, the GPU resource scheduling module is triggered to execute the model instance hot loading process, load the model weight file of the model instance that is suitable for the task type and is in the unloaded state into the GPU memory, and update the corresponding model instance in the model registry to the loaded state. At this time, the model instance becomes the target model instance.

[0068] The inference engine management module is used to find the model registry center based on the target model instance identifier, obtain the inference engine type that executes the target model instance, and then determine the corresponding target inference engine instance. The tagged inference request is added to the request queue of the target inference engine instance. When the tagged inference request is processed, the target inference engine instance performs forward inference on the tagged inference request based on the target model instance to obtain the inference result containing the generated text and inference meta-information, and sends it to the response aggregation module.

[0069] The response aggregation module is used to post-process the inference results, obtain a response message, and return it to the request access gateway, which then pushes the response message to the railway intranet business system that initiated the request.

[0070] The invention provides a private fusion scheduling method and system for multi-source heterogeneous large language models in railway intranets, which has the following advantages:

[0071] 1) Completely resolves the conflict between railway data security and large-scale model capabilities. All large language models are privately deployed on the railway's intranet GPU servers. Inference requests and business data never leave the intranet, eliminating data leakage and compliance risks at the physical isolation level. The system supports the operation of purely domestic large models (Qwen2.5, DeepSeek R1, etc.), eliminating dependence on overseas models and public cloud APIs, and meeting the railway industry's requirements for domestic self-reliance and control.

[0072] 2) Multi-model collaboration covers all railway business needs. Through a task-aware dynamic routing mechanism, text-based question-and-answer requests are automatically routed to Qwen2.5-72B, logic reasoning requests to DeepSeek R1, and multi-modal requests involving text and images to Qwen2.5VL. On an evaluation set covering 800 railway business test requests, compared to using a fixed single model, the overall task accuracy of the dynamic routing scheme improved from 82.4% / 79.6% to 96.3%.

[0073] 3) Achieving high-concurrency and stable inference services with limited GPU resources. In a typical railway intranet hardware configuration with four NVIDIA A800-80GB servers (16 GPUs in total), the system can simultaneously maintain 2-3 online inference service instances for large models through a GPU memory elastic allocation mechanism. In actual testing under 50 concurrent request loads, the P95 response time for text requests was 1.05 seconds, and the P95 response time for multimodal requests was 1.18 seconds, both meeting the 1.2-second business SLA requirement. Compared to a fixed resource allocation scheme, the GPU resource utilization rate of this invention increased from 47.2% to 81.6%, and the maximum supported concurrency under the same hardware conditions increased from 28 to 56.

[0074] 4) Supports hot-swappable models to adapt to the continuous evolution of railway operations. The model registry supports dynamic registration / deregistration of model instances. When railway business departments introduce new large language models, they only need to add model metadata and deploy model weight files in the model registry, without modifying system code or restarting services to access the scheduling system. Attached Figure Description

[0075] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0076] Figure 1 This is a schematic diagram of the principle of the private fusion scheduling method for multi-source heterogeneous large language models for railway intranet provided by the present invention.

[0077] Figure 2 The flowchart of task-aware dynamic routing and elastic scheduling provided by this invention;

[0078] Figure 3 The flowchart of GPU memory elastic allocation and model hot loading provided by this invention;

[0079] Figure 4 This is a schematic diagram of dynamic routing score calculation provided by the present invention. Detailed Implementation

[0080] To make the technical problems solved, the technical solutions, and the beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0081] This invention provides a private fusion scheduling method and system for multi-source heterogeneous large language models in railway intranets. Under the completely isolated and secure environment of the railway intranet, it enables the private deployment and collaborative operation of multiple large language models (text models and visual-language models) through a unified scheduling platform. Through a task-aware dynamic model routing strategy, it automatically selects the most suitable model instance based on the type of task requested by the user. Through a GPU resource elastic allocation mechanism, it supports stable inference services with 50 concurrent requests (P95 response time ≤ 1.2 seconds) under limited hardware resources, meeting the high security, low latency, and high availability business requirements of the railway industry.

[0082] See Figures 1-4 This invention provides a private fusion scheduling method for multi-source heterogeneous large language models in railway intranets. It relates to a method and system for the private deployment, unified scheduling, dynamic switching, and elastic resource allocation of multi-source heterogeneous large language models (Qwen2.5, DeepSeekR1, Qwen2.5VL, etc.) in a high-security environment of railway intranets, including steps S1 to S6:

[0083] Step S1: The model registry maintains the configuration and status data of all deployed model instances, including but not limited to model instance metadata, deployment status, and load status. The model instance metadata includes the model instance identifier, the preferred task type tag, and the inference engine type (vLLM / TGI). The deployment status includes loaded, unloaded, and loading states. The preferred task type tag includes at least one preferred task type. As one implementation, the configuration and status data of the model instance also includes: the GPU device number currently occupied by the model instance and the amount of video memory required for operation, the current number of concurrent requests for the model instance, the historical average response time, the health check status, and the model name, model type (plain text / multimodal), and model parameter scale of the large language model corresponding to the model instance. The configuration and status data of deployed model instances maintained by the model registry provides a decision-making basis for the dynamic routing module and the GPU resource scheduling module; it supports dynamic registration of new model instances or deregistration of existing model instances without restarting the system.

[0084] In the deployment state, the loaded state means that the model weight file of the model instance has been loaded into the GPU memory; the unloaded state includes two types: one is that the model weight file has been unloaded from the GPU memory to the system memory, and the other is that the model weight file has been unloaded from the system memory to the disk; the loading state means that the model weight file is being loaded from the disk or the system memory into the GPU memory.

[0085] The health check status of the model instance includes a healthy status and an unhealthy status;

[0086] The model types include plain text models and multimodal graph-text models; the plain text models are used to handle plain text tasks that do not contain images, including one or more of TextQA, TextSummary, LogicReason, and CodeGen; the multimodal graph-text models are used to handle tasks that contain images; the inference engine type refers to the type of inference engine that runs the model, and the same type of inference engine corresponds to one or more models.

[0087] As one example, the model registry center uses the following data structure to define configuration and status data:

[0088] {

[0089] "model_id": "qwen25-72b-inst-01",

[0090] "model_name": "Qwen2.5-72B-Instruct",

[0091] "model_type": "text_only",

[0092] "param_scale": "72B",

[0093] "specialty_tasks": ["TextQA", "TextSummary", "CodeGen"],

[0094] "deploy_status": "loaded",

[0095] "gpu_devices": ["GPU:0", "GPU:1"],

[0096] "vram_usage_gb": 138.5,

[0097] "inference_engine": "vLLM",

[0098] "current_concurrency": 12,

[0099] "avg_response_ms": 850,

[0100] "health_status": "healthy",

[0101] "is_active": true

[0102] }

[0103] Step S2, Request Access and Standardization: When the request access gateway receives an inference request initiated by the railway intranet business system, it performs request authentication, request rate limiting and request format standardization operations on the inference request to obtain a standardized inference request with a unified inference request format, and passes it to the task classification module;

[0104] Specifically, the access gateway, as the unified entry point of the system, receives inference requests initiated by various business systems of the railway intranet (dispatch and command system, passenger service system, material management system, safety monitoring system, etc.) through a unified API interface; performs request authentication, request rate limiting, and request format standardization to obtain standardized inference requests with a unified inference request format, and then passes the standardized requests to the task classification module.

[0105] As one implementation, request authentication is based on token verification using the unified identity authentication platform within the railway intranet, verifying identity legitimacy through token authentication. Request rate limiting can be achieved by using a token bucket algorithm to control the upper limit of concurrent requests; for example, bucket capacity B=100, token replenishment rate R=50 / second, thereby controlling the number of concurrent requests to not exceed the system's capacity limit. Request format standardization involves converting heterogeneous inference requests from different business systems into a unified inference request protocol.

[0106] The standardized inference request with a unified inference request format has the following fields: request_id (unique identifier); input_text (input text); input_images (input image list, which can be empty); source_system (source business system identifier); request priority; and maximum number of output tokens (max_tokens).

[0107] Step S3, Task type identification: The task classification module determines the task type of the standardized inference request transmitted by the request access gateway, obtains a task type tag, appends the task type tag to the metadata of the standardized inference request, obtains a tagged inference request, and passes it to the dynamic routing module.

[0108] As one embodiment, the task classification module uses a lightweight task type classifier to determine the task type of a request:

[0109] Parse the input modal features (whether it contains image attachments), prompt keyword features (whether it contains logical reasoning keywords such as "reasoning", "calculation", "arrangement") and business system source identifier in the request body;

[0110] Based on a combination strategy of rule engine and lightweight text classification model (based on TF-IDF + SVM, model size <5MB, inference latency <5ms), requests are classified into one of the following task types: TextQA (text question answering), TextSummary (document summarization), LogicReason (logical reasoning and computation), MultiModal (multimodal text and image understanding), and CodeGen (code generation and assistance).

[0111] In another embodiment, the task classification module determines the task type of the standardized inference request and obtains a task type label, including:

[0112] Step S31: Determine whether the standardized inference request contains image input: If the input image list input_images is not empty, it indicates that the request contains image input, and proceed to step S32; if the input image list input_images is empty, it indicates that the request does not contain image input, and proceed to step S33.

[0113] Step S32: If the request contains image input, then the task type is directly determined to be MultiModal (Multi-Modal).

[0114] Step S33: Determine that the request does not contain image input. Perform lightweight text classification on the input text input_text. The method is as follows: extract the TF-IDF feature vector (dimension=2000) of the input text input_text and feed it into a pre-trained SVM classifier. The SVM classifier outputs the probability distribution of each task type, and takes the category with the highest probability as the determined task type. At the same time, the business source identified by source_system is referred to assist in the determination. Among them, each task type includes one or more of the following: text question answering (TextQA), document summarizing (TextSummary), logical reasoning and computation (LogicReason), and code generation and assistance (CodeGen).

[0115] Step S4, Model Routing Decision: The dynamic routing module reads the configuration and status data of each deployed model instance in the model registry center, executes the task-aware model routing strategy, routes out the target model instance, and sends the target model instance identifier and the tagged inference request to the inference engine management module.

[0116] The execution task-aware model routing strategy, which routes out the target model instance, includes:

[0117] The decision determines whether there exists a model instance that is task type compatible and whose deployment status is either loaded or loading; the task type compatibility refers to the task type tag in the tagged inference request being located in the model instance's proficient task type tag.

[0118] If it does not exist, the GPU resource scheduling module is triggered to execute the model instance hot loading process, load the model weight file of the model instance that is suitable for the task type and is in the unloaded state into the GPU memory, and update the corresponding model instance in the model registry to the loaded state. At this time, the model instance becomes the target model instance.

[0119] If such a model exists, all model instances that meet the conditions will be considered as candidate model instances. Taking into account both load status and historical average response time, the best candidate model instance will be selected as the target model instance. Specifically, this includes:

[0120] The following formula is used to calculate the routing score for each candidate model instance, and the candidate model instance with the highest score is selected as the target model instance:

[0121] Score(m) = α × TaskMatch(m) + β × (1 - Load(m)) + γ × (1 / AvgLatency(m))

[0122] Where: Score(m) represents the routing score of candidate model instance m;

[0123] TaskMatch(m)∈ {0, 1} represents the matching degree between the candidate model instance m and the current task type. If the current task type label is in the good task type label of the candidate model instance m, it is 1; otherwise, it is 0.

[0124] Load(m) is the current load rate of candidate model instance m, with a value range of [0, 1]. Load(m) = current_concurrency(m) / max_concurrency(m), where current_concurrency(m) is the current number of concurrent requests of candidate model instance m, and max_concurrency(m) is the maximum number of concurrent requests supported by candidate model instance m.

[0125] AvgLatency(m) is the historical average response time (in seconds) of candidate model instance m.

[0126] α, β, and γ are configurable weighting coefficients, with the default values ​​being α=0.5, β=0.3, and γ=0.2.

[0127] As one implementation method, the health status of model instances can be further used as a condition for filtering candidate model instances. Therefore, the task-aware model routing strategy can be summarized as follows:

[0128] First, based on the requested task type tag, a set of candidate models S_candidate is selected from the model registry that are proficient in that task type and whose current status is "loaded" and "healthy". Therefore, S_candidate = {m | task_type ∈ m.specialty_tasks ∧ m.deploy_status = "loaded" ∧m.health_status = "healthy"};

[0129] If the candidate model set S_candidate is not empty, the aforementioned route score calculation formula is used to calculate the route score Score(m) for each candidate model instance in the candidate model set.

[0130] Select the candidate model instance with the highest Score(m) as the target model instance m_target, and route the request to the inference engine instance corresponding to the target model instance.

[0131] If the candidate model set S_candidate is empty (i.e. there are no adapted models that have been loaded), the GPU resource scheduling module will be triggered to execute the hot loading process of model instances.

[0132] Step S5, Model Inference Execution: The inference engine management module searches the model registry center based on the target model instance identifier to obtain the inference engine type for executing the target model instance, and then determines the corresponding target inference engine instance, adding the tagged inference request to the request queue of the target inference engine instance.

[0133] When processing the labeled inference request, the target inference engine instance performs forward inference on the labeled inference request based on the target model instance. For example, the target inference engine instance (vLLM or TGI) performs efficient KV-Cache management based on the PagedAttention mechanism, performs forward inference on the request, and obtains an inference result containing generated text and inference metadata, such as the actual number of tokens and inference time, and sends it to the response aggregation module. When the target inference engine instance performs forward inference, for models with streaming output, it transmits the inference result token by token through the SSE protocol.

[0134] In practical applications, the inference engine management module has the following functions:

[0135] Manage the lifecycle of all large language model inference engine instances; specifically:

[0136] Maintain an independent inference engine process (based on vLLM or TGI) for each loaded model instance, and configure runtime parameters such as model weight path, tensor parallelism (tensor_parallel_size), maximum number of concurrent requests (max_num_seqs), and maximum GPU memory utilization of KV-Cache (gpu_memory_utilization);

[0137] It provides a unified inference API interface (compatible with OpenAI Chat Completions format), receives inference requests forwarded by the dynamic routing module, calls the inference engine instance of the corresponding model instance to perform forward inference, and returns the generated results;

[0138] For models with streaming output, the SSE (Server-Sent Events) protocol is used to achieve per-token streaming back, reducing the user-perceived first-to-first-token (TTFT) latency.

[0139] When the GPU resource scheduling module receives the unload instruction, it performs the operation of unloading the model weights from the GPU memory to the system memory or disk, thereby releasing the GPU resources.

[0140] Step S6, Response Feedback and Log Recording: The response aggregation module performs post-processing on the inference result to obtain a response message and returns it to the request access gateway, which then pushes the response message to the railway intranet business system that initiated the request.

[0141] In practical applications, the response aggregation module receives the inference results, performs sensitive information filtering (filtering sensitive fields such as ID card numbers and bank card numbers based on regular expression matching and keyword library), encapsulates the results into a standardized response message (containing fields such as request_id, output_text, model_used, inference_time_ms, and token_count), and sends it back to the business system through the request access gateway; at the same time, it writes the complete log of this request (request content summary, task type, routing model, inference time, and token consumption) into the time-series database of the operation and maintenance monitoring module.

[0142] This invention may further include an operation and maintenance monitoring module: real-time collection of operational metrics of each module of the system (request throughput, inference latency of each model, GPU utilization, video memory utilization, request queue depth, error rate, etc.), storage of monitoring data in a time-series database, and provision of a visual operation and maintenance dashboard; when an abnormal increase in model inference latency or GPU resource utilization is detected to exceed the alarm threshold, an operation and maintenance alarm is triggered.

[0143] In this invention, the GPU resource scheduling module can specifically function as follows in the above steps:

[0144] Monitor the real-time resource usage status of all GPU devices, including memory usage, GPU utilization, and temperature. Combine this with the configuration and status data of model instances in the model registry center and business load trends to execute elastic allocation strategies for GPU memory resources.

[0145] Step A1, Memory Pool Management:

[0146] The video memory of all available GPU devices is abstracted into a unified GPU video memory resource pool, and the total video memory capacity V_total(g), the allocated video memory capacity V_alloc(g), and the available video memory capacity V_free(g) of each GPU device are recorded.

[0147] In actual execution, the GPU memory resource pool is first initialized, all available GPU devices are enumerated, the total memory capacity of each GPU is obtained V_total(g), the allocated memory capacity V_alloc(g) = 0 is initialized, and the available memory capacity V_free(g) = V_total(g).

[0148] Based on the business priority configuration, determine the priority weight P(m) of each model; set the GPU memory reservation ratio R_reserve = 0.05, therefore, the actual allocable memory V_avail(g) = V_total(g) × (1 - R_reserve); in the subsequent model scheduling process, update the actual allocable memory in real time according to the GPU memory allocation.

[0149] Step A2, Elastic Scaling Strategy:

[0150] Let the average load rate of all model instances corresponding to the same model within the time window T be Load_avg(model, T);

[0151] If Load_avg(model, T_scale) > H_high and the current GPU memory resource pool has remaining space, horizontal scaling of model instances is triggered. A copy of the same model is launched on an idle GPU device and registered with the model registry center to achieve load balancing distribution of requests. Load_avg(model, T_scale) > H_high means that the average load rate of the model continuously exceeds the high watermark threshold H_high (default 0.80) for a time window of T_scale (default 60 seconds).

[0152] If Load_avg(m, T_shrink) < H_low and there are more than one active model instance for the same model, the model instance shrinking is triggered, the redundant model instances of the same model are closed, and after the existing requests are processed, all model instances are gracefully closed, releasing GPU memory for other models to use; where Load_avg(m, T_shrink) represents the time window during which the average load rate of the model is continuously lower than the low water level threshold H_low (default 0.15) to T_shrink (default 300 seconds);

[0153] When switching business load modes, the scaling up and down of model instances is automatically triggered by changes in load rate, enabling dynamic migration of GPU resources between different model instances;

[0154] Step A3, Hot loading and cold unloading of model instances:

[0155] When the GPU resource scheduling module receives a loading request for the target model instance m, it compares the available memory capacity V_free(g) of the current GPU memory resource pool with the memory usage V_required(m) required by the target model instance m. V_required(m) can be obtained from the model registry center or can be computed according to the model parameter size, quantization accuracy, and tensor parallel configuration.

[0156] If V_free(g) ≥ V_required(m), perform hot loading of model instances: traverse GPU devices to select the best GPU combination that satisfies V_free(g) ≥ V_required(m). The best GPU combination is the combination with the largest available video memory capacity to reduce fragmentation; load the model weight file of the target model instance m into the best GPU combination, and update the target model instance m in the model registry to the loaded state;

[0157] If V_free < V_required, meaning no single GPU or GPU combination meets the memory requirements, then the LRU cold unloading process is executed. Currently loaded model instances are sorted by least recently used time, and lower-priority, least recently used model instances are unloaded sequentially until V_free ≥ V_required. Then, hot loading of model instances is performed. In practical applications, a loading list of model instances sorted by most recently accessed time is maintained, and cold unloading is performed sequentially from the end of the list until the requirements are met or all unloadable model instances have been unloaded.

[0158] Specifically: When executing the LRU cold unloading strategy, a tiered unloading strategy is executed when the model instance is unloaded. The first level of unloading transfers the model weights from GPU memory to system memory, preserving the ability to quickly recover. When system memory is insufficient, the second level of unloading is executed, writing the model weights from memory back to disk, completely releasing resources.

[0159] As one example, hot loading and cold unloading of model instances can be summarized as follows:

[0160] Hot loading: When the dynamic routing module requests a model instance whose current state is "unloaded", the GPU resource scheduling module evaluates the available space in the current memory pool. If necessary, it first cold unloads the least recently used (LRU) low-priority model instance to release memory, and then loads the target model weight from the disk to the GPU memory.

[0161] Cold unloading: Transfer model weights from GPU memory to system memory (preserving the ability to quickly recover) or disk (completely releasing resources) and update the deployment status of the model registry.

[0162] Step A4, GPU memory fragmentation management:

[0163] Continuously monitor the GPU memory fragmentation metric Frag(g) = 1 - MaxContiguousFree(g) / V_free(g), where MaxContiguousFree(g) is the maximum value of continuous available space. When Frag(g) > 0.3, memory defragmentation is triggered, and a brief unload-reload operation is performed on low-load model instances to merge fragmented free memory blocks.

[0164] This invention also provides a private fusion scheduling system for multi-source heterogeneous large language models for railway intranets. The system is deployed in the railway intranet and includes:

[0165] The model registry center is used to maintain the configuration and status data of deployed model instances, including model instance metadata, deployment status, and load status. The model instance metadata includes model instance identifier, task proficiency type tag, and inference engine type. The deployment status includes loaded, unloaded, and loading status. The task proficiency type tag includes at least one task proficiency type.

[0166] The request access gateway is used to receive inference requests initiated by the railway intranet business system, perform request authentication, request rate limiting and request format standardization operations on the inference requests, obtain standardized inference requests with a unified inference request format, and pass them to the task classification module.

[0167] The task classification module is used to determine the task type of the standardized inference request, obtain a task type tag, attach the task type tag to the metadata of the standardized inference request, obtain a tagged inference request, and pass it to the dynamic routing module.

[0168] The dynamic routing module is used to read the configuration and status data of each deployed model instance in the model registry center, perform routing decisions, route the target model instance, and send the target model instance identifier and the tagged inference request to the inference engine management module.

[0169] The execution of routing decisions, routing out the target model instance, includes:

[0170] The decision determines whether there exists a model instance that is task type compatible and whose deployment status is either loaded or loading; the task type compatibility refers to the task type tag in the tagged inference request being located in the model instance's proficient task type tag.

[0171] If it exists, all model instances that meet the conditions will be used as candidate model instances. Taking into account the load status and historical average response time, the best candidate model instance will be selected as the target model instance.

[0172] If it does not exist, the GPU resource scheduling module is triggered to execute the model instance hot loading process, load the model weight file of the model instance that is suitable for the task type and is in the unloaded state into the GPU memory, and update the corresponding model instance in the model registry to the loaded state. At this time, the model instance becomes the target model instance.

[0173] The inference engine management module is used to find the model registry center based on the target model instance identifier, obtain the inference engine type that executes the target model instance, and then determine the corresponding target inference engine instance. The tagged inference request is added to the request queue of the target inference engine instance. When the tagged inference request is processed, the target inference engine instance performs forward inference on the tagged inference request based on the target model instance to obtain the inference result containing the generated text and inference meta-information, and sends it to the response aggregation module.

[0174] The response aggregation module is used to post-process the inference results, obtain a response message, and return it to the request access gateway, which then pushes the response message to the railway intranet business system that initiated the request.

[0175] Compared with the prior art, the beneficial effects of the present invention are:

[0176] 1) Completely resolves the conflict between railway data security and large-scale model capabilities. All large language models are privately deployed on the railway's intranet GPU servers. Inference requests and business data never leave the intranet, eliminating data leakage and compliance risks at the physical isolation level. The system supports the operation of purely domestic large models (Qwen2.5, DeepSeek R1, etc.), eliminating dependence on overseas models and public cloud APIs, and meeting the railway industry's requirements for domestic self-reliance and control.

[0177] 2) Multi-model collaboration covers all railway business needs. Through a task-aware dynamic routing mechanism, text-based question-and-answer requests are automatically routed to Qwen2.5-72B, logic reasoning requests to DeepSeek R1, and multi-modal requests involving text and images to Qwen2.5VL. On an evaluation set covering 800 railway business test requests, compared to using a fixed single model, the overall task accuracy of the dynamic routing scheme improved from 82.4% / 79.6% to 96.3%.

[0178] 3) Achieving high-concurrency and stable inference services with limited GPU resources. In a typical railway intranet hardware configuration with four NVIDIA A800-80GB servers (16 GPUs in total), the system can simultaneously maintain 2-3 online inference service instances for large models through a GPU memory elastic allocation mechanism. In actual testing under 50 concurrent request loads, the P95 response time for text requests was 1.05 seconds, and the P95 response time for multimodal requests was 1.18 seconds, both meeting the 1.2-second business SLA requirement. Compared to a fixed resource allocation scheme, the GPU resource utilization rate of this invention increased from 47.2% to 81.6%, and the maximum supported concurrency under the same hardware conditions increased from 28 to 56.

[0179] 4) Supports hot-swappable models to adapt to the continuous evolution of railway operations. The model registry supports dynamic registration / deregistration of model instances. When railway business departments introduce new large language models, they only need to add model metadata and deploy model weight files in the model registry, without modifying system code or restarting services to access the scheduling system.

[0180] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0181] Example 1: System Overall Architecture and Deployment Topology

[0182] like Figure 1 As shown, this invention discloses a private fusion scheduling system for multi-source heterogeneous large language models for railway intranets, which consists of a request access gateway 201, a task classification module 202, a model registration center 203, a dynamic routing module 204, an inference engine management module 205, a GPU resource scheduling module 206, a response aggregation module 207, and an operation and maintenance monitoring module 208.

[0183] The system is deployed in the railway bureau's intranet computer room. The hardware environment consists of four GPU servers, each equipped with four NVIDIA A800-80GB GPUs (16 in total), an Intel Xeon Platinum 8480+ CPU (56 cores), 512GB of DDR5 memory, and 4TB of NVMe SSD storage. All servers are interconnected via 100Gbps InfiniBand on the intranet and connected to the railway intranet business system via 10 Gigabit Ethernet.

[0184] Three large language model instances are registered in the model registry (203):

[0185] Model Example A: Qwen2.5-72B-Instruct (Plain Text Model), with a parameter size of 72B, is well-suited for tasks such as TextQA, TextSummary, and CodeGen. It is deployed on GPU:0-GPU:1, with a video memory usage of 138.5GB, and the inference engine is vLLM.

[0186] Model Example B: DeepSeek R1-70B (pure text inference model), parameter size 70B, proficient in tasks such as LogicReason and CodeGen, deployed on GPU:2-GPU:3, with a video memory usage of 136.8GB, and the inference engine is vLLM;

[0187] Model example C: Qwen2.5VL-72B-Instruct (Vision-Language Multimodal Model), parameter size 72B, best suited for MultiModal tasks, deployed on GPU:4-GPU:5, with a video memory usage of 142.3GB, and the inference engine is vLLM.

[0188] The remaining GPUs (6-15) are used as an elastic resource pool for horizontal model scaling and hot reloading.

[0189] Example 2: Complete Scheduling Process for Text Question Answering Tasks

[0190] Taking a passenger inquiry request initiated by the railway passenger service system as an example, the complete execution process is as follows:

[0191] 1) Request access (Request access gateway 201).

[0192] The passenger service system sends an HTTP POST request to the access gateway, with the request body as follows:

[0193] {

[0194] "request_id": "CSR-20260330-00158",

[0195] "input_text": "What time does the earliest G-series high-speed train depart from Beijing West to Zhengzhou East? Which stations does it stop at?"

[0196] "input_images": [],

[0197] "source_system": "passenger_service",

[0198] "priority": "normal",

[0199] "max_tokens": 512

[0200] }

[0201] The gateway verifies the validity of the token, the token bucket checks the current concurrency level and finds it to be 23, and the request is approved.

[0202] 2) Task classification (task classification module 202).

[0203] The task classification module detects that `input_images` is empty, extracts TF-IDF features from `input_text`, and then feeds them into an SVM classifier. The output probability distribution is: TextQA=0.89, TextSummary=0.05, LogicReason=0.03, CodeGen=0.03. The task type is determined to be TextQA.

[0204] 3) Dynamic routing (Dynamic routing module 204).

[0205] Read model registry 203 and filter for candidate model instances that are proficient in TextQA and already loaded: S_candidate={Qwen2.5-72B}. If the candidate set contains only one model instance, route directly to Qwen2.5-72B.

[0206] 4) Model reasoning (reasoning engine management module 205).

[0207] The inference engine management module 205 will request to add the request to the vLLM inference queue of Qwen2.5-72B. vLLM allocates KV-Cache based on PagedAttention and performs autoregressive generation. SSE streaming output is enabled, the first token latency is 128ms, a complete answer of 187 tokens is generated, and the total inference time is 762ms.

[0208] 5) Response feedback (response aggregation module 207).

[0209] The response aggregation module 207 performs sensitive information scanning on the output text, encapsulates the response message, and sends it back to the passenger service system via the gateway.

[0210] Example 3: Scheduling and Hot Loading of Multimodal Image and Text Understanding Tasks

[0211] Taking a handwritten material receiving slip photo sent by the material management system for recognition and understanding as an example:

[0212] 1) Request access and task classification:

[0213] The input_images in the request body contains Base64 encoded data of an inbound order photo. When the task classification module 202 detects the image input, it directly determines that the task type is MultiModal.

[0214] 2) Dynamic routing:

[0215] Filter candidate model instances: S_candidate = {Qwen2.5VL-72B}. Assuming that Qwen2.5VL has been scaled down and unloaded due to low load overnight, and deploy_status = "unloaded_to_ram", then hot reloading needs to be triggered.

[0216] 3) Hot loading of models (GPU resource scheduling module 206).

[0217] The GPU resource scheduling module 206 received a hot reloading request. The target model, Qwen2.5VL, requires 142.3GB of video memory. The GPU memory pool was checked: GPU:4-GPU:5 were currently free, totaling 160GB, which is greater than the required video memory. Since the model weights were already in system memory, a hot reload was performed, transferring the weights from CPU RAM to the GPU via the PCIe 5.0 channel, taking approximately 18 seconds. The vLLM inference engine instance was started, and the model registry 203 status was updated to "loaded".

[0218] 4) Model Inference:

[0219] Qwen2.5VL receives images and text prompts, the visual encoder extracts features from the images, and the language model generates a structured understanding of the contents of the input form. The total inference time is 1.08 seconds.

[0220] Example 4: Elastic scaling in high-concurrency scenarios

[0221] During peak railway transport periods, the volume of text-based question-and-answer requests in the passenger service system surged. The load rate of the Qwen2.5-72B instance climbed from the normal 0.35 to 0.85, exceeding the high watermark threshold H_high=0.80 and remaining there for over 60 seconds. The GPU resource scheduling module 206 triggered horizontal scaling: it checked the current idle state of the elastic resource pool GPU:6-GPU:7, finding 160GB of available video memory, which was greater than the video memory required by Qwen2.5-72B; it launched a second vLLM instance of Qwen2.5-72B on GPU:6-GPU:7; it registered a new instance qwen25-72b-inst-02 in the model registry; the dynamic routing module detected the two Qwen2.5-72B instances and achieved load balancing using the Load(m) factor when calculating the routing score.

[0222] After the expansion, the P95 response time for 50 concurrent requests decreased from 1.35 seconds to 0.92 seconds, returning to within the SLA requirements.

[0223] When the peak ended and the request volume dropped, the load rate of both instances dropped below 0.10 and remained below 0.10 for more than 300 seconds. The GPU resource scheduling module 206 triggered scaling down, gracefully shutting down the second instance and releasing the video memory of GPU:6-GPU:7 back to the elastic resource pool.

[0224] Example 5: Specialized routing for logical reasoning tasks:

[0225] The dispatching and command system sends a transportation plan optimization request: "Please calculate the optimal stopping scheme for train G1234 on the Zhengzhou East to Wuhan section based on the following timetable constraints..."

[0226] The task classification module 202 detected logical reasoning keywords such as "calculation," "optimal," and "constraints," and the SVM classifier output a LogicReason probability of 0.92. The dynamic routing module 204 filtered the candidate set; DeepSeek R1's specialty contains LogicReason, while Qwen2.5-72B does not.

[0227] Route score calculation: DeepSeek R1: Score = 0.5×1 + 0.3×(1-0.50) + 0.2×(1 / 0.95) = 0.861

[0228] Qwen2.5-72B: Score = 0.5×0 + 0.3×(1-0.35) + 0.2×(1 / 0.85) = 0.430

[0229] DeepSeek R1 scored higher, so the request was routed to DeepSeek R1. DeepSeek R1 leveraged its reinforcement learning-enhanced inference capabilities to generate a stop plan analysis that included a step-by-step derivation process, with an inference time of 1.15 seconds.

[0230] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A private fusion scheduling method for multi-source heterogeneous large language models in railway intranets, characterized in that, include: Step S1: The model registry maintains the configuration and status data of deployed model instances, including model instance metadata, deployment status, and load status; the model instance metadata includes model instance identifier, competent task type tag, and inference engine type; the deployment status includes loaded status, unloaded status, and loading status; the competent task type tag includes at least one competent task type. Step S2: When the request access gateway receives an inference request initiated by the railway intranet business system, it performs request authentication, request rate limiting, and request format standardization operations on the inference request to obtain a standardized inference request with a unified inference request format, and then passes it to the task classification module. Step S3: The task classification module determines the task type of the standardized inference request, obtains a task type tag, appends the task type tag to the metadata of the standardized inference request, obtains a tagged inference request, and passes it to the dynamic routing module. Step S4: The dynamic routing module reads the configuration and status data of each deployed model instance in the model registry center, executes the task-aware model routing strategy, routes out the target model instance, and sends the target model instance identifier and the tagged inference request to the inference engine management module. The execution task-aware model routing strategy, which routes out the target model instance, includes: The decision determines whether there exists a model instance that is task type compatible and whose deployment status is either loaded or loading; the task type compatibility refers to the task type tag in the tagged inference request being located in the model instance's proficient task type tag. If it exists, all model instances that meet the conditions will be used as candidate model instances. Taking into account the load status and historical average response time, the best candidate model instance will be selected as the target model instance. If it does not exist, the GPU resource scheduling module is triggered to execute the model instance hot loading process, load the model weight file of the model instance that is suitable for the task type and is in the unloaded state into the GPU memory, and update the corresponding model instance in the model registry to the loaded state. At this time, the model instance becomes the target model instance. Step S5: The inference engine management module searches the model registry center based on the target model instance identifier to obtain the inference engine type that executes the target model instance, and then determines the corresponding target inference engine instance, and adds the tagged inference request to the request queue of the target inference engine instance. When the labeled reasoning request is processed, the target reasoning engine instance performs forward reasoning on the labeled reasoning request based on the target model instance to obtain a reasoning result containing the generated text and reasoning meta-information, and sends it to the response aggregation module. In step S6, the response aggregation module performs post-processing on the inference result to obtain a response message and returns it to the request access gateway, which then pushes the response message to the railway intranet business system that initiated the request.

2. The method for private fusion scheduling of multi-source heterogeneous large language models for railway intranets according to claim 1, characterized in that, The configuration and status data of the model instance also include: the GPU device number currently occupied by the model instance and the amount of video memory required for operation, the current number of concurrent requests of the model instance, the historical average response time, the health check status, and the model name, model type, and model parameter scale of the large language model corresponding to the model instance.

3. The method for private fusion scheduling of multi-source heterogeneous large language models for railway intranets according to claim 2, characterized in that, In the deployment state, the loaded state means that the model weight file of the model instance has been loaded into the GPU memory; the unloaded state includes two types: one is that the model weight file has been unloaded from the GPU memory to the system memory, and the other is that the model weight file has been unloaded from the system memory to the disk; the loading state means that the model weight file is being loaded from the disk or the system memory into the GPU memory. The health check status of the model instance includes a healthy status and an unhealthy status; The model types include plain text models and multimodal graph-text models; the plain text models are used to handle plain text tasks that do not contain images, including one or more of TextQA, TextSummary, LogicReason, and CodeGen; the multimodal graph-text models are used to handle tasks that contain images. The inference engine type refers to the type of inference engine that runs the model. The same type of inference engine corresponds to one or more models.

4. The method for privatized fusion scheduling of multi-source heterogeneous large language models for railway intranets according to claim 1, characterized in that, Perform request authentication, request rate limiting, and request format standardization operations on the inference request to obtain a standardized inference request with a unified inference request format, including: The inference request is authenticated, specifically by token verification based on the railway intranet unified identity authentication platform. The inference request is subject to request rate limiting, specifically by using a token bucket algorithm to control the upper limit of concurrent requests; The request format standardization operation is performed on the inference request, specifically converting heterogeneous inference requests from different business systems into a unified request operation that conforms to the inference request protocol. The obtained standardized inference request with a unified inference request format has the following fields: request unique identifier request_id; input text input_text; input image list input_images, which can be empty; source business system identifier source_system; request priority and maximum number of output tokens max_tokens.

5. The method for private fusion scheduling of multi-source heterogeneous large language models for railway intranets according to claim 4, characterized in that, The task classification module determines the task type of the standardized inference request and obtains a task type label, including: Step S31: Determine whether the standardized inference request contains image input: If the input image list input_images is not empty, it indicates that the request contains image input, and proceed to step S32; if the input image list input_images is empty, it indicates that the request does not contain image input, and proceed to step S33. Step S32: Determine if the request includes image input, and directly determine the task type as MultiModal (Multi-Modal). Step S33: If the request does not contain image input, then perform lightweight text classification on the input text input_text. The method is as follows: extract the TF-IDF feature vector of the input text input_text and feed it into a pre-trained SVM classifier; the SVM classifier outputs the probability distribution of each task type, and takes the category with the highest probability as the determined task type; where each task type includes one or more of the following: TextQA, TextSummary, LogicReason, and CodeGen.

6. The method for private fusion scheduling of multi-source heterogeneous large language models for railway intranets according to claim 1, characterized in that, The step of selecting all model instances that meet the conditions as candidate model instances, and comprehensively considering the load status and historical average response time, to determine the best candidate model instance as the target model instance includes: The following formula is used to calculate the routing score for each candidate model instance, and the candidate model instance with the highest score is selected as the target model instance: Score(m) = α × TaskMatch(m) + β × (1 - Load(m)) + γ × (1 / AvgLatency(m)) Where: Score(m) represents the routing score of candidate model instance m; TaskMatch(m)∈ {0, 1} represents the matching degree between the candidate model instance m and the current task type. If the current task type label is in the good task type label of the candidate model instance m, it is 1; otherwise, it is 0. Load(m) is the current load rate of candidate model instance m, with a value range of [0, 1]. Load(m) = current_concurrency(m) / max_concurrency(m), where current_concurrency(m) is the current number of concurrent requests of candidate model instance m, and max_concurrency(m) is the maximum number of concurrent requests supported by candidate model instance m. AvgLatency(m) is the historical average response time of candidate model instance m; α, β, and γ are configurable weighting coefficients, with the default values ​​being α=0.5, β=0.3, and γ=0.

2.

7. The method for private fusion scheduling of multi-source heterogeneous large language models for railway intranets according to claim 1, characterized in that, When the target inference engine instance performs forward inference, for models with streaming output, the inference results are transmitted back token by token via the SSE protocol.

8. The method for private fusion scheduling of multi-source heterogeneous large language models for railway intranets according to claim 1, characterized in that, The GPU resource scheduling module is specifically used for: Monitor the real-time resource usage status of all GPU devices, including memory usage, GPU utilization, and temperature. Combine this with the configuration and status data of model instances in the model registry center and business load trends to execute elastic allocation strategies for GPU memory resources. Step A1, Memory Pool Management: The video memory of all available GPU devices is abstracted into a unified GPU video memory resource pool, and the total video memory capacity V_total(g), the allocated video memory capacity V_alloc(g), and the available video memory capacity V_free(g) of each GPU device are recorded. Step A2, Elastic Scaling Strategy: Let the average load rate of all model instances corresponding to the same model within the time window T be Load_avg(model, T); If Load_avg(model, T_scale) > H_high and the current GPU memory resource pool has remaining space, horizontal scaling of model instances is triggered. A copy of the same model is launched on an idle GPU device and registered with the model registry center to achieve load balancing distribution of requests. Here, Load_avg(model, T_scale) > H_high means that the average load rate of the model continuously exceeds the high watermark threshold H_high for the time window of T_scale. If Load_avg(m, T_shrink) < H_low and there are more than one active model instance for the same model, the model instance is shrunk, the redundant model instances of the same model are closed, and all model instances are gracefully closed after the existing requests are processed, releasing GPU memory; where Load_avg(m, T_shrink) represents the time window during which the average load rate of the model is continuously lower than the low water level threshold H_low. When switching business load modes, the scaling up and down of model instances is automatically triggered by changes in load rate, enabling dynamic migration of GPU resources between different model instances; Step A3, Hot loading and cold unloading of model instances: When the GPU resource scheduling module receives a loading request for the target model instance m, it compares the available memory capacity V_free(g) of the current GPU memory resource pool with the memory usage V_required(m) required by the target model instance m. If V_free(g) ≥ V_required(m), perform hot loading of model instances: traverse GPU devices to select the best GPU combination that satisfies V_free(g) ≥ V_required(m). The best GPU combination is the combination with the largest available video memory capacity to reduce fragmentation; load the model weight file of the target model instance m into the best GPU combination, and update the target model instance m in the model registry to the loaded state; If V_free < V_required, the LRU cold unloading strategy is executed. The currently loaded model instances are sorted by the least recently used time, and the low-priority and least recently used model instances are unloaded in turn until V_free ≥ V_required, and then the model instances are hot-loaded. Specifically: When executing the LRU cold unloading strategy, a tiered unloading strategy is executed when the model instance is unloaded. The first level of unloading transfers the model weights from GPU memory to system memory, preserving the ability for rapid hot recovery. When system memory is insufficient, the second level of unloading is executed, writing the model weights from memory back to disk, completely releasing resources. Step A4, GPU memory fragmentation management: Continuously monitor the GPU memory fragmentation metric Frag(g) = 1 - MaxContiguousFree(g) / V_free(g), where MaxContiguousFree(g) is the maximum value of continuous available space. When Frag(g) > 0.3, memory defragmentation is triggered, and a brief unload-reload operation is performed on low-load model instances to merge fragmented free memory blocks.

9. A private fusion scheduling system for multi-source heterogeneous large language models oriented towards railway intranets, characterized in that, The system is deployed within the railway intranet and includes: The model registry center is used to maintain the configuration and status data of deployed model instances, including model instance metadata, deployment status, and load status. The model instance metadata includes model instance identifier, task proficiency type tag, and inference engine type. The deployment status includes loaded, unloaded, and loading status. The task proficiency type tag includes at least one task proficiency type. The request access gateway is used to receive inference requests initiated by the railway intranet business system, perform request authentication, request rate limiting and request format standardization operations on the inference requests, obtain standardized inference requests with a unified inference request format, and pass them to the task classification module. The task classification module is used to determine the task type of the standardized inference request, obtain a task type tag, attach the task type tag to the metadata of the standardized inference request, obtain a tagged inference request, and pass it to the dynamic routing module. The dynamic routing module is used to read the configuration and status data of each deployed model instance in the model registry center, perform routing decisions, route the target model instance, and send the target model instance identifier and the tagged inference request to the inference engine management module. The execution of routing decisions, routing out the target model instance, includes: The decision determines whether there exists a model instance that is task type compatible and whose deployment status is either loaded or loading; the task type compatibility refers to the task type tag in the tagged inference request being located in the model instance's proficient task type tag. If it exists, all model instances that meet the conditions will be used as candidate model instances. Taking into account the load status and historical average response time, the best candidate model instance will be selected as the target model instance. If it does not exist, the GPU resource scheduling module is triggered to execute the model instance hot loading process, load the model weight file of the model instance that is suitable for the task type and is in the unloaded state into the GPU memory, and update the corresponding model instance in the model registry to the loaded state. At this time, the model instance becomes the target model instance. The inference engine management module is used to find the model registry center based on the target model instance identifier, obtain the inference engine type that executes the target model instance, and then determine the corresponding target inference engine instance. The tagged inference request is added to the request queue of the target inference engine instance. When the tagged inference request is processed, the target inference engine instance performs forward inference on the tagged inference request based on the target model instance to obtain the inference result containing the generated text and inference meta-information, and sends it to the response aggregation module. The response aggregation module is used to post-process the inference results, obtain a response message, and return it to the request access gateway, which then pushes the response message to the railway intranet business system that initiated the request.

Images