Traffic hub vertical field large model training and lightweight deployment method and system

Abstract

The application provides a traffic hub vertical field large model training and light deployment method and system, relates to the technical field of machine learning, and comprises the following steps: collecting traffic hub multi-scene service corpus, marking static knowledge, semi-static knowledge, periodic knowledge and dynamic knowledge into four categories according to knowledge timeliness, and constructing a classification data set; performing discrete cosine transform mapping on the weight increment of a pre-trained large language model to a frequency domain space, dividing four frequency band intervals according to the timeliness category, retaining sparse coefficients in each frequency band as trainable parameters, and adopting a differentiated learning rate to fine-tune a teacher model; transferring the teacher model capability to a light student model through progressive three-stage distillation; deploying the student model to a mobile terminal, establishing an end-cloud collaborative mechanism, and updating parameters according to a frequency band level strategy; and responding to user dialogue requests based on the deployed model. The application drives frequency domain sparse distribution according to knowledge timeliness, realizes ultra-low parameter fine-tuning and mobile terminal light deployment.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a method and system for training and lightweight deployment of large models in the vertical domain of transportation hubs. Background Technology

[0002] With the rapid development of large language model technology, adapting general-purpose large language models to vertical industry sectors has become an important direction for artificial intelligence applications. Transportation hubs, as core nodes of urban transportation networks, handle a large number of passenger travel service demands daily, encompassing various scenarios such as station navigation, train schedule inquiries, transfer guidance, and emergency response. Utilizing large language models to build intelligent service systems for transportation hubs has significant application value.

[0003] In the field of efficient parameter fine-tuning for large language models, low-rank adaptive methods are currently widely used. Patent CN121480595A discloses a dynamic routing parameter efficient fine-tuning method based on a combination of low-rank adaptive and hybrid expert mechanisms. This method introduces a hybrid expert-gated routing mechanism on the low-rank adapter, dynamically selecting different expert modules for parameter updates based on input features. However, this method relies on low-rank matrix factorization as its basic representation, and its representational capability is limited by the size of the rank parameter. Furthermore, the hybrid expert routing mechanism introduces additional computational overhead for the gating network and load balancing constraints.

[0004] Chinese patent application CN121436058A discloses an ultra-low parameter high-efficiency fine-tuning method based on sparse frequency domain projection. This method maps weight increments to the frequency domain using discrete cosine transform, retaining only a small number of sparse coefficients as trainable parameters. However, the frequency selection strategy of this method is based on a general mathematical priority criterion and does not consider the differentiated needs of different types of knowledge in specific application domains. A uniform frequency allocation strategy cannot fully utilize the inherent structural characteristics of domain knowledge.

[0005] In the field of knowledge distillation for large language models, invention patent CN121303257A discloses a progressive domain fine-tuning and knowledge fusion method for engineering, which achieves stepwise transfer of domain knowledge through hierarchical training courses and parallel training with multiple adapters. However, this method does not design differentiated distillation strategies for knowledge with different timeliness and lacks a continuous update mechanism for edge-cloud collaboration, making it difficult to meet the needs of frequent dynamic knowledge updates in transportation hub scenarios.

[0006] In summary, the existing technologies have the following shortcomings: First, the efficient parameter fine-tuning method does not associate the timeliness of domain knowledge with the model parameter update mechanism in its design; second, the model compression and distillation process lacks a progressive capability transfer strategy for the transportation hub domain; third, the granularity of model updates in edge-cloud collaborative deployment is too coarse, and selective parameter transmission and updates are not implemented based on the timeliness of knowledge. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention proposes a method and system for training and lightweight deployment of large-scale models in the vertical domain of transportation hubs. It uses the timeliness of domain knowledge to drive the allocation and updating of sparse parameters in the frequency domain, enabling ultra-low parameter fine-tuning, progressive domain capability distillation, and frequency band-level differentiated edge-cloud collaborative updates for transportation hub service scenarios.

[0008] To solve the above problems, the present invention adopts the following technical solution: In a first aspect, this invention proposes a method for training and lightweight deployment of a large-scale model in the vertical domain of transportation hubs, the method comprising the following steps: S1: Collect service corpora from multiple scenarios in transportation hubs, and label them into four categories according to the timeliness of knowledge: static knowledge, semi-static knowledge, periodic knowledge, and dynamic knowledge, and construct a classification dataset; S2: Define weight increments for the weight matrix of the pre-trained large language model, apply discrete cosine transform to the weight increments to map them to the frequency domain space, divide the frequency domain coefficient matrix into four frequency band intervals according to the four categories, retain a preset number of frequency domain coefficients in each frequency band interval as trainable parameters, and fine-tune the training according to the differential learning rate to obtain the teacher model. S3: Construct a student model and transfer the domain capabilities of the teacher model to the student model through a progressive three-stage distillation process to obtain a lightweight student model; S4: Deploy the lightweight student model to mobile terminal devices, deploy the teacher model to the cloud server, establish an end-to-cloud collaboration mechanism, and update the model parameters on the mobile terminal devices according to the frequency band-level differentiated strategy; S5: Receives user dialogue requests, calls the lightweight student model to generate response text, and returns it to the user.

[0009] A second aspect of this invention provides a large-scale model training and lightweight deployment system for a transportation hub vertical domain, comprising: a processor and a memory; The memory stores programs or instructions that can run on the processor, which, when executed by the processor, implement the steps of the large model training and lightweight deployment method for the transportation hub vertical domain as described in the first aspect.

[0010] A third aspect of the present invention provides a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the large-scale model training and lightweight deployment method for the transportation hub vertical domain as described in the first aspect.

[0011] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: 1. This invention couples the timeliness characteristics of knowledge in the transportation hub domain with the frequency domain sparse parameter allocation, mapping static knowledge, semi-static knowledge, periodic knowledge and dynamic knowledge to different frequency bands, so that the update granularity of model parameters matches the update frequency of knowledge, achieving differentiated learning efficiency for different timeliness knowledge while maintaining an extremely low number of trainable parameters.

[0012] 2. This invention designs a progressive three-stage distillation strategy, which transfers the domain capabilities of the teacher model step by step from general knowledge of transportation to specific knowledge of hub services, and prevents the knowledge learned in previous stages from being forgotten in subsequent stages by capability retention regularization terms.

[0013] 3. This invention introduces a frequency band-level differentiated update strategy in edge-cloud collaborative deployment. Based on the timeliness of knowledge, only the frequency domain coefficients of high-frequency and mid-frequency bands are transmitted and updated, which significantly reduces the amount of data transmitted per update and the computing overhead of mobile terminals. Attached Figure Description

[0014] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0015] Figure 1 This is a flowchart illustrating a method for training and lightweight deployment of a large model in the vertical domain of a transportation hub, as provided in an embodiment of the present invention.

[0016] Figure 2 This is an interactive diagram illustrating the end-to-cloud collaboration mechanism provided in an embodiment of the present invention.

[0017] Figure 3 This is a schematic diagram of a large-scale model training and lightweight deployment system for a transportation hub vertical domain, provided as an embodiment of the present invention. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0019] It should be noted that the technical terms used in this invention are only for the purpose of describing specific embodiments and are not intended to limit the scope of protection of this invention. Unless otherwise specified, all instruments and equipment used in the following embodiments of this invention can be purchased from the market or prepared by existing methods.

[0020] The following description, in conjunction with the accompanying drawings, details the method for training and lightweight deployment of large-scale models in the vertical domain of transportation hubs provided by the present invention through specific embodiments and application scenarios.

[0021] Reference manual attached Figure 1 The diagram illustrates a flowchart of a method for training and lightweight deployment of a large model in the vertical domain of a transportation hub, provided by an embodiment of the present invention.

[0022] This invention provides a method for training and lightweight deployment of a large model in the vertical domain of transportation hubs, which may include the following steps: S1: Collect service corpora from multiple scenarios in transportation hubs, and label them into four categories according to the timeliness of knowledge: static knowledge, semi-static knowledge, periodic knowledge, and dynamic knowledge, and construct a classification dataset.

[0023] In one possible implementation, multi-scenario service data of the target transportation hub is collected. The sources of the service data include: (1) Passenger inquiry and answer records, which are collected from the station customer service center and intelligent inquiry terminals for the past three years. (2) Station service manuals, which include standardized documents such as "Passenger Service Guide", "Station Facility Description" and "Emergency Response Plan". (3) Train timetable database, which is accessed through the railway operation management system to obtain timetables, fares and remaining ticket information of trains stopping there. (4) Real-time operation announcements, including dynamic data sources such as train delay notices, temporary suspension notices and weather warning information.

[0024] Specifically, the service corpus covers six service scenarios: station navigation, train schedule inquiry, transfer guidance, security check instructions, baggage regulations, and emergency response. After data cleaning, deduplication, and quality screening, valid question-and-answer pairs were retained.

[0025] In one possible implementation, the service corpus is labeled into four categories based on the timeliness of the knowledge.

[0026] Specifically, static knowledge refers to space and facility information updated at a frequency ranging from monthly to grade level, including station layout information, exit location information, and functional area distribution information.

[0027] Specifically, semi-static knowledge refers to rules and procedures updated weekly to monthly, including security regulations, baggage restrictions, and service hours.

[0028] Specifically, periodic knowledge refers to operational information updated daily to quarterly, including train timetables, schedules, and transfer plans.

[0029] Specifically, dynamic knowledge refers to real-time status information updated at a frequency ranging from minutes to hours, including train delay information, weather warning information, and passenger flow status information.

[0030] S2: Define weight increments for the weight matrix of the pre-trained large language model, apply discrete cosine transform to the weight increments to map them to the frequency domain space, divide the frequency domain coefficient matrix into four frequency band intervals according to the four categories, retain a preset number of frequency domain coefficients in each frequency band interval as trainable parameters, and fine-tune the training according to the differential learning rate to obtain the teacher model.

[0031] In one possible implementation, weight increments are defined for the weight matrices of each Transformer layer in the pre-trained large language model. Each Transformer layer contains the query matrix used in the attention mechanism. Key matrix Value matrix Output matrix Four weight matrices, and the gating matrix in the feedforward network. Upper projection matrix Lower projection matrix Three weight matrices.

[0032] in, , , , Let represent the weight matrices for the query, key, value, and output linear transformation of the attention mechanism, respectively. , , These represent the weight matrices for the MLP gate, up projection, and down projection of the feedforward network, respectively.

[0033] In one possible implementation, the weight increment of the weight matrix is ​​represented as an increment matrix of the same dimension as the weight matrix. After training, the weight matrix is ​​used to obtain the fine-tuned weight values ​​using the following formula:

[0034] in, W This represents the original weight matrix that is frozen. W' This represents the fine-tuned weight matrix. Let represent the weight increment matrix, and W , W' , All three have the same dimensions.

[0035] In one possible implementation, the increment matrix Applying a two-dimensional discrete cosine transform yields the frequency domain coefficient matrix. F .

[0036] In this embodiment of the invention, the frequency characteristics of frequency domain coefficients are correlated with the timeliness characteristics of knowledge in the transportation hub domain: static knowledge changes slowly and is suitable for encoding with low-frequency coefficients. Dynamic knowledge changes frequently and is suitable for encoding with high-frequency coefficients.

[0037] In one possible implementation, the frequency domain coefficient matrix F The frequency range is based on the frequency coordinates. L The 1-norm divides the frequency range into four frequency bands. L The 1-norm is defined as:

[0038] in, u and v These represent the two components of the frequency coordinate. Represents the corresponding frequency coordinates L 1-norm value.

[0039] The specific division method is as follows: the frequency coordinates satisfy... u+v <K Region 1 is divided into the low-frequency band, corresponding to static knowledge. The frequency coordinates satisfy... K 1 ≤u+v <K Region 2 is divided into the mid-to-low frequency band, corresponding to semi-static knowledge. The frequency coordinates satisfy... K 2 ≤u+ v <K Region 3 is divided into the mid-frequency band, corresponding to periodic knowledge. The frequency coordinates satisfy... u+v≥K Region 3 is divided into high-frequency bands, corresponding to dynamic knowledge. Among them... K 1 、K 2 、K 3 represents the preset frequency band boundary parameters and satisfies K 1 <K 2 <K 3 (Those skilled in the art can set the size of the frequency band boundary parameters according to the actual application scenario and knowledge distribution ratio, but this invention does not limit it).

[0040] In one possible implementation, within each frequency band interval, a preset number of frequency domain coefficients with the largest amplitude are retained as trainable parameters, while the remaining frequency domain coefficients are fixed to zero. Assume the low-frequency band retains... N One coefficient, mid-to-low frequency band retained. N Two coefficients, mid-frequency band retained. N Three coefficients, high-frequency bands retained N Four coefficients. Among them, N 1, N 2, N 3, N 4 represents the number of frequency domain coefficients retained as trainable parameters in the low frequency band, mid-low frequency band, mid frequency band, and high frequency band, respectively (those skilled in the art can set the size of the number of coefficients retained in each frequency band according to the actual application scenario and model architecture, but this invention does not limit it).

[0041] In one possible implementation, a differentiated learning rate associated with the timeliness of knowledge is set for each frequency band interval. Let the base learning rate be... The learning rate increment factor is and The learning rates for the four frequency bands are as follows:

[0042] in, , , , These represent the learning rates for low-frequency, mid-low-frequency, mid-frequency, and high-frequency bands, respectively. This represents the base learning rate corresponding to the low-frequency band. This represents the learning rate increment factor. and They represent The second and third powers.

[0043] In one possible implementation, the classification dataset is input into a pre-trained large language model for fine-tuning. The forward propagation step of the training process is as follows: the frequency domain coefficients preserved in each frequency band interval are restored to the parameter space through a two-dimensional inverse discrete cosine transform.

[0044] S3: Construct a student model and transfer the domain capabilities of the teacher model to the student model through a progressive three-stage distillation process to obtain a lightweight student model.

[0045] In one possible implementation, the domain capabilities of the teacher model are progressively transferred to the student model through a three-stage domain distillation process. The three stages are: a general knowledge distillation stage for transportation, a hub service-specific knowledge distillation stage, and a capability calibration stage.

[0046] In one possible implementation, the first stage is a distillation stage of general knowledge in the transportation domain. The student model is trained using a general corpus of transportation knowledge. The first stage uses inverse KL divergence as the distillation loss, calculated as follows:

[0047] in, This represents the loss from reverse KL divergence distillation. x Represents the token. The teacher model represents the lexical units. x The output probability distribution, Representing the student model for lexical units x The output probability distribution, This represents the natural logarithm function.

[0048] In this embodiment of the invention, the inverse KL divergence is based on the output probability distribution of the teacher model. As a weighted benchmark, greater learning weights are applied to the student model in regions where the teacher model has higher confidence, prompting the student model to prioritize learning the knowledge from the deterministic output of the teacher model.

[0049] In one possible implementation, the second stage is a hub-service-specific knowledge distillation stage. The student model is trained using a transportation hub-specific corpus. The second stage introduces a capability-preserving regularization term based on the inverse KL divergence distillation loss, calculated as follows:

[0050] in, This indicates the ability to maintain regularity. x Indicates word elements, This indicates the student model at the current training step. t for lexical elements x The output probability distribution, This indicates that the student model has reached the lexical level at the end of the first stage. x The output probability distribution, This represents the natural logarithm function.

[0051] In this embodiment of the invention, the capability preservation regularization term constrains the output distribution of the student model to not deviate from the distribution at the end of the first stage, thereby preventing the student model from forgetting the general traffic knowledge it has already learned when learning hub-specific knowledge.

[0052] In one possible implementation, the joint loss function used in the second and third stages consists of three weighted summation terms: a distillation loss term, a capability preservation regularization term, and a frequency domain approximation loss term. The calculation formula is as follows:

[0053] in, Denotes the joint loss function. This represents the loss from reverse KL divergence distillation. This indicates the ability to maintain regularity. Represents the frequency domain approximation loss term. and This represents the preset weighting coefficients, used to adjust them individually. and The relative contribution in the joint loss (those skilled in the art can set the weighting coefficient according to the actual application scenario, but this invention does not limit it).

[0054] In one possible implementation, the third stage is the capability calibration stage. Capability benchmark test sets are constructed for each of the four categories. The question-answering accuracy of the student model on each test subset is evaluated, and compensation fine-tuning is performed for categories whose accuracy does not meet a preset calibration threshold. After completing these three stages, a lightweight student model is obtained.

[0055] S4: Deploy the lightweight student model to mobile terminal devices and the teacher model to the cloud server, establish an end-to-cloud collaboration mechanism, and update the model parameters on the mobile terminal devices according to the frequency band-level differentiated strategy.

[0056] Reference manual attached Figure 2 The diagram illustrates the interaction of the end-to-cloud collaboration mechanism provided in this embodiment of the invention.

[0057] In one possible implementation, a lightweight student model is deployed to a mobile terminal device, while a teacher model is deployed to a cloud server. The mobile terminal device runs the lightweight student model to provide real-time passenger question-and-answer services, while the cloud server runs the teacher model for complex query processing and incremental model updates.

[0058] In one possible implementation, the mobile terminal device calculates the average log probability at the token level as a confidence score for the response text output by the lightweight student model, using the following formula:

[0059] in, Indicates the query sample q Confidence score of the corresponding response text. T This indicates the length of the token sequence in the response text. The first part of the response text i One token, i Indicates the token index. Indicates the first i The sequence of preceding tokens before this token. This indicates generating the first token given the preceding token sequence.i The conditional probability of each token This represents the natural logarithm function.

[0060] In one possible implementation, the mobile terminal device periodically flags users whose confidence scores fall below a preset threshold. The query samples are uploaded to the cloud server (those skilled in the art can set the size of the preset threshold according to the actual service quality requirements, but this invention does not limit it).

[0061] In one possible implementation, the cloud server performs incremental fine-tuning on the teacher model based on uploaded query samples. After the incremental fine-tuning is completed, update parameters are generated, which are the changes in frequency domain coefficients within each frequency band interval.

[0062] In one possible implementation, update parameters are sent to the terminal device according to a frequency band-level differentiation strategy. The frequency band-level differentiation strategy is as follows: only the frequency domain coefficient update amounts corresponding to the high frequency band and the mid frequency band are sent, while the frequency domain coefficients corresponding to the low frequency band and the mid-low frequency band remain unchanged.

[0063] In this embodiment of the invention, the frequency domain coefficients corresponding to dynamic knowledge in the high-frequency band need to be updated frequently. The frequency domain coefficients corresponding to periodic knowledge in the mid-frequency band need to be updated periodically. The static and semi-static knowledge corresponding to the low-frequency and mid-low-frequency bands are updated less frequently and do not need to be transmitted in each update, thereby significantly reducing the amount of data transmission.

[0064] S5: Receives user dialogue requests, calls the lightweight student model to generate response text, and returns it to the user.

[0065] In one possible implementation, the mobile terminal device receives a query request sent by the user through a dialog interface, inputs the query request into a lightweight student model, generates the corresponding response text, and returns it to the user.

[0066] In this embodiment of the invention, the dialogue requests cover a variety of scenarios such as in-station navigation queries, train information queries, transfer plan inquiries, security check regulation inquiries, and emergency service requests. The lightweight student model provides accurate and timely response services for different types of query requests based on a frequency domain coding structure driven by knowledge timeliness.

[0067] As a preferred embodiment of the present invention, the following embodiments aim to illustrate the specific implementation and experimental evaluation results of the present invention in the intelligent passenger service scenario of transportation hubs, so as to help those skilled in the art to more deeply understand and apply the technical solution. It should be emphasized that the data sources, model parameters and index results given in this embodiment are not the only limitations of the present invention, and the relevant details can be flexibly adjusted according to actual needs.

[0068] 1. Dataset and Experiment Setup (1) Data Source: This embodiment uses a passenger intelligent service system of a large high-speed rail hub as the application scenario. The training data used comes from 127,000 original records of historical question and answer data collected from the customer service center and intelligent inquiry terminal of the hub over the past three years, as well as 12 standardized documents such as "Passenger Service Guide", "Station Facility Description" and "Emergency Response Plan". After data cleaning, deduplication and quality screening, 58,263 valid question and answer pairs were retained. Three annotators with railway passenger service experience were organized to independently annotate the service corpus. The final category label was determined by majority voting mechanism, and the Kappa coefficient of annotation consistency reached 0.847. The classification dataset is distributed by timeliness category as follows: 11,726 static knowledge items, 9,584 semi-static knowledge items, 21,847 periodic knowledge items, and 15,106 dynamic knowledge items. The training, validation, and test sets are divided in an 8:1:1 ratio, with 46,610 samples in the training set, 5,826 in the validation set, and 5,827 in the test set. The general corpus for the transportation domain contains 32,000 samples, covering topics such as railway travel knowledge, public transportation transfer rules, and ticketing policies. The competency benchmark test set contains 500 test samples per category, totaling 2,000 samples.

[0069] (2) Experimental Environment and Setup: The experimental hardware environment consisted of eight NVIDIA A100 80GB graphics processing units. The deep learning framework adopted the DeepSpeed ​​ZeRO-3 distributed training strategy combined with gradient checkpointing technology to reduce memory usage. The teacher model selected Qwen2-72B as the pre-trained large language model. This model was open-sourced by Alibaba Cloud, adopted the Transformer architecture, contained 80 Transformer layers, had a hidden layer dimension of 8192, 64 attention heads, 8 key-value heads, a feedforward network intermediate layer dimension of 29568, a vocabulary size of 152064, and a total of approximately 72.7 billion parameters. The student model selected Qwen2-0.5B, also open-sourced by Alibaba Cloud, adopted the same Transformer architecture as Qwen2-72B, contained 24 Transformer layers, had a hidden layer dimension of 896, 14 attention heads, a feedforward network intermediate layer dimension of 4864, and a total of approximately 494 million parameters.

[0070] 2. Specific parameter configuration Teacher model frequency domain fine-tuning parameters: For an 8192×8192 dimensional attention weight matrix, each Transformer layer contains the query matrix from the attention mechanism. W Q (Dimensions 8192×8192), Key Matrix W K (Dimensions 8192×1024), Value Matrix WV （ (Dimensions 8192×1024), Output Matrix W O Four weight matrices (8192×8192 dimensions) and the gating matrix in the feedforward network. (Dimensions 8192×29568), Upward projection matrix (Dimensions 8192×29568), Lower Projection Matrix (Dimensions 29568×8192) Three weight matrices. Frequency band boundaries. K 1 = 1024 K 2=4096、 K 3=8192, the number of retained coefficients for each frequency band N 1=32、 N 2=64、 N 3=96、 N 4 = 128, each matrix retains 320 trainable frequency domain coefficients. The Qwen2-72B model has 80 layers, each with 7 weight matrices, for a total of approximately 179,200 trainable parameters, representing 0.000025% of the total 72.7 billion parameters. Base learning rate. =2×10 -6 Learning rate increment factor =2.5, then the learning rate for the low-frequency band is 2×10. -6 The learning rate for the low-to-mid frequency band is 5×10. -6 The mid-frequency learning rate is 1.25 × 10⁻⁶. -5 The high-frequency learning rate is 3.125 × 10⁻⁶. -5 The batch size was set to 4 samples per card, the global batch size was 32, the gradient accumulation steps were 4, the AdamW optimizer was used, the weight decay coefficient was 0.01, the cosine annealing learning rate scheduling strategy was adopted, the warm-up steps were 200, the total training steps were 14566, and the training time for a single session was approximately 96 hours.

[0071] Distillation parameters: First stage training: 5 rounds, batch size: 128, learning rate: 5×10⁻⁶ -5 The second phase of training consists of 5 rounds. The third phase consists of 3 rounds of training, with a calibration threshold of 80%. The joint loss function weight coefficients... =0.3、 =0.1.

[0072] End-to-end cloud collaboration parameters: The mobile terminal device is a smartphone carried by the passenger. In this embodiment, an Android smartphone equipped with a Qualcomm Snapdragon 8 Gen3 mobile platform and 12GB of RAM is used as an example. The lightweight student model, after INT4 quantization, has a model size of 347MB. It uses the llama.cpp inference engine for local inference, with a latency of approximately 0.8 seconds for the first token of a single query, a generation speed of approximately 18 tokens per second, and a peak memory usage of approximately 780MB. The cloud server is equipped with 8 NVIDIA A100 graphics processing units. Confidence threshold. =-1.8.

[0073] 3. Experimental Results and Beneficial Effects (1) Comparative Experiment of Teacher Model Fine-tuning Methods This experiment compares the method of this invention with full-parameter fine-tuning, low-rank adaptive fine-tuning (LoRA, r=64), and standard frequency-domain sparse fine-tuning (domain-aware uniform frequency allocation). All four methods are trained on the same classification dataset and evaluated on the same test set. The evaluation metric is the question-answering accuracy for each time-sensitive category of knowledge, using semantic matching based on BERTScore with a threshold of 0.85.

[0074] Table 1 shows the results of a comparative experiment on teacher model fine-tuning methods: Table 1. Comparative Experiment Results of Teacher Model Fine-tuning Methods

[0075] Experimental results show that, under the condition that the number of trainable parameters is the same as that of standard frequency domain sparse fine-tuning (both are 179,200, accounting for 0.000025%), the method of the present invention achieves an overall accuracy of 90.9%, which is 5.3 percentage points higher than the 85.6% of standard frequency domain sparse fine-tuning, 2.6 percentage points higher than the 88.3% of LoRA fine-tuning, and even exceeds the 90.4% of full-parameter fine-tuning by 0.5 percentage points. The method of the present invention achieves a dynamic knowledge accuracy of 86.3%, which is 10.2 percentage points higher than the 76.1% of standard frequency domain sparse fine-tuning. This indicates that the present invention, through knowledge timeliness-driven frequency band allocation and differentiated learning rate strategies, enables the frequency domain coefficients in the high-frequency and mid-frequency bands to obtain more sufficient learning signals.

[0076] (2) Comparative Experiment of Student Model Distillation Methods This experiment compares the progressive three-stage distillation method of the present invention with standard KL divergence one-stage distillation and reverse KL divergence two-stage distillation.

[0077] Table 2 shows the comparative experimental results of the student model distillation method: Table 2 Comparative Experiment Results of Student Model Distillation Methods

[0078] Experimental results show that the progressive three-stage distillation method of this invention achieves an overall accuracy of 84.0%, reaching 92.4% of the teacher model's overall accuracy of 90.9%, and is 8.9 percentage points higher than the standard KL one-stage distillation's 75.1%. Considering that the student model's parameter count is only 0.68% of the teacher model's, this performance retention rate indicates that the progressive distillation strategy of this invention can effectively transfer the domain capabilities of large models to lightweight models.

[0079] In summary, the method of this invention outperforms the comparative methods in terms of fine-tuning effect, model distillation performance, and edge-cloud update efficiency, verifying the effectiveness of the large model training and lightweight deployment method for the transportation hub vertical domain of this invention.

[0080] Reference manual attached Figure 3 The diagram shows a structural schematic of a large-scale model training and lightweight deployment system for a transportation hub vertical domain provided by an embodiment of the present invention.

[0081] This invention provides a large-scale model training and lightweight deployment system 30 for the vertical domain of transportation hubs, including: a processor 301 and a memory 302; The memory 302 stores programs or instructions that can run on the processor 301. When the program or instructions are executed by the processor 301, they implement the steps of the above-described method for training and lightweight deployment of large models in the vertical domain of transportation hubs, and can achieve the same technical effect. To avoid repetition, the present invention will not elaborate further.

[0082] It should be understood that the processor 301 in this embodiment of the invention may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0083] It should also be understood that the memory 302 in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DR RAM).

[0084] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0085] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0086] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0087] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0088] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0089] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0090] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0091] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0092] This invention provides a readable storage medium that stores a program or instructions. When the program or instructions are executed by a processor, they implement the steps of the above-described method for training and lightweight deployment of a large model in the vertical domain of transportation hubs, and achieve the same technical effect. To avoid repetition, this invention will not elaborate further.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A method for training and lightweight deployment of a large-scale model in the vertical domain of transportation hubs, characterized in that, Includes the following steps: S1: Collect service corpora from multiple scenarios in transportation hubs, and label them into four categories according to the timeliness of knowledge: static knowledge, semi-static knowledge, periodic knowledge, and dynamic knowledge, and construct a classification dataset; S2: Define weight increments for the weight matrix of the pre-trained large language model, apply discrete cosine transform to the weight increments to map them to the frequency domain space, divide the frequency domain coefficient matrix into four frequency band intervals according to the four categories, retain a preset number of frequency domain coefficients as trainable parameters in each frequency band interval, and fine-tune the training according to the differential learning rate to obtain the teacher model. S3: Construct a student model, and transfer the domain capabilities of the teacher model to the student model through a progressive three-stage distillation process to obtain a lightweight student model; S4: Deploy the lightweight student model to a mobile terminal device, deploy the teacher model to a cloud server, establish an end-to-cloud collaboration mechanism, and update the model parameters on the mobile terminal device according to a frequency band-level differentiated strategy; S5: Receive user dialogue request, call the lightweight student model to generate response text and return it to the user.

2. The method for training and lightweight deployment of large-scale models in the vertical domain of transportation hubs according to claim 1, characterized in that, In S1, the static knowledge is space and facility information with an update frequency of monthly to grade level; the semi-static knowledge is rule and process information with an update frequency of weekly to monthly level; the periodic knowledge is operational information with an update frequency of daily to quarterly level; and the dynamic knowledge is real-time status information with an update frequency of minute to hour level.

3. The method for training and lightweight deployment of large-scale models in the vertical domain of transportation hubs according to claim 1, characterized in that, In S2, the specific process of the discrete cosine transform includes the following sub-steps: S301: Represent the weight increment of the weight matrix as an increment matrix with the same dimension as the weight matrix; S302: Apply a two-dimensional discrete cosine transform to the increment matrix to obtain the frequency domain coefficient matrix; S303: Divide the frequency domain coefficient matrix into four frequency band intervals according to the L1 norm of the frequency coordinates, specifically including: dividing the low frequency region into the low frequency band corresponding to the static knowledge, dividing the mid-low frequency region into the mid-low frequency band corresponding to the semi-static knowledge, dividing the mid frequency region into the mid frequency band corresponding to the periodic knowledge, and dividing the high frequency region into the high frequency band corresponding to the dynamic knowledge.

4. The method for training and lightweight deployment of large-scale models in the vertical domain of transportation hubs according to claim 1, characterized in that, In S2, the differentiated learning rate is set as follows: the base learning rate is used as the learning rate for the low-frequency band, and the learning rates for the mid-low frequency band, mid-frequency band, and high-frequency band are the products of the base learning rate and the first, second, and third powers of the learning rate increment factor, respectively.

5. The method for training and lightweight deployment of large-scale models in the vertical domain of transportation hubs according to claim 1, characterized in that, The progressive three-stage distillation described in S3 includes: In the first stage, a general transportation knowledge from the teacher model is distilled into the student model using a general corpus in the transportation domain. In the second stage, the hub service knowledge of the teacher model is distilled into the student model using a transportation hub-specific corpus, and a capability preservation regularization term is introduced. The third stage involves constructing a competency benchmark test set according to the four categories to calibrate the student model's capabilities.

6. The method for training and lightweight deployment of large-scale models in the vertical domain of transportation hubs according to claim 5, characterized in that, The first and second stages use inverse KL divergence as distillation loss; the joint loss function used in the second and third stages includes three weighted summation terms: distillation loss term, capability preservation regularization term, and frequency domain approximation loss term.

7. The method for training and lightweight deployment of large-scale models in the vertical domain of transportation hubs according to claim 1, characterized in that, In S4, the frequency band-level differentiation strategy is as follows: only the frequency domain coefficient update amounts corresponding to the high frequency band and the mid frequency band are issued, while the frequency domain coefficients corresponding to the low frequency band and the mid-low frequency band remain unchanged.

8. The method for training and lightweight deployment of large-scale models in the vertical domain of transportation hubs according to claim 1, characterized in that, S4 further includes: the mobile terminal device calculates a confidence score for the output response text, uploads query samples with confidence scores lower than a preset threshold to the cloud server, and the cloud server performs incremental fine-tuning of the teacher model based on the uploaded query samples.

9. A large-scale model training and lightweight deployment system for the vertical domain of transportation hubs, characterized in that, include: Processor and memory; The memory stores programs or instructions that can run on the processor, which, when executed by the processor, implement the steps of the method for training and lightweight deployment of large models in the transportation hub vertical domain as described in any one of claims 1 to 8.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the method for training and lightweight deployment of large models in the vertical domain of transportation hubs as described in any one of claims 1 to 8.

Images