An end-to-end large model training deployment and agent creation integrated platform

Abstract

The application discloses an end-to-end large model training deployment and agent creation integrated platform, and relates to the technical field of artificial intelligence. The platform is sequentially connected from top to bottom with a user interaction layer, an API gateway layer, a business service layer, an inference engine layer and a data infrastructure layer; relying on simplified operation of a conversational agent, corpus generation evaluation and online low-latency inference tasks are respectively carried by independent double inference engines, three-level corpus automatic synthesis, lightweight LoRA fine-tuning hot loading and LLM automatic semantic evaluation capabilities are integrated, a complete closed loop of private domain data processing, model training and deployment iteration is connected, and an industry customized agent is built in one-stop. The application can solve the pain points of fragmented functions, wasted computing power, high operation threshold and inefficient iteration of existing tools, and has the compliance deployment advantages of multi-tenant isolation, local data not out of domain and multi-hardware adaptation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically to an end-to-end integrated platform for large model training and deployment and agent creation. Background Technology

[0002] As large language models are being implemented in industries such as finance and manufacturing, enterprises are building industry-customized intelligent agents based on their own compliance documents, business dialogues, and other private domain data. The complete process includes private domain corpus preparation, model fine-tuning, inference deployment, performance evaluation, and iterative optimization. Currently, the market toolchain is fragmented. One category consists of training tools like LLaMA Factory, Unsloth, and Axolotl, which only enable model fine-tuning; another category includes application platforms like Dify, FastGPT, and Coze, which only support RAG and agent orchestration and lack complete fine-tuning capabilities; the remainder consists of scattered tools for annotation, deployment, and metric calculation. Enterprises building intelligent agents need to switch between multiple heterogeneous systems, with each tool's technology stack, data format, and permission system being incompatible, leading to broken development chains and low deployment efficiency.

[0003] In terms of corpus processing, existing tools cannot generate training samples adapted to memory, reasoning, and multi-turn dialogue based on the original documents in a hierarchical manner. Data preparation relies on manual annotation, and the annotation cycle for millions of words of data can take two to three months. The data becomes invalid quickly after the documents are updated. There is a lack of document EDA analysis, LLM automated quality inspection, and semantic deduplication. It is difficult to identify sample illusions and duplication defects by relying solely on manual sampling.

[0004] Model training tools are geared towards algorithm engineers, relying on command lines and complex forms, making them inaccessible to non-business personnel. The lack of process isolation in multi-task parallelism can easily lead to memory leaks and process crashes. Furthermore, the lack of experimental tracking and version management means that LoRA weights cannot be easily integrated into inference services, and manual migration can result in version confusion.

[0005] The inference deployment adopts a single-model, single-instance architecture. When multiple business models are in parallel, the GPU utilization rate is less than 20%, resulting in serious waste of computing power. The existing platform does not support LoRA dynamic hot reloading, cannot share the base engine, and is mostly only adapted to single-machine GPU private deployment. It lacks containerized multi-environment operation and maintenance, and has insufficient quantitative adaptation to Ascend NPU, making it unable to take into account private, cloud, and edge deployment scenarios.

[0006] Model evaluation relies solely on manual sampling or BLEU and ROUGE text metrics. Manual evaluation suffers from significant subjective bias and is not scalable. Text metrics cannot identify semantic illusions or citation errors. Furthermore, the lack of an automated semantic evaluation link (LLM-as-Judge) prevents evaluation results from being fed back into the optimization dataset and thus hinders the formation of a closed-loop iteration.

[0007] The overall architecture lacks a unified microservice foundation and multi-tenant management. Corpus, model, and evaluation data are stored in a scattered manner. There is a lack of unified authentication, tenant isolation, and operation auditing. Private domain data cannot be stored locally without leaving the domain, making it difficult to meet industry compliance requirements. The modules are highly coupled, making it difficult to integrate with third-party systems. It is impossible to complete the entire chain of operations, including document import, corpus generation, fine-tuning, deployment, evaluation, and agent release, in one stop.

[0008] In summary, existing toolchains are fragmented in function and lack key capabilities such as hierarchical automatic corpus generation, multi-LoRA shared inference, conversational low-code operation, and automated semantic evaluation. They also suffer from drawbacks such as long development cycles, high computing costs, high barriers to entry, low iteration efficiency, and weak compliance guarantees. As a result, they are unable to meet the needs of enterprises to build industry intelligent agents in a low-cost, secure, and efficient manner. Overcoming the shortcomings of the existing technologies mentioned above is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0009] In view of this, the present invention provides an end-to-end integrated platform for large model training and deployment and agent creation, which solves the problems existing in the background technology.

[0010] To achieve the above objectives, the present invention provides the following technical solution: An end-to-end integrated platform for training, deploying, and creating large models and intelligent agents, consisting of interconnected user interaction layer, API gateway layer, business service layer, inference engine layer, and data and infrastructure layer from top to bottom; The user interaction layer is used to receive user business requests and schedule all backend operations based on the intelligent agent workflow. The API gateway layer serves as a unified traffic entry point, used to implement request routing and distribution, load balancing, and access authentication. The business service layer is configured with multiple independent and interconnected business service units, which are used to complete private data processing, model fine-tuning training, lightweight model deployment, and automated evaluation of model performance, respectively. The inference engine layer consists of two sets of inference clusters that share computing power but are isolated from each other. One set is responsible for data generation and model evaluation tasks, while the other set is responsible for online low-latency inference services. The data and infrastructure layer is used for unified management and control of platform computing resources, and persistent storage of business data, model assets and experimental records generated throughout the entire process; The interconnectedness of each level forms an automated closed loop encompassing private data processing, model fine-tuning and training, multi-model inference services, model performance evaluation, and agent iterative optimization, enabling one-stop completion of customized large-scale models and industry-specific intelligent agents for vertical domains.

[0011] Optionally, the user interaction layer integrates a corpus workbench, a training workbench, a deployment workbench, and an evaluation center. The user interaction layer is based on Next.js to build a conversational intelligent agent interaction interface, and uses a tool calling mechanism to parse the user's natural language business needs. Each workbench is configured with an independent state machine to orderly promote the business process, and the user can complete the entire process configuration without having to manually fill in forms across pages.

[0012] Optionally, the business service layer adopts the FastAPI asynchronous microservice architecture, which specifically includes data-interoperable and independent corpus services, training services, deployment services, and evaluation services.

[0013] Optionally, the corpus service uses PyMuPDF to parse enterprise private documents, perform EDA exploratory data analysis, complete text hierarchical sampling and global scene recognition, and automatically generate three-level hierarchical training corpora. The three-level hierarchical training corpora are short sentence question-and-answer samples corresponding to knowledge memorization, scene chain samples corresponding to logical reasoning, and multi-turn dialogue samples corresponding to natural interaction. The corpus service is equipped with LLM quality assessment and semantic similarity deduplication mechanism to complete corpus quality control.

[0014] Optionally, the training service is based on the MS-Swift and PEFT frameworks to implement LoRA / QLoRA lightweight fine-tuning and DPO / RLHF human preference alignment training, and supports multi-GPU distributed training. Each training task is run in an isolated subprocess to avoid memory leaks, and it is connected to the MLflow component to complete full experimental tracking of training hyperparameters, metrics, and model versions.

[0015] Optionally, the deployment service uses the vLLM engine to achieve dynamic hot reloading of lightweight LoRA adapters. Multiple LoRA adapters share the same base inference engine, and new models can be deployed without restarting the service. The deployment service has multi-tenant computing power and data isolation capabilities, and configures a three-level model release review process: draft, pending review, and already deployed.

[0016] Optionally, the evaluation service adopts the LLM-as-Judge large model judging mechanism, which scores semantics from 0 to 100 points on two dimensions: factual accuracy and contextual coherence, and simultaneously calculates BLEU and ROUGE auxiliary text indicators; the evaluation outputs a standardized report, and automatically feeds back the identified defective samples such as illusions, missing details, and citation errors to the corpus service for iterative optimization of the dataset.

[0017] Optionally, the inference engine layer is divided into a generation engine group and a service engine group; the generation engine group is configured with a large number of parameters to perform three-level corpus synthesis and LLM judge evaluation tasks; the service engine group is equipped with a lightweight base model and loads a LoRA adapter to carry low-latency online business inference requests for end users; the hardware computing resources of the two engine groups are isolated from each other, and the execution of tasks does not compete for resources.

[0018] Optionally, the data and infrastructure layer uses PostgreSQL to store metadata, and shares files to store training corpora, LoRA model weights, and experimental logs, and uniformly schedules and manages GPU computing power clusters; the platform completes full API gateway authentication based on JWT Token, realizes tenant data isolation through user ID, ensures that enterprise private documents and training data are stored locally without leaving the domain, and retains complete audit logs for key operations including training startup, model deployment, and data deletion.

[0019] Optionally, the platform uses Docker Compose to complete one-click containerization orchestration of all services, distinguishing three independent running environments: production, development, and MLflow experiments. The platform is also compatible with local private deployment and cloud deployment scenarios, and can be configured with a lightweight remote access method for clients, supporting quantitative adaptation and operation of Ascend NPU and general-purpose GPU computing power.

[0020] As can be seen from the above technical solution, compared with the prior art, the present invention discloses an integrated platform for end-to-end large model training and deployment and agent creation, which has the following beneficial effects: (1) The platform integrates the entire process of corpus preparation, model fine-tuning, inference deployment and automatic evaluation. It relies on conversational agent interaction to replace the traditional complex form configuration. Business personnel can independently complete the construction of vertical models and industry intelligent agents without professional AI knowledge, solving the industry pain points of switching between multiple tools and cumbersome operations.

[0021] (2) Relying on EDA document analysis, three types of training samples are automatically generated in layers: memory, reasoning, and multi-turn dialogue. Combined with LLM automatic quality inspection and semantic deduplication, it replaces the high-cost and long-cycle manual annotation, shortening the data preparation time from two to three months to days, while improving the diversity and quality of training data.

[0022] (3) The two computing power clusters for generation and online inference are isolated and do not compete with each other; the LoRA adapters of multiple services share the same base to achieve dynamic hot loading, eliminating the need to repeatedly deploy the complete model, greatly improving GPU utilization, reducing enterprise computing power procurement costs, and taking into account both evaluation quality and low latency requirements for online inference.

[0023] (4) Get rid of the limitations of manual sampling and basic text indicators, quantify the quality of model output from the semantic dimension, automatically collect defective samples back to the corpus module to optimize the dataset, build a data-driven automated iterative link, and shorten the model optimization cycle.

[0024] (5) Supports one-click orchestration of Docker containers, compatible with private and cloud deployments, equipped with remote operation and maintenance client, and is compatible with general-purpose GPUs and Ascend NPU domestic computing power, which can meet the deployment requirements of multiple industries such as finance, manufacturing, government affairs, and education.

[0025] (6) Based on JWT authentication and user ID tenant isolation, the enterprise's private data is stored locally without leaving the domain. The supporting model is launched with a multi-level audit and full-process operation audit mechanism to meet the data compliance management requirements of highly sensitive industries.

[0026] (7) The training process is independently isolated to avoid memory leakage. It integrates MLflow to fully record hyperparameters, metrics, and model versions, and supports multiple fine-tuning schemes such as LoRA / QLoRA / DPO / RLHF, taking into account both distributed multi-card training and fault-tolerant operation. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0028] Figure 1 This is a structural diagram of an integrated platform for end-to-end large model training, deployment, and agent creation provided by the present invention. Figure 2 This invention provides a schematic diagram of the overall data iteration value closed loop. Figure 3 A schematic diagram of the three-level hierarchical training corpus generation system provided by this invention; Figure 4 This is a schematic diagram of the end-to-end full-link data flow provided by the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] This invention discloses an end-to-end integrated platform for large model training and deployment and agent creation, such as... Figure 1 As shown, from top to bottom, there are interconnected user interaction layer, API gateway layer, business service layer, inference engine layer, and data and infrastructure layer; The user interaction layer is used to receive user business requests and schedule all backend operations based on the intelligent agent workflow. The API gateway layer serves as a unified traffic entry point, used to implement request routing and distribution, load balancing, and access authentication. The business service layer is configured with multiple independent and interconnected business service units, which are used to complete private data processing, model fine-tuning training, lightweight model deployment, and automated evaluation of model performance, respectively. The inference engine layer consists of two sets of inference clusters that share computing power but are isolated from each other. One set is responsible for data generation and model evaluation tasks, while the other set is responsible for online low-latency inference services. The data and infrastructure layer is used for unified management and control of platform computing resources, and persistent storage of business data, model assets and experimental records generated throughout the entire process; The interconnectedness of each level forms an automated closed loop encompassing private data processing, model fine-tuning and training, multi-model inference services, model performance evaluation, and agent iterative optimization, enabling one-stop completion of customized large-scale models and industry-specific intelligent agents for vertical domains.

[0031] Before formally introducing each module in a layered manner, let's first illustrate the application value of this invention in the context of real-world enterprise implementation scenarios. Currently, enterprises generally face three typical implementation challenges in building industry-specific intelligent agents: First, in the financial compliance customer service scenario, enterprises possess 2 million words of compliance documents and product manuals. Traditional solutions require 3 annotators to spend 2 months creating training data, followed by 3-4 weeks of optimization by algorithm engineers, resulting in an overall cycle exceeding 3 months. This is highly susceptible to the problem of training data becoming invalid after policy updates. Second, in the manufacturing knowledge transfer scenario, equipment maintenance experience is scattered across paper manuals and the memories of veteran employees. Small and medium-sized enterprises lack dedicated AI teams, making it difficult to build a dedicated "digital mentor" question-and-answer model. Third, in the multi-business scenario of a group, customer service, legal affairs, and HR each have independent models. The traditional single-model, single-instance deployment architecture results in GPU hardware utilization of less than 20%, leading to serious waste of computing resources. This invention's integrated platform designs a full-link closed-loop architecture to address the pain points of the above scenarios, enabling the completion of the entire model development process without switching between multiple sets of tools.

[0032] The top-level logic of this platform's design is to break down the existing fragmented toolchain in the market: existing tools are divided into two categories: training tools that only support fine-tuning and development platforms that only support RAG application building. These two types are incompatible and lack two crucial components: automated corpus governance and quantitative semantic evaluation. (Refer to...) Figure 2This invention integrates data engineering, model training, inference deployment, and performance evaluation into a data-driven iterative closed loop through a five-layer unified microservice architecture, solving the root cause problems of switching between multiple tools and process breaks from the underlying architecture.

[0033] Furthermore, the user interaction layer integrates a corpus workbench, a training workbench, a deployment workbench, and an evaluation center. The user interaction layer is based on Next.js to build a conversational intelligent agent interaction interface, and uses a tool calling mechanism to parse the user's natural language business needs. Each workbench is configured with an independent state machine to orderly promote the business process, and the user can complete the entire process configuration without having to manually fill in forms across pages.

[0034] Traditional ML development platforms all adopt a form-driven interaction mode, requiring more than 8 steps of manual operation across pages to complete a model training, and also requiring users to master the meaning of a large number of professional hyperparameters, making them only suitable for professional algorithm engineers. The conversational intelligent agent workflow of this invention relies on state machine constraints on business processes and tool calling to automatically schedule the background. It only requires 6 rounds of natural language interaction to complete the same task. Through innovation in human-computer interaction, it lowers the threshold for platform use, allowing business personnel to independently complete model customization.

[0035] Furthermore, the business service layer adopts a FastAPI asynchronous microservice architecture, specifically including data-interoperable yet independent corpus services, training services, deployment services, and evaluation services. These four types of microservices are decoupled and developed and deployed independently, while achieving data interoperability through a unified gateway. Each business process can be scaled up and isolated from faults independently without affecting the overall operation of the platform.

[0036] The logic of this microservice decomposition stems from the pain points of industry implementation: the computing power requirements, operation cycles, and operation and maintenance rules for data preparation, model training, online inference, and effect evaluation are completely heterogeneous. If they are coupled into a single program, resource contention and mutual interference of tasks will occur. In this embodiment, after decomposition, four types of services perform their respective functions, and at the same time, data flow is realized through the gateway to form an end-to-end closed loop.

[0037] Furthermore, the corpus service uses PyMuPDF to parse enterprise-owned documents, perform EDA exploratory data analysis, complete text hierarchical sampling and global scene recognition, and automatically generate a three-level hierarchical training corpus; among which, such as Figure 3 As shown, the three-level hierarchical training corpus consists of short question-and-answer samples corresponding to knowledge memorization, scenario chain samples corresponding to logical reasoning, and multi-turn dialogue samples corresponding to natural interaction. The corpus service is equipped with LLM quality assessment and semantic similarity deduplication mechanism to complete the corpus quality control.

[0038] Text stratified sampling is achieved through the following steps: S1: Based on PyMuPDF, the input document is structured and parsed to extract text blocks and hierarchical metadata; the hierarchical metadata includes chapter titles, paragraph text, table content, chart annotations and their hierarchical position information in the document; S2: Use a pre-trained sentence vector model to vectorize and embed each text block to generate the corresponding semantic vector; S3: Using cosine similarity as the distance metric, agglomerative hierarchical clustering algorithm is used to group semantic vectors into topic clusters, and the optimal number of clusters is automatically determined based on the silhouette coefficient; S4: Stratified sampling is performed based on the proportion of text volume of each topic cluster to the total text to ensure that the distribution of each topic in the training corpus matches the original document in a balanced manner.

[0039] Global scene recognition is achieved through the following steps: S5: Extract key entities for each topic cluster using a named entity recognition model, record the frequency and location of entity occurrence, and construct an entity-topic association matrix; S6: Match the pre-set business scenario rule library, and output candidate scenario tags and matching scores based on the specific trigger entities and matching weights of each scenario; S7: After concatenating the rule matching results with the topic cluster summary text, input the result into the large language model to complete the secondary scene classification, output scene labels with confidence scores, and mark topic clusters with confidence scores below the preset threshold as pending manual confirmation. S8: The three-level corpus generation process uses hierarchical sampling text blocks and corresponding scene labels as global context inputs. Based on the business scenario, short sentence question-and-answer samples, scene chain samples, and multi-turn dialogue samples are generated and then sent to LLM to complete quality assessment and semantic deduplication.

[0040] The three-level corpus stratification design has clear theoretical support, corresponding to the three cognitive levels in Bloom's Taxonomy of Educational Objectives: Level 1 short-sentence question-and-answer matching for the "memory and comprehension" objective, enabling the model to accurately remember basic facts in the domain; Level 2 scenario-chain sample matching for the "application and analysis" objective, training the model's ability to connect and reason about multiple knowledge points; and Level 3 multi-turn dialogue sample matching for the "synthesis and interaction" objective, optimizing the model's natural dialogue interaction effect. Experiments have verified that training with only a single level of data will result in limitations in the model's capabilities. The model trained with only Level 1 data cannot handle complex reasoning scenarios. Adding Level 2 samples significantly improves reasoning ability, and supplementing with Level 3 samples further optimizes dialogue fluency. Mixed training with three levels of samples can comprehensively improve the model's overall performance.

[0041] Meanwhile, the introduction of EDA exploratory data analysis before generating layered corpora is a core innovative logic of this module: traditional tools only generate question-and-answer samples independently block by block, lacking a global understanding of the full text's theme, entities, and scenarios, resulting in scattered data themes and incomplete scenario coverage; this platform first samples documents in layers, generates a global profile, automatically identifies business scenarios, and then injects the global context into the corpus generation process, greatly improving the consistency and scenario coverage of the generated samples.

[0042] Furthermore, the training service is based on the MS-Swift and PEFT frameworks to implement LoRA / QLoRA lightweight fine-tuning and DPO / RLHF human preference alignment training, and supports multi-GPU distributed training. Each training task is run in an isolated subprocess to avoid memory leaks, and it connects to the MLflow component to complete full experimental tracking of training hyperparameters, metrics, and model versions.

[0043] Among them, process isolation is the core innovative design of this training module. The underlying engineering logic it solves is: when multiple training tasks share the GPU, CUDA context leakage and mutual pollution of deep learning framework states will occur. The crash of a single task will terminate all queued training tasks. This solution isolates each training task through independent sub-processes, realizes task fault isolation, and ensures the stability of multi-task parallel training.

[0044] In addition, the platform is compatible with LoRA lightweight fine-tuning and DPO / RLHF preference alignment, unlike open-source training tools that only support basic fine-tuning. It takes into account both the needs of model knowledge fitting and humanized optimization of responses. It is equipped with MLflow complete experiment tracking, which solves the shortcomings of traditional script training, such as no version record, untraceable indicators, and difficulty in locating underfitting and overfitting problems.

[0045] Furthermore, the deployment service uses the vLLM engine to achieve dynamic hot loading of lightweight LoRA adapters. Multiple LoRA adapters share the same base inference engine, and new models can be deployed without restarting the service. The deployment service has multi-tenant computing power and data isolation capabilities, and configures a three-level model release review process: draft, pending review, and already deployed.

[0046] In scenarios involving concurrent loading of multiple LoRA adapters for various services, the deployment service is configured with an LRU memory cache eviction scheduling policy that includes tenant isolation and persistent protection. The specific execution steps are as follows: S1: Pre-configure the LoRA memory budget limit Vmax and the maximum concurrent adapter loading threshold Nmax for the base engine; maintain an activity record table for the LoRA adapters loaded in the memory, the record table stores the adapter identifier, tenant identifier, loading time, recent inference time, cumulative call count, and persistent flag bit; the persistent flag bit is used to identify core business adapters that do not participate in the elimination process; S2: When a new LoRA loading request is received, the total video memory usage (Vcurrent) of all adapters is calculated based on the activity record table, and the sum of Vcurrent and the estimated video memory (Vnew) of the adapter to be loaded is calculated. S3: If the total video memory usage is ≤ Vmax and the total number of adapters is ≤ Nmax, the new LoRA will be hot-loaded into the video memory and the activity record will be initialized. S4: If the total video memory usage exceeds Vmax or the number of adapters reaches Nmax, filter the adapters whose resident flag is negative, and prioritize the elimination of the adapter with the earliest inference time according to the LRU rule. S5: Unload the selected adapters to release video memory, and repeat the elimination process until the remaining video memory can accommodate the new LoRA. Then load the new adapter and update the activity record table. S6: When a business inference request is received, the corresponding LoRA adapter is matched based on the tenant identifier; if the adapter is already in memory, inference is executed directly and its activity record is updated; if it is not in memory, the loading process of S2~S5 is triggered, and inference is executed after loading is completed.

[0047] This LoRA shared inference architecture is a core innovation designed for the group's multi-business scenarios. Traditional solutions deploy a complete base inference instance for each business model, resulting in significant redundant GPU memory usage and, in actual testing, GPU hardware utilization of less than 20%. This solution, however, only loads a lightweight LoRA adapter of tens of MB, allowing multiple businesses to share the same base. New business models can be deployed without restarting services, significantly improving GPU computing power utilization and reducing enterprise hardware procurement costs. The accompanying three-level release review process matches the compliance requirements of the financial and government sectors, addressing the security shortcomings of open-source inference tools that lack deployment control and allow unverified models to be directly provided to the public.

[0048] Furthermore, the evaluation service adopts the LLM-as-Judge large model judging mechanism, which scores semantics from 0 to 100 points on two dimensions: factual accuracy and contextual coherence. It also calculates BLEU and ROUGE auxiliary text metrics simultaneously. The evaluation outputs a standardized report and automatically feeds back the identified defective samples such as illusions, missing details, and citation errors to the corpus service for iterative optimization of the dataset.

[0049] Existing evaluation schemes suffer from two inherent flaws: manual sampling suffers from significant subjective bias and cannot be scaled; BLEU and ROUGE only count text overlap, failing to identify core business defects such as semantic illusions, citation errors, and logical breaks. This invention introduces a large-scale model as a judge to perform semantic-level scoring, quantifying the model's actual business performance. Simultaneously, it establishes a data feedback channel between the evaluation and corpus services, constructing an automated iterative closed loop, fundamentally solving the problems of traditional schemes relying on manual screening and lengthy iteration cycles.

[0050] The platform underwent internal testing and validation, training a model on a dataset generated from a 100-page internal company manual. The overall pass rate for the 100 test samples was 78%. For the 22 defective samples that failed the evaluation, attribution analysis revealed the following causes: 41% were due to errors in remembering details, 27% were due to model illusions, 18% were due to missing key information, and 14% were due to incorrect clause citations. Based on this error analysis, the platform can automatically expand the training samples at the corresponding level, specifically fixing model defects and intuitively demonstrating the optimization logic of the evaluation closed loop.

[0051] Furthermore, the inference engine layer is divided into a generation engine group and a service engine group. The generation engine group is configured with a large number of parameters to perform three-level corpus synthesis and LLM judge evaluation tasks. The service engine group is equipped with a lightweight base model and loads a LoRA adapter to carry low-latency online business inference requests for end users. The hardware computing resources of the two engine groups are isolated from each other, and the execution of tasks does not compete for resources.

[0052] The underlying design logic of the dual-engine separation architecture is based on the fact that the computing power and model specifications requirements of the two types of tasks are completely conflicting: corpus synthesis and semantic evaluation pursue high generation quality, requiring models with a large number of parameters; online user inference pursues low latency and high concurrency, suitable for a lightweight base with LoRA; if the same batch of computing power is shared, the large evaluation model will compete for online inference resources, causing the terminal question-answering latency to soar. This solution physically isolates the hardware resources of the two types of tasks, balancing the quality of evaluation generation and the response speed of online business.

[0053] Furthermore, the data and infrastructure layer uses PostgreSQL to store metadata, and shares files to store training corpora, LoRA model weights, and experimental logs, while uniformly scheduling and managing GPU computing power clusters. The platform uses JWT Token to complete full API gateway authentication, enabling tenant data to be isolated by user ID, and ensuring that enterprise private documents and training data are stored locally without leaving the domain. Complete audit logs are retained for key operations including training startup, model deployment, and data deletion.

[0054] This layer of security architecture is designed to meet the stringent requirements of highly compliant industries such as finance and government. Existing open-source tools generally lack multi-tenant data isolation, operation auditing, and control over local data not leaving the domain, posing a risk of leakage of sensitive documents in the enterprise's private domain. This platform builds a compliance control system from three layers: storage, authentication, and operation logs, adapting to the needs of confidential business implementation.

[0055] The above five-layer architecture is linked together to form a complete business flow chain, such as Figure 4 As shown, the platform's complete end-to-end data flow is divided into four main stages: data preparation, model training, model deployment, and automated evaluation. The data generated in each stage can be automatically fed back to complete iterative optimization.

[0056] After introducing the various modules of the platform, the following are the end-to-end performance benchmarks of the platform, which intuitively reflect the efficiency of the entire chain operation: Using standard test samples, it takes about 15 minutes to generate a complete QA training corpus from a 100-page PDF document; about 30 minutes for 5000 training samples, 3 rounds of training, and 4-card distributed training; the dynamic loading latency of the LoRA adapter is less than 5 seconds; the P50 value of the first token latency in online inference is less than 700ms; and the overall time for automated semantic evaluation of 500 test sets is about 10 minutes, which is dozens of times more efficient than manual evaluation.

[0057] To accommodate the computing power requirements of enterprises of different sizes, the platform offers a standardized hardware deployment solution. The minimum hardware configuration requirements are four GPUs with 24GB of VRAM, a 16-core CPU, 64GB of RAM, and 500GB of SSD storage; the recommended production environment configuration is eight GPUs with 24GB or more VRAM, a 32-core CPU, 128GB of RAM, and 2TB of NVMe high-speed storage. The platform's GPU computing power is uniformly divided into three resource pools, corresponding to the generation engine group, the online service engine group, and the distributed training task group, respectively. Computing power in each group is allocated on demand and resources are isolated and scheduled.

[0058] Furthermore, the platform completes one-click containerization orchestration of all services based on Docker Compose, distinguishing three independent operating environments: production, development, and MLflow experiments. The platform is also compatible with local private deployment and cloud deployment scenarios, configures lightweight remote access methods for clients, and supports quantitative adaptation and operation of Ascend NPU and general-purpose GPU computing power.

[0059] The containerized multi-environment isolation design solves the problems of dependency conflicts and inconsistent configurations between development and production environments in traditional script deployments. It is also compatible with general-purpose GPUs and domestically developed Ascend NPUs, overcoming the hardware limitations of open-source tools that only support NVIDIA graphics cards, thus expanding the application scenarios for domestically developed solutions. The lightweight client access mode eliminates the need for high-end computing hardware on the terminal, requiring only remote scheduling of the backend computing cluster, making it suitable for multi-branch offices and lightweight edge computing scenarios.

[0060] The following is a complete implementation example using the construction of a teaching intelligent agent in the education industry, demonstrating the platform's practical operation process: The hardware utilizes a Huawei TaiShan 200 Linux server, an i7-13700K Windows client host, and an NVIDIA A100 computing host; the underlying operating system is Ubuntu 24.04TLS, and the supporting software versions include vllm-ascend 0.2.0, CANN 8.0, Docker 25.0.0, MLflow, etc., with Python 3.12.4 used as the development language. The specific implementation process is as follows: 1) By selecting the teaching scenario base model through the deployment service, environment adaptation and deployment can be completed with one click; 2) Access the corpus service, input the scenario keywords "middle school math knowledge point Q&A", and automatically generate layered training corpus based on EDA analysis; 3) Configure the learning rate to 0.001 and the training rounds to 80 on the training workbench, upload the generated corpus and start multi-card LoRA fine-tuning; 4) After training is completed, submit the three-level review process and remotely publish it to the campus intranet via the client; 5) Integrate with the RAG module to upload electronic textbooks and build a knowledge base, enabling interactive question-and-answer sessions between the intelligent agent and textbook documents; 6) For courseware images and other materials, multimodal support functions are used to link text and image knowledge points; 7) The platform has a built-in system diagnostic auxiliary module that can monitor hardware load in real time and automatically provide expansion and optimization suggestions when resource bottlenecks occur.

[0061] During the platform's deployment and application, comprehensive countermeasures are implemented to address risks across the entire value chain: For risks such as synthetic corpus illusions and data bias, a three-tiered quality assessment system of format validation, semantic deduplication, and LLM (Local Level Management) is established; to address the bottleneck of single-machine GPU computing power, LoRA lightweight training is currently used to lower the hardware threshold; to address the dependence on a single base model, the platform architecture decouples the model layer; to address the issue of increased latency from multiple LoRA concurrent inferences, the underlying vLLM engine incorporates an LRU cache eviction mechanism to optimize GPU memory usage; and to address compliance risks associated with sensitive financial and government data, a strict mechanism is implemented to ensure local data remains within its domain, multi-level release review, and full operation auditing.

[0062] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0063] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An end-to-end integrated platform for large-scale model training, deployment, and agent creation, characterized in that, From top to bottom, the system consists of interconnected layers: user interaction layer, API gateway layer, business service layer, inference engine layer, and data and infrastructure layer. The user interaction layer is used to receive user business requests and schedule all backend operations based on the intelligent agent workflow. The API gateway layer serves as a unified traffic entry point, used to implement request routing and distribution, load balancing, and access authentication. The business service layer is configured with multiple independent and interconnected business service units, which are used to complete private data processing, model fine-tuning training, lightweight model deployment, and automated evaluation of model performance, respectively. The inference engine layer consists of two sets of inference clusters that share computing power but are isolated from each other. One set is responsible for data generation and model evaluation tasks, while the other set is responsible for online low-latency inference services. The data and infrastructure layer is used for unified management and control of platform computing resources, and persistent storage of business data, model assets and experimental records generated throughout the entire process; The interconnectedness of each level forms an automated closed loop encompassing private data processing, model fine-tuning and training, multi-model inference services, model performance evaluation, and agent iterative optimization, enabling one-stop completion of customized large-scale models and industry-specific intelligent agents for vertical domains.

2. The end-to-end large model training and deployment integrated platform for agent creation according to claim 1, characterized in that, The user interaction layer integrates a corpus workbench, a training workbench, a deployment workbench, and an evaluation center. The user interaction layer is based on Next.js to build a conversational intelligent agent interaction interface. It adopts a tool calling mechanism to parse the user's natural language business requirements. Each workbench is configured with an independent state machine to promote the business process in an orderly manner. Users can complete the entire process configuration without having to manually fill in forms across pages.

3. The end-to-end large model training deployment and agent creation integrated platform according to claim 1, characterized in that, The business service layer adopts the FastAPI asynchronous microservice architecture, which specifically includes data-interoperable and independent corpus services, training services, deployment services, and evaluation services.

4. The end-to-end large model training and deployment integrated platform for agent creation according to claim 3, characterized in that, The corpus service uses PyMuPDF to parse enterprise private documents, perform EDA exploratory data analysis, complete text hierarchical sampling and global scene recognition, and automatically generate three-level hierarchical training corpora. The three-level hierarchical training corpora are short sentence question-and-answer samples corresponding to knowledge memory, scene chain samples corresponding to logical reasoning, and multi-turn dialogue samples corresponding to natural interaction. The corpus service is equipped with LLM quality assessment and semantic similarity deduplication mechanism to complete corpus quality control.

5. The end-to-end large model training and deployment and agent creation integrated platform according to claim 3, characterized in that, The training service is based on MS-Swift and PEFT frameworks to implement LoRA / QLoRA lightweight fine-tuning and DPO / RLHF human preference alignment training, and supports multi-GPU distributed training. Each training task is run in an isolated subprocess to avoid memory leaks, and it is connected to MLflow components to complete full experimental tracking of training hyperparameters, metrics, and model versions.

6. The end-to-end large model training and deployment and agent creation integrated platform according to claim 3, characterized in that, The deployment service uses the vLLM engine to achieve dynamic hot loading of lightweight LoRA adapters. LoRA adapters from multiple services share the same base inference engine, and new models can be deployed without restarting the service. The deployment service has multi-tenant computing power and data isolation capabilities, and a three-level model release review process: draft, pending review, and already deployed.

7. The end-to-end large model training and deployment integrated platform for agent creation according to claim 3, characterized in that, The evaluation service adopts the LLM-as-Judge large model judging mechanism, which scores semantics from 0 to 100 points on two dimensions: factual accuracy and contextual coherence. It also calculates BLEU and ROUGE auxiliary text metrics simultaneously. The evaluation outputs a standardized report and automatically feeds back the identified defective samples such as illusions, missing details, and citation errors to the corpus service for iterative optimization of the dataset.

8. The end-to-end large model training deployment and agent creation integrated platform according to claim 1, characterized in that, The inference engine layer is divided into a generation engine group and a service engine group. The generation engine group is configured with a large number of parameters to perform three-level corpus synthesis and LLM judge evaluation tasks. The service engine group is equipped with a lightweight base model and loads a LoRA adapter to carry low-latency online business inference requests for end users. The hardware computing resources of the two engines are isolated from each other, and the execution of tasks does not compete for resources.

9. The end-to-end large model training deployment and agent creation integrated platform according to claim 1, characterized in that, The data and infrastructure layer uses PostgreSQL to store metadata, and is equipped with shared file storage for training corpora, LoRA model weights, and experimental logs, and uniformly schedules and manages GPU computing power clusters; the platform completes full API gateway authentication based on JWT Token, realizes tenant data isolation through user ID, ensures that enterprise private documents and training data are stored locally without leaving the domain, and retains complete audit logs for key operations including training startup, model deployment, and data deletion.

10. The end-to-end large model training deployment and agent creation integrated platform according to claim 1, characterized in that, The platform is based on Docker Compose to complete the containerization and orchestration of the entire service with one click, and distinguishes three independent running environments: production, development, and MLflow experiments. The platform is also compatible with local private deployment and cloud deployment scenarios, and can be configured with a lightweight remote access method for clients. It supports quantitative adaptation and operation of two types of hardware computing power: Ascend NPU and general-purpose GPU.

Images