Structured and miniaturized large model deployment system and method oriented to vertical field

Abstract

The invention discloses a structured and miniaturized large model deployment system and method oriented to the vertical field, and relates to the field of model deployment, a front-end CPU real-time lightweight identification and scheduling module is deployed on a front-end processor to operate a lightweight model in a CPU of terminal equipment to generate low-dimensional structured data; deploying a cloud / edge GPU high-order reasoning service module and an intelligent evaluation optimization module on a cloud / edge processor to perform high-order reasoning based on low-dimensional structured data and perform intelligent optimization on a reasoning link; and deploying a structured world model and dynamic memory library module and a domain-specific model generation module on a rear-end processor to construct a domain-specific model and store industry knowledge by using the general large model according to a capability graph of a vertical domain by applying a miniaturized compression technology. The problems that the vertical integration capability of the frame is weak, the contradiction between the end-side computing power dependence and the cost is prominent, and the cloud model capability and the end-side resources are mismatched can be solved.

Description

Technical Field

[0001] This application relates to the field of model deployment, and in particular to a structured, miniaturized, large-scale model deployment system and method for vertical domains. Background Technology

[0002] Currently, the core challenge in deploying structured, miniaturized, and large-scale models for vertical industries lies in achieving efficient and stable model services in resource-intensive edge environments through a hardware-software collaborative deployment framework. As the demand for AI capabilities deepens in vertical industries such as smart manufacturing, smart healthcare, and unmanned retail, traditional cloud-based inference models face bottlenecks in terms of real-time performance, bandwidth costs, and data privacy. Therefore, researching a deployment framework that connects "model-compilation-runtime-hardware" has become crucial for promoting the democratization of AI. Existing deployment framework technologies mainly revolve around the following two directions.

[0003] In integrated model compression and compilation frameworks, researchers are dedicated to building a complete toolchain from model optimization to code generation. Such frameworks (e.g., TVM, Apache MXNet, TensorFlow Lite Micro) typically integrate optimization techniques such as quantization, pruning, and knowledge distillation, and provide a unified intermediate representation (IR) for cross-platform deployment. Specific strategies include compilation pipelines supporting mixed-precision quantization, automatic operator segmentation and fusion based on hardware characteristics, and customized graph optimization strategies for specific domains (e.g., perception models for autonomous driving). These frameworks can effectively improve performance under static models and known workloads, but their optimization process is often disconnected from dynamically changing actual input data and operating environments, resulting in poor resource utilization efficiency.

[0004] In the area of ​​runtime adaptive and collaborative reasoning frameworks, various dynamic management solutions have emerged to address the uncertainties and task diversity of edge environments. These frameworks focus on flexibly scheduling computing resources and model structures during the deployment phase. Examples include conditional computation frameworks that dynamically adjust model execution paths through lightweight monitors, edge-cloud collaborative frameworks that allocate computing tasks between the edge and cloud based on device status, and containerized frameworks that provide dynamic loading and resource sharing for multi-model applications. While these methods improve system flexibility, the introduced scheduling decision overhead, complex cross-node communication, and accuracy loss due to model partitioning remain challenging issues that need to be balanced in practical deployments.

[0005] The existing deployment framework has the following problems: (1) Weak vertical integration capability of the framework Existing deployment frameworks are mostly general-purpose designs, lacking deep adaptation to vertical domain business logic, data characteristics, and computing patterns, making it difficult to automatically generate highly customized deployment solutions through configuration.

[0006] (2) The contradiction between the dependence on edge computing power and cost is prominent. Current solutions for achieving intelligence on the device generally rely on integrating dedicated NPUs / GPUs, resulting in high hardware costs and power consumption; or they are forced to run severely performance-crippled miniature models, leading to a poor intelligent experience. This dilemma of "high cost" and "low performance" has hindered the large-scale adoption of the technology.

[0007] (3) Mismatch between cloud-based model capabilities and edge-side resources The existing architecture struggles to achieve optimal allocation of computing power. Forcing complex models to be deployed on the edge leads to a significant degradation in model capabilities due to resource constraints, making it unable to handle complex scenarios. Conversely, relying entirely on the cloud for processing is limited by bandwidth and latency, failing to meet real-time requirements. There is a lack of efficient coordination mechanisms between the powerful computing advantages of the cloud and the urgent real-time intelligence needs of the terminal. Summary of the Invention

[0008] The purpose of this application is to provide a structured, miniaturized, large-scale model deployment system and method for vertical domains, which can solve the problems of weak framework vertical integration capabilities, prominent contradiction between edge computing power dependence and cost, and mismatch between cloud model capabilities and edge resources.

[0009] To achieve the above objectives, this application provides the following solution: In the first aspect, this application provides a structured miniaturized large-scale model deployment system for vertical domains, including: a front-end processor, a cloud / edge processor and a back-end processor; The front-end processor is equipped with a front-end CPU real-time lightweight identification and scheduling module. The cloud / edge processor is equipped with a cloud / edge GPU high-level inference service module and an intelligent evaluation and optimization module. The back-end processor is equipped with a structured world model and dynamic memory module and a domain-specific model generation module. The front-end CPU real-time lightweight identification and scheduling module is used to run lightweight models in the CPU of terminal devices, generate low-dimensional structured data according to the task specifications defined in the capability graph, and upload the low-dimensional structured data to the cloud / edge GPU high-level inference service module. The domain-specific model generation module is used to construct domain-specific models from general large models based on the capability map of the vertical domain using miniaturization and compression technology; The cloud / edge GPU advanced inference service module is used to deploy domain-specific models on cloud or lightweight edge GPU nodes. It uses the domain-specific models to perform advanced inference based on low-dimensional structured data and matching data, derives inference results, and transmits the inference results to the intelligent evaluation and optimization module and terminal devices. The matching data is the result of semantic matching between low-dimensional structured data, structured world models, and industry knowledge stored in the dynamic memory module. The intelligent evaluation and optimization module is used to generate evaluation metrics based on inference results and corresponding real data; and to optimize the inference chain of the cloud / edge GPU high-level inference service module based on the evaluation metrics.

[0010] Secondly, this application provides a method for deploying structured, miniaturized, large-scale models for vertical domains, including: Deploy a real-time lightweight front-end CPU identification and scheduling module on the front-end processor; Deploy cloud / edge GPU high-level inference service modules and intelligent evaluation and optimization modules on cloud / edge processors; Deploy the structured world model and dynamic memory module and the domain-specific model generation module on the back-end processor; The lightweight model is run on the CPU of the terminal device using the front-end CPU real-time lightweight recognition and scheduling module. Low-dimensional structured data is generated according to the task specifications defined in the capability graph, and the low-dimensional structured data is uploaded to the cloud / edge GPU high-level inference service module. The domain-specific model generation module is used to construct a domain-specific model based on the capability map of the vertical domain from the general large model, and miniaturization and compression techniques are applied. Deploy domain-specific models on cloud or lightweight edge GPU nodes using cloud / edge GPU advanced inference service modules; The system utilizes domain-specific models to perform high-order reasoning based on low-dimensional structured data and matching data, derives reasoning results, and transmits these results to the intelligent evaluation and optimization module and terminal devices. The matching data is the result of semantic matching between low-dimensional structured data, the structured world model, and industry knowledge stored in the dynamic memory module.

[0011] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a structured, miniaturized, large-scale model deployment system and method for vertical domains. A front-end CPU real-time lightweight identification and scheduling module is deployed on the front-end processor to run lightweight models and generate low-dimensional structured data on the terminal device's CPU. A cloud / edge GPU high-order inference service module and an intelligent evaluation and optimization module are deployed on the cloud / edge processor to perform high-order inference based on the low-dimensional structured data and intelligently optimize the inference chain. A structured world model and dynamic memory module and a domain-specific model generation module are deployed on the back-end processor to construct domain-specific models and store industry knowledge based on the capability map of the vertical domain using miniaturization compression technology. This application utilizes only the CPU for low-computing operations on the device terminal, while applying domain-specific models in the cloud / edge GPU for more complex inference calculations. This solves the problems of weak framework vertical integration capabilities, prominent contradictions between edge computing power dependence and cost, and mismatch between cloud model capabilities and edge resources. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A schematic diagram of functional modules for a structured, miniaturized, large-scale model deployment system for vertical domains, provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a structured, miniaturized, large-scale model deployment method for vertical domains, provided as an embodiment of this application. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] In one exemplary embodiment, such as Figure 1 As shown, a structured, miniaturized, large-scale model deployment system for vertical industries is provided, including: a front-end processor, a cloud / edge processor, and a back-end processor; The front-end processor is equipped with a front-end CPU real-time lightweight identification and scheduling module.

[0017] The cloud / edge processor is equipped with a cloud / edge GPU high-level inference service module and an intelligent evaluation and optimization module.

[0018] The backend processor is equipped with a structured world model and dynamic memory module and a domain-specific model generation module.

[0019] The front-end CPU real-time lightweight identification and scheduling module is used to run lightweight models on the CPU of terminal devices, generate low-dimensional structured data according to the task specifications defined in the capability graph, and upload the low-dimensional structured data to the cloud / edge GPU high-level inference service module.

[0020] "Scheduling" refers to the process of outputting instructions or transmitting information by calculating received information. For example, in an image recognition task, the CPU is responsible for compressing the received information, deciding which part of the image to extract and compress into lower-dimensional data for transmission to the next stage.

[0021] In terminal devices, only the CPU is needed to complete basic perception and scheduling tasks. This application designs a "CPU real-time recognition + semantic instruction generator" module (i.e., a front-end CPU real-time lightweight recognition and scheduling module), which performs low-computational operations such as object detection, region filtering, and feature extraction through lightweight operators, traditional vision methods, and binarization networks. Based on the task specifications defined in the capability graph, this module only generates and uploads low-dimensional structured data, without transmitting complete images, significantly reducing bandwidth requirements.

[0022] The lightweight operator uses lightweight network skeletons such as MobileNetV3 and ShuffleNetV2, and combines them with neural architecture search technology to search for the optimal minimalist network structure for specific scenarios.

[0023] Binary Neural Networks (BNNs) are used for binary classification or simple classification tasks. Their weights and activation values ​​are compressed to 1-2 bits, enabling high inference speeds on CPUs.

[0024] The domain-specific model generation module is used to construct domain-specific models from general large models based on the capability map of the vertical domain using miniaturization and compression techniques.

[0025] The general-purpose large model is functionally decomposed based on the capability graph of the vertical domain, forming sub-modules such as domain semantic models, structured knowledge units, and reasoning modules. Through capability-aware pre-trained knowledge reselection, hierarchical distillation, structured pruning, and quantization strategies, parameters and knowledge highly relevant to the domain task are extracted from the general-purpose large model, achieving miniaturization and compression of the model, enabling it to run efficiently on cloud or edge GPUs.

[0026] Among them, structured pruning is based on the module division of the capability map, and performs "module-level" or "attention-head-level" pruning.

[0027] The hierarchical distillation process introduces "multi-granularity supervision." Logical distillation is performed not only at the output of the final layer of the model but also at the intermediate layers for "feature map alignment," enabling the miniaturized model (domain-specific model) to learn the implicit representations within the generalized larger model. Simultaneously, a "thought chain dataset" is constructed to distill the reasoning process for key reasoning capabilities.

[0028] The quantization strategy employs "mixed precision quantization". Different bit widths are used for quantization of layers with different sensitivities in the miniaturized model. INT8 quantization is used for the embedding layer and the first and last layers. During the training phase, the core computation layer uses "quantization-aware training" to simulate quantization error and ensure that the model accuracy loss is minimized.

[0029] The cloud / edge GPU advanced inference service module is used to deploy domain-specific models on cloud or lightweight edge GPU nodes. It uses the domain-specific models to perform advanced inference based on low-dimensional structured data and matching data, derives inference results, and transmits the inference results to the intelligent evaluation and optimization module and terminal devices. The matching data is the result of semantic matching between low-dimensional structured data, structured world models, and industry knowledge stored in the dynamic memory module.

[0030] Deploying structured, domain-specific models on cloud or lightweight edge GPU nodes, these models are responsible for performing complex computations such as high-order semantic reasoning, cross-temporal and spatiotemporal information fusion, structured decision-making, and task planning. Structured data uploaded from the front end triggers the execution of corresponding capability units, and the system optimizes the execution order and efficiency of the inference chain based on capability dependencies.

[0031] Among them, cross-temporal information fusion utilizes recurrent neural networks (RNN) or Transformer encoders to model continuous time-series data from the same or related devices in order to identify behavioral patterns or state change trends.

[0032] For the structured world model and dynamic memory modules, the backend maintains a continuously updated vertical domain world model, encompassing information such as target type, behavior patterns, operational knowledge, causal logic, and scene state machine. This model is built upon an industry knowledge graph and uses automated technology to extract entities and relationships from domain documents, achieving semantic enhancement and accelerated reasoning. The frontend recognition results can be matched with this world model in real time, improving recognition accuracy and contextual understanding capabilities.

[0033] The memory bank primarily stores key information that arises during a task. The difference between it and the world model is that both can be viewed as "memories" for storing information to facilitate reasoning and computation. However, the world model originates from important information about the tasks that might occur within a larger model, while the memory bank is primarily generated during the execution of the task. For example, a skilled worker may have acquired sufficient knowledge of factory operations in school before entering the workforce, establishing a certain cognitive foundation. However, upon starting work in a particular factory, they may find that much of the operational knowledge requires focused learning or even relearning, while some previously acquired knowledge becomes irrelevant. Therefore, this worker needs to dynamically adjust their operational knowledge, simultaneously relearning (supplementing detailed knowledge and forgetting unnecessary knowledge) the knowledge learned in school (updating the world model) and summarizing their work experience (updating the memory bank).

[0034] The automated knowledge extraction and graph construction process utilizes graph neural networks (GNNs) to learn representations of the knowledge graph, mapping entities and relations into low-dimensional vectors. This enables efficient "semantic retrieval" through vector similarity calculation.

[0035] The intelligent evaluation and optimization module is used to generate evaluation metrics based on inference results and corresponding real data; and to optimize the inference chain of the cloud / edge GPU high-level inference service module based on the evaluation metrics.

[0036] An automated intelligent evaluation and optimization module is built based on capability graphs to monitor the performance of various capability units across multiple dimensions, including accuracy, response time, and energy consumption. By plotting performance degradation curves between model size, inference speed, and overall performance, the system assists in optimizing compression strategies. Based on the evaluation results, the system can automatically identify bottlenecks and recommend targeted optimization measures, enabling continuous iteration and closed-loop optimization after deployment.

[0037] The multidimensional evaluation index system includes: Business metrics: accuracy, recall, F1 score, average precision.

[0038] Performance metrics: end-to-end latency, throughput (QPS).

[0039] Resource metrics: CPU / GPU utilization, memory usage, front-end device power consumption, and network bandwidth usage.

[0040] Automation bottleneck analysis and recommendations: The system will plot a "three-rate curve"—the relationship between model size (number of parameters), inference speed (FPS), and task accuracy. By analyzing the inflection points of the curves, it will intelligently recommend the optimal compression ratio.

[0041] In one exemplary embodiment of this application, the intelligent evaluation and optimization module is further used to optimize the resource scheduling of the cloud / edge GPU high-order inference service module, and dynamically allocate GPU resources according to the real-time load to perform the inference process according to task priority.

[0042] In one exemplary embodiment of this application, the intelligent evaluation and optimization module is further configured to adjust the model parameters of the lightweight model in the CPU of the terminal device according to the evaluation index.

[0043] In one exemplary embodiment of this application, the intelligent evaluation and optimization module is further configured to store the inference results output by the domain-specific model into the structured world model and dynamic memory module.

[0044] In one exemplary embodiment of this application, the intelligent evaluation and optimization module is further configured to adjust the detection threshold referenced by the domain-specific model when performing inference or to add training samples of the domain-specific model based on the inference results output by the domain-specific model.

[0045] This application proposes a front-end and back-end closed-loop optimized collaborative architecture consisting of "semantic pre-selection—structured data upload—large model inference—policy feedback—intelligent evaluation." The front-end CPU handles only lightweight computation and instruction generation, while cloud or edge GPUs perform complex inference, returning high-level semantic results and task instructions to the terminal. Finally, the intelligent evaluation module completes parameter updates and closed-loop optimization. This mechanism can still achieve highly intelligent inference in vertical domains even in low-computing-power front-end environments and can autonomously optimize itself, making it suitable for scenarios such as drone inspection, industrial quality inspection, security monitoring, robot collaboration, and vehicle platforms.

[0046] In one exemplary embodiment of this application, such as Figure 2 As shown, a method for deploying structured, miniaturized, large-scale models for vertical domains is provided, including: S1: Deploy a front-end CPU real-time lightweight identification and scheduling module on the front-end processor.

[0047] S2: Deploy cloud / edge GPU advanced inference service modules and intelligent evaluation and optimization modules on cloud / edge processors.

[0048] S3: Deploy the structured world model and dynamic memory module and the domain-specific model generation module on the back-end processor.

[0049] S4: Utilize the front-end CPU real-time lightweight recognition and scheduling module to run a lightweight model on the terminal device's CPU, generate low-dimensional structured data according to the task specifications defined in the capability graph, and upload the low-dimensional structured data to the cloud / edge GPU high-level inference service module.

[0050] S5: Using the domain-specific model generation module, the general large model is transformed into a domain-specific model based on the capability map of the vertical domain, and miniaturization and compression techniques are applied to construct the domain-specific model.

[0051] S6: Deploy domain-specific models on cloud or lightweight edge GPU nodes using cloud / edge GPU advanced inference service modules.

[0052] S7: Utilize domain-specific models to perform high-order reasoning based on low-dimensional structured data and matching data, derive reasoning results, and transmit the reasoning results to the intelligent evaluation and optimization module and terminal devices; the matching data is the result of semantic matching between low-dimensional structured data, structured world models, and industry knowledge stored in the dynamic memory module.

[0053] In one exemplary embodiment of this application, the structured miniaturized large model deployment method for vertical domains further includes: Deploy an intelligent evaluation and optimization module on the cloud / edge processor; use the intelligent evaluation and optimization module to generate evaluation metrics based on the inference results and corresponding real data; optimize the inference chain of the cloud / edge GPU high-order inference service module based on the evaluation metrics.

[0054] In one exemplary embodiment of this application, the structured miniaturized large model deployment method for vertical domains further includes: The intelligent evaluation and optimization module optimizes resource scheduling for the cloud / edge GPU high-level inference service module, dynamically allocating GPU resources based on real-time load to perform the inference process according to task priority.

[0055] In one exemplary embodiment of this application, the structured miniaturized large model deployment method for vertical domains further includes: The intelligent evaluation and optimization module adjusts the model parameters of the lightweight model in the CPU of the terminal device according to the evaluation indicators.

[0056] In one exemplary embodiment of this application, the structured miniaturized large model deployment method for vertical domains further includes: The intelligent evaluation and optimization module stores the inference results output by the domain-specific model into the structured world model and dynamic memory module.

[0057] The following is combined with Figure 1The deployment method of this application will be explained using two different application scenarios.

[0058] Scenario 1: Smart Factory Quality Inspection Scenario Terminal device layer → Lightweight recognition and instruction generation of front-end CPU real-time lightweight recognition and scheduling module, specifically refers to: based on the part surface image taken by the industrial camera, the size data collected by the sensor, etc., the front-end CPU runs a lightweight model to preliminarily determine whether the part has obvious defects (such as scratches, deformation), and generates a "suspected defect -> upload" instruction.

[0059] The front-end CPU real-time lightweight identification and scheduling module uploads low-dimensional data to the cloud and edge GPU high-level inference service modules. Specifically, this refers to uploading compressed feature vectors of suspected defective images to the cloud.

[0060] The operational data feedback from the cloud / edge GPU high-level inference service module to the intelligent evaluation module specifically refers to: the domain-specific model transmitting the completed identification and classification results to the intelligent evaluation and optimization module, which then makes a judgment based on the real data provided by the workers and initiates the subsequent optimization process.

[0061] The cloud / edge GPU high-level inference service module outputs task planning results to the terminal device layer, specifically referring to making corresponding adjustments to the quality inspection equipment, such as the shooting angle and sensor parameters.

[0062] The optimization of the inference link between the intelligent evaluation module and the cloud / edge GPU advanced inference service module specifically refers to: the intelligent evaluation module providing feedback to optimize the resource scheduling of the inference module, dynamically allocating GPU resources according to the real-time load, prioritizing high-risk batches, and optimizing the inference link to accelerate the inference recognition speed.

[0063] The intelligent evaluation module optimizes the scheduling parameters of the front-end CPU real-time lightweight identification and scheduling module, specifically by adjusting the parameters of the lightweight model of the front-end CPU based on the evaluation results.

[0064] Intelligent assessment module → Updated world model with structured world model and dynamic memory, specifically: adding new defect types to the knowledge graph and associating them with possible production processes (such as mold wear, abnormal assembly pressure).

[0065] The intelligent evaluation module → domain-specific model generation module's model optimization recommendation specifically refers to: based on newly discovered defect patterns, recommending adjustments to the detection threshold or enhancing the training samples for a certain type of defect.

[0066] Domain-specific model generation module → pre-trained model of cloud / edge GPU high-order inference service module, specifically refers to the deployment of domain-specific models derived from the functions of large models on cloud / edge GPUs.

[0067] The structured world model and dynamic memory module → semantic matching and inference acceleration of the cloud / edge GPU high-level inference service module specifically refers to comparing the current product defect situation with the frequently occurring situations in the memory, thereby improving the inference path and accelerating the inference process.

[0068] The functional breakdown of the large model → domain-specific model generation module specifically refers to: performing operations such as knowledge reselection, hierarchical distillation, structured pruning, and quantification strategies on the large model to obtain a small-scale factory assembly line inspection-specific model.

[0069] The large model → knowledge graph construction of structured world model and dynamic memory bank specifically refers to: integrating common product defect types based on information in the large model, adding corresponding detailed features, and recording historical defect data, production line parameters, etc.

[0070] Scenario 2: Smart Medical Remote Monitoring Scenario Terminal device layer → Lightweight recognition and instruction generation of the front-end CPU real-time lightweight recognition and scheduling module. Specifically, based on the ECG waveform, blood oxygen saturation time series data, etc. collected by the wearable device, the front-end CPU runs a lightweight model and performs real-time heart rhythm abnormality detection. If suspected atrial fibrillation is found, an "emergency upload" instruction is triggered.

[0071] The front-end CPU real-time lightweight identification and scheduling module uploads low-dimensional data to the cloud and edge GPU high-level inference service module. Specifically, this refers to uploading compressed feature vectors of suspected anomalies to the cloud / edge GPU.

[0072] The operational data feedback from the cloud / edge GPU high-level inference service module to the intelligent assessment module specifically refers to: the domain-specific model transmitting the inference results of the identified condition to the intelligent assessment optimization module, which then makes a judgment based on the real data recorded by the doctor and initiates the subsequent optimization process.

[0073] The cloud / edge GPU high-level inference service module outputs task planning results to the terminal device layer, specifically referring to corresponding adjustments to the monitoring equipment, such as detection interval time and detection sensor parameters.

[0074] The inference chain optimization from the intelligent evaluation module to the cloud / edge GPU advanced inference service module specifically refers to: The intelligent assessment module optimizes the resource scheduling of the inference module by dynamically allocating GPU resources based on real-time load, prioritizing the processing of high-risk medical records, and optimizing the inference chain to accelerate inference and recognition speed.

[0075] The intelligent evaluation module optimizes the scheduling parameters of the front-end CPU real-time lightweight identification and scheduling module, specifically by adjusting the parameters of the lightweight model of the front-end CPU based on the evaluation results.

[0076] The intelligent assessment module is an updated world model of structured world model and dynamic memory bank. Specifically, it refers to: accumulating rare case characteristics, optimizing the recognition accuracy of similar situations in the future, or adding knowledge nodes such as "specific waveform variations after the use of a certain drug".

[0077] The intelligent assessment module → domain-specific model generation module's model optimization recommendation specifically refers to: based on newly discovered medical record physiological information, recommending adjustments to the detection threshold or enhancing the training samples for a certain type of disease.

[0078] Domain-specific model generation module → Pre-trained models for cloud / edge GPU high-order inference service module, specifically referring to: domain-specific models decomposed from large model functions and deployed on cloud / edge GPUs. The structured world model and dynamic memory bank → semantic matching and inference acceleration of the cloud / edge GPU high-level inference service module, specifically refers to: matching the current symptoms with the typical case database to quickly generate differential diagnosis suggestions, thereby improving the inference path and accelerating the inference process.

[0079] The functional breakdown of the large model → domain-specific model generation module specifically refers to: performing operations such as knowledge reselection, hierarchical distillation, structured pruning, and quantification strategies on the large model to obtain a small-scale medical monitoring-specific model.

[0080] The large model → knowledge graph construction of a structured world model and dynamic memory bank specifically refers to: integrating medical guidelines, drug databases, and individual patient differences (such as age and complications) based on information in the large model, adding corresponding detailed features, and recording historical monitoring data.

[0081] This application has the following technical advantages: (1) Lower terminal computing power requirements and deployment costs In contrast to the current situation, existing solutions typically require integrating an NPU or high-performance GPU into the device to achieve high intelligence on the terminal, or running a very small model that severely sacrifices performance, resulting in high costs or poor user experience.

[0082] In this application, a front-end CPU-based real-time lightweight recognition and scheduling module is set up to decouple functions, separating complex "semantic reasoning" from basic "perceptual recognition." The terminal is only responsible for the latter, an area where traditional algorithms and lightweight networks excel, which the CPU is fully capable of handling. Techniques optimized for the CPU environment, such as "lightweight operators, traditional vision methods, and binarization networks," are used instead of forcing the execution of a massive Transformer model, thus technically ensuring feasibility under low computing power conditions. In this application, the terminal device only requires the CPU to run, greatly reducing hardware costs and power consumption.

[0083] (2) Less bandwidth required for data transmission In contrast to the current situation, traditional cloud-based AI solutions require continuous uploading of complete images or video streams captured by cameras, which places extremely high demands on network bandwidth, results in high latency, and involves the transmission of a large amount of private data.

[0084] In this application, a front-end CPU real-time lightweight identification and scheduling module is set up, adopting a "semantic pre-selection" strategy: the terminal is no longer a simple data pipeline, but has a preliminary semantic understanding capability (generating instructions based on the capability map). It only extracts and uploads "valuable information," rather than "raw data," compressing the raw, highly redundant pixel information into an extremely refined structured semantic description, fundamentally solving the bandwidth bottleneck. This application only uploads "low-dimensional structured data" (such as "coordinates (x,y), one 'damaged insulator' was found"), text instructions with extremely small data volume, reducing bandwidth requirements by several orders of magnitude.

[0085] (3) Higher performance cloud inference assurance In contrast to the current situation, micro-models deployed directly on the terminal often have significantly reduced capabilities due to insufficient parameters, making them unable to handle complex or long-tail scenarios.

[0086] This application utilizes a domain-specific model generation module and a high-order inference service module on cloud / edge GPUs to generate domain-specific models through targeted optimization and compression. By employing techniques such as "hierarchical distillation" and "structured pruning," it avoids simply shrinking large models but selectively retains the knowledge and capabilities most needed in the vertical domain, achieving a "small but precise" approach. The high-order inference service module on cloud / edge GPUs leverages the computing power of these GPUs to specialize in complex tasks such as "high-order semantic reasoning" and "cross-temporal information fusion," concentrating computing power for complex reasoning to ensure the accuracy and depth of the final decision. Combining a structured world model with a dynamic memory world model provides contextual and common-sense reasoning capabilities not available in terminal models. This application places the most complex inference tasks on cloud / edge GPUs, enabling the execution of a compressed yet highly capable "domain-specific model," significantly outperforming deployment of the inference module on the terminal. Due to considerations of weight and size (deploying models also requires hardware support; overly bloated models can impact device performance and increase costs), maximum performance optimization cannot be achieved.

[0087] (4) Better vertical domain adaptability and knowledge integration capabilities In contrast to the current situation: general-purpose large models may perform poorly in specific domains and lack industry knowledge; while traditional industry software lacks deep semantic understanding and reasoning capabilities.

[0088] This application utilizes a domain-specific model generation module and a structured world model and dynamic memory module to achieve domain knowledge internalization: the pre-training and distillation process of the domain-specific model is based on a "vertical domain capability graph," so that the model parameters themselves contain domain knowledge. External knowledge enhancement is also possible; the structured world model and dynamic memory module construct an explicit and updatable industry knowledge base (world model), providing strong background knowledge support for cloud-based inference. When a terminal identifies a "valve," the cloud can immediately associate it with its operating procedures, common failure modes, etc., thereby achieving a deep level of understanding. The deployment framework of this application, from the domain-specific model to the memory, is tailored to a specific domain and deeply integrates industry knowledge.

[0089] (5) Higher maintainability and continuous optimization capability In contrast to the current situation, most deployed models are static, and optimization requires manual intervention, which is time-consuming and costly.

[0090] This application utilizes an intelligent evaluation and optimization module (as the core of the closed-loop drive) to not only evaluate accuracy but also correlate it with engineering indicators such as model size, inference speed, and energy consumption, plotting loss curves to provide a quantitative basis for optimization. The evaluation results can be directly fed back to the domain-specific model (guiding the tuning of model compression strategies) and the front-end CPU real-time lightweight identification and scheduling module (potentially optimizing front-end scheduling strategies), forming a complete closed loop of "deployment-monitoring-analysis-optimization." This enables the system to adapt to changes in data distribution and continuously improve efficiency. This application constructs an automated evaluation and optimization closed loop that can continuously monitor performance and automatically tune it, achieving system self-evolution.

[0091] Existing framework deployments suffer from insufficient collaboration across toolchain components: the toolchain, from model training, compression, and conversion to target platform deployment, exhibits a severe "silo effect," with inconsistent standards and loosely coupled interfaces across components. This not only prevents optimization strategies from being consistently transferred across stages but also forces developers to invest significant effort in integration and debugging, substantially increasing the overall collaboration cost and complexity. This application adopts a general large-model deployment framework, organically connecting the pre-defined components within this framework. This avoids the lengthy large-model deployment process required to start from scratch for a new, specific problem, and simultaneously reduces the integration and debugging costs caused by differing standards between undefined components.

[0092] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A vertical domain-oriented structured miniaturized large model deployment system, characterized in that, The application relates to a method for constructing a structured world model and a dynamic memory library and a domain-specific model generation module. The front-end CPU real-time lightweight identification and scheduling module is used for running a lightweight model in a CPU of a terminal device, generating low-dimensional structured data according to a task specification defined in a capability graph, and uploading the low-dimensional structured data to the cloud / edge GPU high-order inference service module. The domain-specific model generation module is used for constructing a domain-specific model by applying a miniaturized compression technology to a general large model according to a capability graph of a vertical field. The cloud / edge GPU high-order inference service module is used for deploying the domain-specific model on a cloud or a lightweight edge GPU node, performing high-order inference based on low-dimensional structured data and matching data by using the domain-specific model, obtaining an inference result, and transmitting the inference result to the intelligent evaluation optimization module and the terminal device. The intelligent evaluation optimization module is used for generating an evaluation index according to the inference result and corresponding real data, and optimizing an inference link of the cloud / edge GPU high-order inference service module according to the evaluation index. The intelligent evaluation optimization module is also used for optimizing resource scheduling of the cloud / edge GPU high-order inference service module, dynamically allocating GPU resources according to real-time loads to perform an inference process according to a task priority. The intelligent evaluation optimization module is also used for adjusting model parameters of the lightweight model in the CPU of the terminal device according to the evaluation index. The intelligent evaluation optimization module is also used for storing the inference result output by the domain-specific model into the structured world model and dynamic memory library module. The intelligent evaluation optimization module is also used for adjusting a detection threshold referred to by the domain-specific model when performing inference or adding a training sample of the domain-specific model according to the inference result output by the domain-specific model.

2. The vertically field-oriented structured miniaturized macro-model deployment system of claim 1, wherein, The application relates to a method for constructing a structured world model and a dynamic memory library and a domain-specific model generation module.

3. The vertically field-oriented structured miniaturized macro-model deployment system of claim 1, wherein, The front-end CPU real-time lightweight identification and scheduling module is used for running a lightweight model in a CPU of a terminal device, generating low-dimensional structured data according to a task specification defined in a capability graph, and uploading the low-dimensional structured data to the cloud / edge GPU high-order inference service module.

4. The vertically field-oriented structured miniaturized macro-model deployment system of claim 1, wherein, The domain-specific model generation module is used for constructing a domain-specific model by applying a miniaturized compression technology to a general large model according to a capability graph of a vertical field.

5. The vertically field-oriented structured miniaturized macro-model deployment system of claim 1, wherein, The cloud / edge GPU high-order inference service module is used for deploying the domain-specific model on a cloud or a lightweight edge GPU node.

6. A method for deploying structured, miniaturized, large-scale models for vertical domains, characterized in that, The cloud / edge GPU high-order inference service module is used for deploying the domain-specific model on a cloud or a lightweight edge GPU node. ​ ​ ​ ​ ​ ​ The field-specific model is used to perform high-order reasoning based on low-dimensional structured data and matching data, obtain a reasoning result, and transmit the reasoning result to an intelligent evaluation optimization module and a terminal device; the matching data is a result of semantic matching of the low-dimensional structured data with the structured world model and industry knowledge stored in a dynamic memory library module.

7. The vertically field-oriented structured miniaturized macro model deployment method of claim 6, wherein, The method further includes: deploying the intelligent evaluation optimization module on a cloud edge processor; generating evaluation indexes according to the reasoning result and corresponding real data by using the intelligent evaluation optimization module; and optimizing a reasoning link of the cloud edge GPU high-order reasoning service module according to the evaluation indexes.

8. The vertically field-oriented structured miniaturized macro model deployment method of claim 6, wherein, The method further includes: optimizing resource scheduling of the cloud edge GPU high-order reasoning service module by using the intelligent evaluation optimization module, and dynamically allocating GPU resources according to real-time loads to perform a reasoning process according to a task priority.

9. The vertically field-oriented structured miniaturized macro model deployment method of claim 6, wherein, The method further includes: adjusting model parameters of a lightweight model in a terminal device CPU according to the evaluation indexes by using the intelligent evaluation optimization module.

10. The vertically field-oriented structured miniaturized macro model deployment method of claim 6, wherein, The method further includes: storing the reasoning result output by the field-specific model into the structured world model and the dynamic memory library module by using the intelligent evaluation optimization module.

Images