Pre-training model engineering delivery capability grading evaluation system

Abstract

The invention discloses a pre-training model engineering delivery capability grading evaluation system comprising an acquisition module used for acquiring various data information of a to-be-evaluated provider; a processing module; and a comparison module. According to the pre-training model engineering delivery capability grading evaluation system, related data are acquired from six aspects of enterprise basic capability, model consultation capability, data engineering capability, model delivery capability, model application capability and model operation capability through an acquisition module, that is, the system is evaluated from the six aspects; the method comprises the following steps: automatically evaluating an evaluation result according to a contrast sample to obtain a grade, and finally carrying out numerical value output on the provider capability of which the grade is evaluated, namely, outputting the grade in a digital form according to a set score weight to display and carrying the reason of a main score point, namely, carrying out refined processing on the grade of the evaluation result. And the intuition of the evaluation result is further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of evaluation systems, and in particular to a pre-training model engineering delivery capability grading evaluation system. Background Art

[0002] A pre-trained model is a deep learning model that is trained on a wide range of data sets and fine-tuned with specialized field data to meet scenario task requirements. Pre-trained model delivery cloud management service refers to a technical service that provides enterprises with end-to-end model delivery solutions based on a cloud computing environment. The technical service capabilities of pre-trained model delivery cloud management service providers include six major capability areas: enterprise basic capabilities, model consulting capabilities, data engineering capabilities, model delivery capabilities, intelligent body building capabilities, and model operation capabilities. In order to improve and improve the technical service level of pre-trained model delivery cloud management service providers, an evaluation system needs to be used to evaluate them. Summary of the invention

[0003] The purpose of the present invention is to provide a pre-trained model engineering delivery capability grading assessment system to enhance and improve the technical service level of pre-trained model delivery cloud management service providers.

[0004] To achieve the above object, the present invention provides the following technical solution: a pre-training model engineering delivery capability grading evaluation system, comprising:

[0005] A collection module, which is used to collect various data information of the provider to be evaluated, including enterprise basic capabilities, model consulting capabilities, data engineering capabilities, model delivery capabilities, model application capabilities, and model operation capabilities;

[0006] A processing module, the processing module is used to classify and store the data collected by the collection module, and to back up the classified data, and to add a timestamp to the backed up data;

[0007] A comparison module, which uses the rating standards preset in the sample library as a template to compare and analyze the classified data to determine its grade;

[0008] A numerical output module outputs and displays the assessed provider capabilities in the form of specific numbers based on preset scoring grade assignment weights, and the weight assignment is adjusted and set according to needs.

[0009] Preferably, the basic capabilities of the enterprise include enterprise qualifications, personnel skills, management system, financial status and case analysis.

[0010] Preferably, the model consulting capability includes demand assessment, resource assessment, business assessment and feasibility assessment, and the model consulting capability refers to the cloud management service provider providing customers with resource, business and feasibility demand assessment and model scenario, selection, data and resource planning solution design based on the model delivery project.

[0011] Preferably, the data engineering capabilities include training data management and knowledge base management, and training data management includes data collection, data cleaning, data labeling and data conversion, while knowledge base management includes knowledge base construction and knowledge base docking.

[0012] Preferably, the model delivery capability includes model adaptation, model training, model tuning and model verification, and model adaptation includes environment configuration, POC side view and model deployment, while model training includes training and push platform and reasoning scenario, model tuning includes parameter adjustment, reinforcement learning and prompt word construction, and the model verification includes performance verification, experience verification, reliability verification and stability verification.

[0013] Preferably, the model application capabilities include intelligent agent construction and RAG application, and the intelligent agent construction includes task decomposition, workflow orchestration, template library construction and intelligent agent collaboration, while the RAG application includes atomic capability assessment, secondary development capabilities and RAG workflow orchestration.

[0014] Preferably, the model operation capability includes operation and maintenance services, operation and maintenance center, model application security and continuous service improvement, and the operation and maintenance services include model hosting, model monitoring, fault handling, time response and training services; the operation center includes service orchestration and metering and billing; the model application security includes model image security, model itself security, integrated application security and vulnerability protection detection; the continuous service improvement includes effect evaluation and user feedback.

[0015] Preferably, the graded evaluation derived from the comparison module is divided into five levels from low to high, namely, exploration level, practice level, excellent level, outstanding level and leadership level.

[0016] Preferably, the exploratory-level service capability represents that the pre-trained model delivery cloud management service provider has a preliminary scale of personnel reserves and can provide basic consulting, delivery and other capabilities; the practical-level service capability represents that the pre-trained model delivery cloud management service provider has a certain scale of personnel reserves and case experience, can provide basic consulting, data engineering, model delivery and operation, model application and other services, and has certain tool capabilities; the excellent-level service capability represents that the pre-trained model delivery cloud management service provider has a certain scale of personnel reserves and case experience, provides relatively complete consulting, data engineering, model delivery and operation, model application and other services, and has certain tools and processes; the superior-level service capability represents The pre-trained model delivery cloud management service provider has a complete personnel reserve, and its cases cover multiple industries and multiple enterprises. It can provide complete consulting, data engineering, model delivery and operation, model application and other services, provide advanced technical support, and has a complete service process, a complete tool chain and platform; the leading service capability represents the pre-trained model delivery cloud management service provider has a complete personnel reserve, case experience and innovation capabilities, and provides expert consulting and strategic planning services in the field of artificial intelligence; provides highly customized model delivery and deployment services; provides intelligent data management, model deployment, operation and maintenance monitoring and operation management and other related tools, demonstrating the industry-leading model delivery technical service capabilities.

[0017] Preferably, the data classified by the processing module is backed up to the cloud, and the processing module completes the classification of the data through a classification algorithm, and the classification algorithm formula is:

[0018]

[0019] Where xi is the sample data, xj is the data to be classified, d(xi,xj) is the classification result. At the same time, for data xi and xj, if their feature space is an n-dimensional real vector space xi=(xi1,xi2,xi3...xin), xj=(xj1,xj2,xj3...xjn).

[0020] Compared with the prior art, the invention has the following beneficial effects: the pre-trained model engineering delivery capability grading evaluation system,

[0021] The present invention collects relevant data from six aspects, namely, enterprise basic capabilities, model consulting capabilities, data engineering capabilities, model delivery capabilities, model application capabilities and model operation capabilities, through a collection module, that is, the system is evaluated from the above six aspects, and the evaluation results are automatically evaluated according to the control samples. The provider capabilities with the evaluated levels are finally output numerically, that is, the levels are output in digital form according to the set score weights and the reasons for the main scoring points are attached, that is, the levels of the evaluation results are refined, and the intuitiveness of the evaluation results is further improved. In addition, the staff can view and verify the collected original data through the timestamps classified and backed up by the processing module, and thereby ensure the accuracy of the evaluation results, ensure that the evaluation of the system has a basis, and thereby improve the objectivity of the evaluation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The system control process of the present invention;

[0023] Figure 2 A schematic diagram of the project composition collected by the collection module of the present invention;

[0024] Figure 3 A schematic diagram of the consulting capability of the model of the present invention;

[0025] Figure 4 It is a schematic diagram of the data engineering capability of the present invention;

[0026] Figure 5 It is a schematic diagram of the delivery capability composition of the model of the present invention;

[0027] Figure 6 It is a schematic diagram of the application capability of the model of the present invention;

[0028] Figure 7 This is a schematic diagram of the operational capabilities of the model of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] See also Figure 1-7 , the present invention provides a technical solution: a pre-training model engineering delivery capability grading evaluation system,

[0031] The collection module is used to collect various data information of the providers to be evaluated. The data information types include enterprise basic capabilities, model consulting capabilities, data engineering capabilities, model delivery capabilities, model application capabilities and model operation capabilities;

[0032] The processing module is used to classify and store the data collected by the collection module, back up the classified data, and add a timestamp to the backed up data;

[0033] Comparison module: The comparison module uses the preset rating standards in the sample library as a template to compare and analyze the classified data to determine its grade;

[0034] Numerical output module: The numerical output module outputs and displays the evaluated provider capabilities in the form of specific numbers based on the preset scoring grade weights. At the same time, the weight assignment is adjusted and set according to the needs.

[0035] Basic enterprise capabilities include enterprise qualifications, personnel skills, management system, financial status and case analysis. Personnel skills refer to the practitioners of pre-trained model delivery cloud management service providers having AI-related technical certifications, including the AIGC skill certificate of the Ministry of Industry and Information Technology (ICDC) and the American Artificial Intelligence Association. Certified Artificial Intelligence Engineer (CAIE TM )China Artificial Intelligence Association (CAAI) Artificial Intelligence Occupational Skills Certification (AIOC) AI prompt word engineer; management system refers to the corporate organizational system, corporate management system and personnel training system of the pre-trained model delivery service provider; financial status refers to the source and division status of the service provider's operating funds in a certain period of time; case experience refers to the accumulation of case libraries for pre-trained model delivery service providers to provide large model cloud management services to customers.

[0036] The basic capabilities of enterprises are classified as follows:

[0037] Exploration level: The company has a strategic cooperation framework, agreement or cooperation certification with cloud service providers and model manufacturers; the staff has relevant technical certifications such as cloud computing, artificial intelligence, data storage, etc., and the certification ratio exceeds 20%; the company has a quality management system; the company has a financial audit system; and has at least one pre-trained model delivery cloud management service project case;

[0038] Practice level: The company has a strategic cooperation framework, agreement or cooperation certification with cloud service providers and model manufacturers, and the number of service providers is not less than 2; the staff has relevant technical certifications in cloud computing, artificial intelligence, data storage, chips, operating systems, etc., with a certification ratio of more than 30%, and more than 10% of the certified experts in the AI ​​field; the company has obtained relevant quality management system certification; the company has a financial audit report for the past year; and has at least 2 pre-trained model delivery cloud management service project cases;

[0039] Excellent: The company has a strategic cooperation framework, agreement or cooperation certification with cloud service providers and large model manufacturers, and the number of service providers is not less than 3; the staff has relevant technical certifications in cloud computing, artificial intelligence, data storage, chips, operating systems, etc., the certification ratio exceeds 40%, and the number of certified experts in the AI ​​field exceeds 10%; the company has obtained relevant quality management system certification, and the company has a training plan and relevant supporting materials; the company has financial audit reports for the past two years and is in good financial condition; it has at least 3 pre-trained model delivery cloud management service project cases;

[0040] Excellent level: The company has a strategic cooperation framework, agreement or cooperation certification with cloud service providers and large model manufacturers, and the number of service providers is not less than 5; the staff has relevant technical certifications in cloud computing, artificial intelligence, data storage, chips, operating systems, etc., the certification ratio exceeds 40%, and certified experts in the AI ​​field exceed 15%; the company has obtained relevant quality management system certification and has security management certification; the company has training plans and related supporting materials; the company has financial audit reports for the past three years and is in good financial condition; it has at least 5 pre-trained model delivery cloud management service project cases, covering multiple industries and enterprise types;

[0041] Leading level: The company has a strategic cooperation framework, agreement or cooperation certification with cloud service providers and large model manufacturers, and the number of service providers is not less than 8; the staff has relevant technical certifications in cloud computing, artificial intelligence, data storage, chips, operating systems, etc., the certification ratio exceeds 50%, and the number of certified experts in the AI ​​field exceeds 20%; the company has obtained relevant quality management system certification and has certification in security management; the company has a training plan and related supporting materials, clearly defines the training needs, requirements, goals and achievable training results; provides personnel with independent learning channels; the company has financial audit reports for the past three years and is in excellent financial condition; it has at least 8 pre-trained model delivery cloud management service project cases, covering three industries, and covering large, medium and small enterprises;

[0042] Requirements analysis capability classification:

[0043] Exploration level: Analyze customer resources and business environment, and provide basic analysis reports;

[0044] Practice level: Analyze the customer's resource situation and provide basic analysis reports, including data, computing power, tools, platform, cloud environment, etc. Analyze the customer's business environment and provide basic analysis reports, including business application layer, model management layer, platform layer, data layer, etc.

[0045] Excellent: Analyze the customer's resource situation and provide a relatively complete analysis report, including data, computing power, tools, platform, cloud environment, etc.; analyze the customer's business environment and provide a relatively complete analysis report, including the business application layer, model management layer, platform layer, data layer, etc.; conduct feasibility analysis on customer solutions and provide basic analysis reports;

[0046] Excellence: Analyze the customer's resource situation and provide a complete analysis report, including data, computing power, tools, platform, cloud environment, etc.; analyze the customer's business environment and provide a complete analysis report, including the business application layer, model management layer, platform layer, data layer, etc.; conduct a feasibility analysis on the customer's solution and provide a relatively complete analysis report, including technical feasibility, economic benefits, regulatory compliance and implementation timetable, etc.;

[0047] Leading level: Analyze customer resource conditions and provide comprehensive analysis reports, including data, computing power, tools, platforms, cloud environments, etc.; analyze customer business environments and provide comprehensive analysis reports, including business application layer, model management layer, platform layer, data layer, etc.; conduct feasibility analysis on customer solutions and provide comprehensive analysis reports, including technical feasibility, economic benefits, regulatory compliance, and implementation schedule, etc.; support the use of emerging technologies such as multi-process and multi-tool to conduct feasibility analysis on customer solutions and provide comprehensive analysis reports;

[0048] Program planning ability classification:

[0049] Exploration level: Provide basic solution design based on business scenarios, customer needs and infrastructure conditions;

[0050] Practice level: Provide basic solution design based on business scenarios, customer needs and infrastructure conditions, such as scenario planning, model planning, data planning or resource planning;

[0051] Excellent: Based on business scenarios, customer needs and infrastructure conditions, provide a relatively complete solution design, such as scenario planning, model planning, data planning, and resource planning, and must cover two or more;

[0052] Excellent: Based on business scenarios, customer needs and infrastructure conditions, a comprehensive solution design is provided, such as scenario planning, model planning, data planning and resource planning, which must cover two or more. The planning solution has a certain process standardization and can be applied to large, medium and small enterprises.

[0053] Leading level: Based on business scenarios, customer needs and infrastructure conditions, provide comprehensive solution design, such as scenario planning, model planning, data planning and resource planning, and must cover two or more; the solution design has a certain process standardization and can be applied to large / medium / small enterprises; the solution design has a certain industry adaptability and can be applied to multiple fields such as finance, education, transportation, industry, and medical care;

[0054] Model consulting capabilities include demand assessment, resource assessment, business assessment and feasibility assessment, and model consulting capabilities refer to the cloud management service provider providing customers with resource, business and feasibility demand assessments as well as model scenario, selection, data and resource planning solution designs based on model delivery projects.

[0055] Data engineering capabilities include training data management and knowledge base management, and training data management includes data collection, data cleaning, data labeling and data conversion. At the same time, knowledge base management includes knowledge base construction and knowledge base docking.

[0056] Data Engineering Service Capability Classification:

[0057] Exploration level: Possess basic data collection, cleaning, data annotation and data conversion methods;

[0058] Practical level: Have basic data collection, cleaning, data annotation and data conversion methods; use a variety of semi-automatic / automatic tools for data cleaning, data annotation and data conversion, such as web crawlers, vectorization tools, standardization tools, etc.;

[0059] Excellent: Have relatively complete data collection, cleaning, data labeling and data conversion methods; use a variety of semi-automatic / automatic tools for data cleaning, data labeling and data conversion, such as web crawlers, vectorization tools, standardization tools, etc.; use a variety of advanced technical methods / strategies for data cleaning, data labeling and data conversion, such as scenario-based data collection, multimodal data labeling, rule-based data cleaning, etc.;

[0060] Excellent: Have complete data collection, cleaning, data annotation and data conversion methods; use a variety of semi-automated / automated tools for data cleaning, data annotation and data conversion, such as web crawlers, vectorization tools, standardization tools, etc.; use a variety of advanced technical methods / strategies for data cleaning, data annotation and data conversion, such as scenario-based data collection, multimodal data annotation, rule-based data cleaning, etc.; have a professional data development team and standardized process processing mechanism for data cleaning, data annotation and data conversion

[0061] Leading level: Have complete data collection, cleaning, data labeling and data conversion methods; use a variety of semi-automated / automated tools for data cleaning, data labeling and data conversion, such as web crawlers, vectorization tools, standardization tools, etc.; use a variety of advanced technical methods / strategies for data cleaning, data labeling and data conversion, such as scenario-based data collection, multimodal data labeling, rule-based data cleaning, etc.; have a professional data development team and standardized process processing mechanism for data cleaning, data labeling and data conversion; use intelligent AI algorithms to develop data processing platforms for data cleaning, data labeling and data conversion, and have the ability to continuously optimize and iterate;

[0062] Knowledge base construction service capability classification:

[0063] Exploration level: It has basic knowledge base management methods and supports manual entry;

[0064] Practical level: Have basic knowledge base management methods, including knowledge base construction and knowledge base docking; use a variety of semi-automated / automated tools to build knowledge bases, including data storage, synchronization and transmission, as well as knowledge retrieval, update and maintenance mechanisms;

[0065] Excellent: Have a relatively complete knowledge base management method, including knowledge base construction, knowledge base docking, etc.; use a variety of semi-automatic / automatic tools to build the knowledge base, including data storage, synchronization and transmission, as well as knowledge retrieval, update and maintenance mechanisms, etc.; have a standardized knowledge base docking process, including interface specification definition, data exchange and processing mechanism, system integration method, etc.;

[0066] Excellent: Possessing a complete knowledge base management method, including knowledge base construction and knowledge base docking; using a variety of semi-automated / automated tools to build the knowledge base, including data storage, synchronization and transmission, as well as knowledge retrieval, update and maintenance mechanisms; having a standardized knowledge base docking process, including interface specification definition, data exchange and processing mechanism, system integration method, etc.; having advanced reasoning ability, and being able to independently discover new knowledge and add it to the knowledge base;

[0067] Leading level: Possessing a complete knowledge base management method, including knowledge base construction and knowledge base docking; using a variety of semi-automated / automated tools to build the knowledge base, including data storage, synchronization and transmission, as well as knowledge retrieval, update and maintenance mechanisms; having a standardized knowledge base docking process, including interface specification definition, data exchange and processing mechanism, system integration method, etc.; having advanced reasoning ability, being able to independently discover new knowledge and add it to the knowledge base; achieving cross-domain knowledge integration, being able to automatically expand the knowledge base docking capability according to application requirements, and adapting to changing business scenarios;

[0068] Model delivery capabilities include model adaptation, model training, model tuning and model verification. Model adaptation includes environment configuration, POC side view and model deployment. At the same time, model training includes training and push platform and reasoning scenario. Model tuning includes parameter adjustment, reinforcement learning and prompt word construction. Model verification includes performance verification, experience verification, reliability verification and stability verification.

[0069] Model adaptation service capability classification:

[0070] Exploration level: provides basic environment configuration, model deployment methods, and model adaptation;

[0071] Practice level: Provide basic environment configuration, model deployment methods, and model adaptation; provide more complete model deployment methods, such as public cloud, private cloud, or physical machine;

[0072] Excellent: Provides a relatively complete environment configuration method, including AI computing unit and server resource preparation, model compatibility, etc.; provides a relatively complete model deployment method, such as public cloud or private cloud or physical machine; provides a relatively complete POC test method, including model scenario design, implementation verification, effect analysis, etc.;

[0073] Excellent: Provides a complete environment configuration method, including AI computing unit and server resource preparation, model compatibility, etc.; provides a complete model deployment method, such as public cloud, private cloud, hybrid cloud, physical machine and other different modes, and must support two or more; provides a complete POC test method, including model scenario design, implementation verification, effect analysis, etc.; has standardized environment configuration, model deployment, POC test process, which has been formally implemented within the team;

[0074] Leading level: Provide a complete environment configuration method, including AI computing unit and server resource preparation, model compatibility, etc.; provide a complete model deployment method, such as public cloud, private cloud, hybrid cloud, physical machine and other modes, and must support two or more; provide a complete POC testing method, including model scenario design, implementation verification, effect analysis, etc.; have standardized environment configuration, model deployment, and POC testing processes, which have been formally implemented within the team; use multiple tools / platforms / systems for environment configuration, model deployment, POC testing, and adapt to multiple application scenarios.

[0075] Model training and push service capability classification:

[0076] Exploration level: It has a basic model training and promotion platform and can perform simple model reasoning;

[0077] Practical level: It has a basic model training platform, including data preprocessing tools, model training framework, optimizer, hardware accelerator, inference engine and other components; it can perform model inference in a specific scenario, such as semantics, speech, image, etc.

[0078] Excellent: It has a relatively complete model training platform, including data preprocessing tools, model training framework, optimizer, hardware accelerator, and inference engine components; it can perform model inference in a specific scenario, such as semantics, speech, and images; the model inference speed is relatively fast, with a delay of about 1 second;

[0079] Excellent: Equipped with a complete model training platform, including data preprocessing tools, model training framework, optimizer, hardware accelerator, and inference engine components; capable of performing model inference in a specific scenario, such as semantics, speech, and images; fast model inference speed, with a latency of around 100 milliseconds; capable of handling medium-scale / complex model training tasks;

[0080] Leading level: It has a complete model training platform, including data preprocessing tools, model training framework, optimizer, hardware accelerator and inference engine components; it can perform model inference in a specific scenario, such as semantics, voice, image, etc.; the model inference speed is fast, with a delay of about 50 milliseconds; it can handle large-scale / complex model training tasks; the model training and inference platform has autonomous learning and optimization capabilities, and can support large-scale customized projects;

[0081] Model tuning service capability classification:

[0082] Exploration level: Provides basic parameter adjustment and reinforcement learning tools, methods, and strategies for model tuning;

[0083] Practical level: Provides basic parameter adjustment and reinforcement learning tools, methods and strategies for model tuning; supports fine-tuning using common fine-tuning algorithms, such as Prompt-tuning, P-tuning, Prefix-tuning, Adapter-tuning, LoRA, Delta-tuning, Fullfine-tuning, etc.

[0084] Excellent: Provides relatively complete parameter adjustment and reinforcement learning tools, methods and strategies for model tuning; supports fine-tuning using common fine-tuning algorithms, such as Prompt-tuning, P-tuning, Prefix-tuning, Adapter-tuning, LoRA, Delta-tuning, Fullfine-tuning, etc.; provides relatively complete reinforcement learning tools, methods and strategies, such as Ray framework, DeepSpeed-Chat framework, policy gradient reinforcement learning algorithm, proximal policy optimization algorithm, etc.;

[0085] Excellent level: Provides complete parameter adjustment and reinforcement learning tools, methods and strategies for model tuning; supports fine-tuning using common fine-tuning algorithms, such as Prompt-tuning, P-tuning, Prefix-tuning, Adapter-tuning, LoRA, Delta-tuning, Fullfine-tuning, etc. Provides complete reinforcement learning tools, methods and strategies, such as Ray framework, DeepSpeed-Chat framework, policy gradient reinforcement learning algorithm, proximal policy optimization algorithm, etc. Provides visualization tools for model parameter adjustment, which facilitates users to view and manage various parameters and results in the fine-tuning process Supports the automated execution capability of reinforcement learning pipelines, and provides iterative fine-tuning to improve model effects;

[0086] Leading level: Provides complete parameter adjustment and reinforcement learning tools, methods and strategies for model tuning; supports fine-tuning using common fine-tuning algorithms, such as Prompt-tuning, P-tuning, Prefix-tuning, Adapter-tuning, LoRA, Delta-tuning, Fullfine-tuning, etc.; provides complete reinforcement learning tools, methods and strategies, such as Ray framework, DeepSpeed-Chat framework, policy gradient reinforcement learning algorithm, proximal policy optimization algorithm, etc.; provides visualization tools for model parameter adjustment, so that users can view and manage various parameters and results during the fine-tuning process; supports the automated execution of reinforcement learning pipelines, and provides iterative fine-tuning to improve model effects; has a visual expert interface that can display the answers output by the model, fill in better answers, and evaluate the effects of answers, etc.

[0087] Model validation service capability classification:

[0088] Exploration level: Provides basic model verification methods for models;

[0089] Practical level: Provide basic model verification methods for the tuned models, such as performance verification, experience verification, reliability verification, or stability verification;

[0090] Excellent: For the tuned models, a relatively complete model verification method is provided, such as performance verification, experience verification, reliability verification, and stability verification, and two or more verification methods must be supported;

[0091] Excellent: Provides complete model verification methods for the tuned models, such as performance verification, experience verification, reliability verification, and stability verification methods, and must support two or more methods; can complete model verification within the specified time, and one verification method takes no more than half a day;

[0092] Leading level: Provide comprehensive model verification methods for the tuned models, such as performance verification, experience verification, reliability verification, and stability verification methods, and support two or more methods; be able to complete model verification within the specified time, and one verification method takes no more than half a day (to be discussed); be able to issue third-party model test verification reports, with no less than 2 companies;

[0093] Model application capabilities include intelligent agent construction and RAG application. Intelligent agent construction includes task decomposition, workflow orchestration, template library construction and intelligent agent collaboration. At the same time, RAG application includes atomic capability assessment, secondary development capabilities and RAG workflow orchestration.

[0094] Intelligent agent construction service capability classification:

[0095] Exploration level: does not have the ability to build intelligent agents;

[0096] Practice level: Development projects with intelligent agents, but the product is in the testing stage and has not yet been released;

[0097] Excellent: At least one agent has been released and has the following basic capabilities: In terms of task decomposition capability, the overall task can be decomposed into several small tasks, each of which has a clear goal and output, and the dependencies between each small task are clear and can be completed independently; in terms of workflow arrangement capability, it can build a workflow containing several workflow nodes and linear, branching and loop structures to ensure the correctness of the workflow logic;

[0098] Excellent: At least 3 agents have been released, and they have the following relatively complete capabilities: In terms of task decomposition capability, they can decompose the overall task into several small tasks, each of which has a clear goal and output, and can provide the specific steps, required resources and expected results for the execution of the small tasks, and ensure that the logical order between tasks is reasonable; in terms of workflow orchestration capability, they can not only build workflows with simple structures, but also build workflows with multiple complex structures (such as parallel processing, conditional branches, exception handling, etc.), and realize task scheduling and execution; in terms of template library construction capability, they can create several templates, and the template library has version management and update functions; in terms of agent collaboration capability, they can realize basic information exchange and task allocation between two agents;

[0099] Leading level: It has multiple intelligent products in multiple industries and possesses the following complete capabilities: in terms of task decomposition capability, it has established a complete task decomposition and optimization mechanism, prioritized each small task, and was able to dynamically adjust according to the task execution situation; in terms of workflow orchestration capability, it can not only build workflows with simple structures, but also build workflows containing a variety of complex structures (such as parallel processing, conditional branches, exception handling, etc.), and realize task scheduling and execution; in terms of workflow orchestration capability, it has established a complete workflow orchestration and optimization system to realize efficient and automated execution of workflows; in terms of template library construction capability, it can create several templates, and the template library has version management and update functions; in terms of template library construction capability, it has established a complete template library management and optimization mechanism to realize intelligent recommendation and automatic update of templates; in terms of intelligent agent collaboration capability, it can realize information exchange and task allocation between multiple intelligent agents, and ensure the accuracy of information transmission and balanced distribution of tasks;

[0100] RAG application service capability classification:

[0101] Exploration level: no RAG application;

[0102] Practice level: Development projects with RAG applications, but the products are in the testing phase and have not yet been released;

[0103] Excellent: At least one RAG application has been released and has the following basic capabilities: Able to process and encode documents or data fragments in text format, and the supported document types include at least plain text and simple HTML format; the encoded vector representation can basically reflect the semantic information of the document; able to retrieve relevant documents from the index database based on the customer's text query, support simple Boolean query and keyword matching, and sort the search results by relevance; able to generate simple text answers or summaries based on the retrieved relevant documents, the generated content is grammatically correct, without obvious errors, and the generated content basically covers the main information of the retrieved documents;

[0104] Excellent: At least one RAG application has been released, and it has the following relatively complete capabilities: Supported document types include not only simple plain text, but also common document formats such as PDF, DOC, and complex format documents such as PPT and XLS; Support detailed feature extraction of images in documents, including color, shape, texture, etc., and the encoded vector representation can capture subtle semantic differences in documents; Support more complex query syntax, such as fuzzy query, range query, etc., to achieve semantic query, that is, it can understand the intent of the query and retrieve semantically related documents, and the search results support multiple sorting methods, such as by time, by popularity, etc.; Support the generation of various types of text, such as news reports, product descriptions, etc., and the generated content is more natural and fluent in grammar and expression;

[0105] Leading level: At least one RAG application has been released, and it has the following complete capabilities: it can process and encode multimedia content (such as audio and video), support feature extraction of complex elements such as tables and charts in documents, and the encoded vector representation performs well in cross-modal retrieval; it supports cross-modal retrieval, that is, it can retrieve relevant documents based on multimedia content such as pictures and audio, and realize real-time retrieval, that is, it can give retrieval results while the user enters the query, and the retrieval results support personalized recommendations, that is, optimize the result order according to the user's historical behavior; it can generate personalized customization of content, and can optimize the generated results according to user preferences and historical behavior, support the generation of multimedia content such as pictures, audio, video, etc. to assist text expression, support the generation of interactive content such as question-and-answer dialogues, virtual characters, etc. to enhance the user experience, and the generated content is more accurate and rich in semantics;

[0106] Model operation capabilities include operation and maintenance services, operation and maintenance centers, model application security, and continuous service improvement. Operation and maintenance services include model hosting, model monitoring, fault handling, time response, and training services; the operation center includes service orchestration and metering and billing; model application security includes model image security, model security, integrated application security, and vulnerability protection detection; continuous service improvement includes effect evaluation and user feedback. Operation and maintenance management service capability classification:

[0107] Exploration level: It does not have a basic model hosting cloud platform, and can only deploy the model in a physical server. The computing, storage, and network resources required by the model cannot be flexibly allocated; it does not have the ability to monitor the model; it can detect and record model failures, and the failure response time does not exceed 4 hours; it cannot take temporary measures for failures, and there is no record of the failure handling process; it does not have an event library, and can only record some key events; it does not have the ability to provide training services;

[0108] Practice level: It has a basic public cloud model hosting platform, and uses the basic computing, storage, and network resources provided by the public cloud to host the model; it has basic model monitoring capabilities, and can monitor whether the model interface call is successful and the CPU and GPU operation status, but cannot monitor in real time; the fault response time is shortened to less than 2 hours, and the fault can be divided into 5 levels according to the severity of the fault; the fault handling process is simply recorded without forming detailed documents; it has a basic event library that can record all events that occur; it provides basic model operation manuals and emergency handling process documents, including basic information such as product overview, usage steps, and safety warnings;

[0109] Excellent: With complete public cloud model hosting, automatic expansion, fault tolerance and disaster recovery mechanisms are added to the basic cloud services, and operation and maintenance monitoring and alarm functions are provided; with relatively complete model monitoring capabilities, it can monitor in real time whether the model interface call is successful and the CPU and GPU operation status; the fault response time is shortened to less than 1 hour; it can take temporary measures for faults, such as replacing spare models, implementing repairs, verifying tests, and feeding back processing results; it can record the fault handling process in detail and form a fault handling report; it has a relatively complete event library, which can not only record all the events that occur, but also record the key steps of the event handling process, and basically classify the events into at least three levels: high, medium and low; it provides a detailed model operation manual and emergency handling process video tutorial; it conducts at least one customer team training, and the training time is not less than 2 hours;

[0110] Excellent level: It has a complete private cloud model hosting platform, supports customization, and provides customized private cloud solutions according to customer needs, including dedicated hardware, customized software environment, etc.; it has complete model monitoring capabilities, which can not only monitor the operating parameters of the model in real time, but also push abnormal information on key parameters in real time; it realizes fault self-healing capabilities, and 70% of faults can be automatically repaired without human intervention; it conducts in-depth analysis of the fault handling process, refines fault modes and impact analysis; it prevents common faults, formulates and implements preventive measures; it has a complete event library, which can provide a complete record of the event handling process, including each step of the operation and results; it can retrieve events by name, product, time and other dimensions; it can automatically notify relevant personnel when an event occurs, and distribute and circulate according to preset rules; it provides detailed model operation manuals and emergency handling process video tutorials; it conducts at least three customer team trainings, each training lasting no less than 4 hours;

[0111] Leading level: A hybrid cloud model hosting platform combines the advantages of public and private clouds to provide flexible, scalable and secure model hosting solutions; establish a complete model supervision mechanism to monitor CPU and GPU usage, resources, and model image status in real time, push abnormal information in real time, automatically restart abnormal services, automatically roll back to the last stable version, and record detailed logs; implement highly automated fault handling and recovery mechanisms, with almost zero fault response time; optimize the fault handling process to form a best practice library for fault handling; establish an intelligent event response mechanism to realize intelligent event analysis, provide and execute instant solutions, and realize instant response and handling of events without manual intervention; establish a complete training service system;

[0112] Operation center capability classification:

[0113] Exploration level: does not have the ability to orchestrate services; can measure computing resource usage with an error of no more than 10%; accurately measure the number of service calls with an error of no more than 5%; accurately measure the amount of data processed with an error of no more than 10%

[0114] Practice level: Possess the ability to orchestrate services; be able to manage at least 2 model services and perform basic scheduling and combination. The number of model services working together does not exceed 2; the measurement error of computing resource usage is reduced to within 5%; the measurement error of service call times is reduced to within 2%; the measurement error of data processing volume is reduced to within 5%;

[0115] Excellent: With relatively complete service orchestration capabilities, it can manage at least 3 model services and perform efficient scheduling and combination; the number of collaborative model services can reach 5; with relatively complete metering and billing capabilities, it can achieve real-time metering of computing resource usage, with an error control within 2%; real-time metering of service call times, with an error control within 1%; real-time metering of data processing volume, with an error control within 2%

[0116] Excellent: With complete service orchestration capabilities, it can manage at least x10200 model services and realize intelligent and automated scheduling and combination; the number of model services working together can reach 15; high-precision computing resource usage measurement is achieved, with the error controlled within 1%; high-precision real-time measurement of service call times, with the error controlled within 0.5%; high-precision real-time measurement of data processing volume, with the error controlled within 1%

[0117] Leading level: Establish a complete service orchestration mechanism; realize intelligent scheduling and combination of any number of model services; the number of model services working together is unlimited, and can support large-scale complex business processes; the business process combination time is shortened to less than 5 minutes, and instant response is achieved; a set of accurate metering and billing mechanisms are recommended; ultra-high-precision computing resource usage measurement is achieved, with almost zero error; the number of service calls is measured in real time without error; the data processing volume is measured in real time without error, and complex data processing scenarios are supported;

[0118] Model application security service capability classification:

[0119] Exploration level: image scanning is performed at least once a month; known vulnerabilities are fixed within 30 days after discovery; data sharing audits are performed at least once a quarter;

[0120] Practice level: Have relatively complete model application security capabilities; complete repairs within 7 days after the known vulnerabilities are discovered; conduct data sharing audits at least once a month; and conduct at least one security test for each new interface before going online;

[0121] Excellent: Possess complete model application security capabilities; perform image scanning at least once a day; further shorten the time to fix known vulnerabilities, and establish clear vulnerability reporting channels to ensure that new vulnerabilities found can be recorded within 24 hours; perform data sharing audits at least once a week; and perform at least two security tests on each new interface before going online;

[0122] Excellent: With more complete model application security capabilities, and some functions are automated: any new image must be scanned before deployment, and known vulnerabilities are repaired with automated tools; an automated vulnerability scanning and reporting system is established; any data sharing activities are monitored and audited in real time; each new interface is automatically tested for security before going online; the system is penetrated and tested, covering at least 80% of the main functions of the system;

[0123] Leading level: Introducing advanced code analysis tools, such as AI-based code auditing; being able to share vulnerability information in real time to ensure that vulnerabilities are prevented before they are discovered; combining AI technology to conduct automated and intelligent penetration testing, covering all system functions;

[0124] Service continuous improvement capability classification:

[0125] Exploration level: no service improvement capabilities;

[0126] Practice level: Have basic service improvement capabilities; conduct effect evaluation regularly, at least once a year; the evaluation content covers basic aspects of model application effect, operation and maintenance effect, and service quality; have user feedback channels, such as customer service email or online forms;

[0127] Excellent: Possess relatively complete service improvement capabilities; conduct effect evaluation every six months to ensure timely discovery and resolution of problems; the evaluation content is more detailed, including model stability, user satisfaction, operation and maintenance efficiency, etc., but there are no quantitative indicators to measure the effect; set up a variety of user feedback channels, including social media, online customer service, etc.; collect and analyze user feedback every six months and form a report, and the feedback response time is within 12 hours;

[0128] Excellent: Possessing perfect service improvement capabilities; conducting effect evaluation once a quarter to ensure continuous improvement and optimization; comprehensive evaluation content, providing multiple quantitative indicators for effect measurement, and detailed data analysis support; perfect user feedback channels, including real-time online chat, user community, etc.; collecting and analyzing user feedback once a quarter, with detailed improvement measures, and feedback response time within 6 hours; the proportion of solving user feedback problems is stable at more than 90%;

[0129] Leading level: Establish a complete service improvement mechanism: conduct an effectiveness evaluation once a month to ensure real-time understanding of the model operation and maintenance status; the evaluation content not only covers all key performance indicators, but also includes advanced analysis such as user behavior analysis and market trend forecasting; the user feedback channel is fully automated and intelligent, and can capture and analyze customer emotions and needs in real time; user feedback is collected and analyzed in real time to form an immediate improvement strategy; the proportion of user feedback issues resolved exceeds 95%, and there is a prevention mechanism to prevent similar problems from happening again.

[0130] The graded assessment results of the comparison module are divided into five levels from low to high, namely exploration level, practice level, excellent level, outstanding level and leadership level.

[0131] Exploration-level service capabilities represent that the pre-training model delivery cloud management service provider has a preliminary scale of personnel reserves and can provide basic consulting and delivery capabilities; practice-level service capabilities represent that the pre-training model delivery cloud management service provider has a certain scale of personnel reserves and case experience, can provide basic consulting, data engineering, model delivery and operation, model application and other services, and has certain tool capabilities; excellent-level service capabilities represent that the pre-training model delivery cloud management service provider has a certain scale of personnel reserves and case experience, provides relatively complete consulting, data engineering, model delivery and operation, model application and other services, and has certain tools and processes; outstanding-level service capabilities represent that the pre-training model delivery cloud management service provider has a certain scale of personnel reserves and case experience, provides relatively complete consulting, data engineering, model delivery and operation, model application and other services, and has certain tools and processes; The pre-training model delivery cloud management service provider has a complete personnel reserve, and its cases cover multiple industries and multiple enterprises. It can provide complete consulting, data engineering, model delivery and operation, model application and other services, provide advanced technical support, and has a complete service process, a complete tool chain and platform; the leading service capability representative pre-training model delivery cloud management service provider has a complete personnel reserve, case experience and innovation capabilities, and provides expert consulting and strategic planning services in the field of artificial intelligence; provides highly customized model delivery and deployment services; provides intelligent data management, model deployment, operation and maintenance monitoring and operation management and other related tools, demonstrating the industry-leading model delivery technical service capabilities.

[0132] The data classified by the processing module is backed up to the cloud, and the processing module completes the data classification through the classification algorithm, and the classification algorithm formula is:

[0133]

[0134] Where xi is the sample data, xj is the data to be classified, d(xi,xj) is the classification result. At the same time, for data xi and xj, if their feature space is an n-dimensional real vector space xi=(xi1,xi2,xi3...xin), xj=(xj1,xj2,xj3...xjn).

[0135] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A pre-trained model engineering delivery capability grading evaluation system, characterized by ,include: A collection module, which is used to collect various data information of the provider to be evaluated, including enterprise basic capabilities, model consulting capabilities, data engineering capabilities, model delivery capabilities, model application capabilities, and model operation capabilities; A processing module, the processing module is used to classify and store the data collected by the collection module, and to back up the classified data, and to add a timestamp to the backed up data; A comparison module, which uses the rating standards preset in the sample library as a template to compare and analyze the classified data to determine its grade; A numerical output module outputs and displays the assessed provider capabilities in the form of specific numbers based on preset scoring grade assignment weights, and the weight assignment is adjusted and set according to needs.

2. According to claim 1, a pre-training model engineering delivery capability grading evaluation system is characterized by: The basic capabilities of an enterprise include enterprise qualifications, personnel skills, management system, financial status and case analysis.

3. According to claim 1, a pre-training model engineering delivery capability grading evaluation system is characterized by: The model consulting capability includes demand assessment, resource assessment, business assessment and feasibility assessment, and the model consulting capability refers to the cloud management service provider providing customers with resource, business and feasibility demand assessment and model scenario, selection, data and resource planning solution design based on the model delivery project.

4. According to claim 1, a pre-training model engineering delivery capability grading evaluation system is characterized by: The data engineering capabilities include training data management and knowledge base management, and training data management includes data collection, data cleaning, data labeling and data conversion. At the same time, knowledge base management includes knowledge base construction and knowledge base docking.

5. According to claim 1, a pre-training model engineering delivery capability grading evaluation system is characterized by: The model delivery capabilities include model adaptation, model training, model tuning and model verification, and model adaptation includes environment configuration, POC side view and model deployment. At the same time, model training includes training and push platform and reasoning scenario, model tuning includes parameter adjustment, reinforcement learning and prompt word construction, and the model verification includes performance verification, experience verification, reliability verification and stability verification.

6. A pre-training model engineering delivery capability grading evaluation system according to claim 1, characterized in that: The model application capabilities include intelligent agent construction and RAG application, and intelligent agent construction includes task decomposition, workflow orchestration, template library construction and intelligent agent collaboration. At the same time, RAG application includes atomic capability evaluation, secondary development capabilities and RAG workflow orchestration.

7. A pre-training model engineering delivery capability grading evaluation system according to claim 1, characterized in that: The model operation capability includes operation and maintenance services, operation and maintenance center, model application security and continuous service improvement, and the operation and maintenance services include model hosting, model monitoring, fault handling, time response and training services; the operation center includes service orchestration and metering and billing; the model application security includes model image security, model itself security, integrated application security and vulnerability protection detection; the continuous service improvement includes effect evaluation and user feedback.

8. A pre-training model engineering delivery capability grading evaluation system according to claim 1, characterized in that: The graded assessment derived from the comparison module is divided into five levels from low to high, namely, exploration level, practice level, excellent level, outstanding level and leadership level.

9. A pre-training model engineering delivery capability grading evaluation system according to claim 8, characterized in that: The exploratory service capability means that the pre-trained model delivery cloud management service provider has a preliminary scale of personnel reserves and can provide basic consulting, delivery and other capabilities; Practical service capability means that the pre-trained model delivery cloud management service provider has a certain scale of personnel reserves and case experience, can provide basic consulting, data engineering, model delivery and operation, model application and other services, and has certain tool capabilities; excellent service capability means that the pre-trained model delivery cloud management service provider has a certain scale of personnel reserves and case experience, provides relatively complete consulting, data engineering, model delivery and operation, model application and other services, and has certain tools and processes; Excellent service capability means that the pre-trained model delivery cloud management service provider has a complete personnel reserve, cases covering multiple industries and enterprises, and can provide comprehensive consulting, data engineering, model delivery and operation, model application and other services, provide advanced technical support, have a complete service process, and a complete tool chain and platform; Leading-level service capabilities mean that the pre-trained model delivery cloud management service provider has a complete personnel reserve, case experience and innovation capabilities, and provides expert consulting and strategic planning services in the field of artificial intelligence; provides highly customized model delivery and deployment services; provides intelligent data management, model deployment, operation and maintenance monitoring and operation management and other related tools, demonstrating industry-leading model delivery technical service capabilities.

10. A pre-training model engineering delivery capability grading evaluation system according to claim 1, characterized in that: The data classified by the processing module is backed up to the cloud, and the processing module completes the classification of the data through a classification algorithm, and the classification algorithm formula is: Where xi is the sample data, xj is the data to be classified, d(xi,xj) is the classification result. At the same time, for data xi and xj, if their feature space is an n-dimensional real vector space xi=(xi1,xi2,xi3...xin), xj=(xj1,xj2,xj3...xjn).

Images