Model evaluation method, electronic device, storage medium, and program product

Abstract

The present disclosure provides a model evaluation method, an electronic device, a storage medium and a program product, which can be applied to the field of artificial intelligence technology. The model evaluation method comprises: determining at least one target safety operation associated with a to-be-evaluated model based on a selection operation of a user for a plurality of safety operations; embedding the at least one target safety operation into an initial pipeline to obtain a target pipeline; and in response to a model iteration event related to the to-be-evaluated model being triggered, executing the target pipeline to obtain an evaluation result of the to-be-evaluated model.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more specifically, to a model evaluation method, electronic device, storage medium, and program product. Background Technology

[0002] DevSecOps is a modern software delivery and security paradigm that integrates development, security, and operations. It is a deep extension of the DevOps concept into the security field. Its core is to break down the traditional departmental barriers of "security being placed at the back" and achieve "security shifting to the left." It embeds security capabilities into the entire software lifecycle, ensuring agile business iteration while bringing security capabilities to the forefront of R&D, significantly reducing vulnerability remediation costs, and building an intrinsic security defense line. It can meet the needs of the engineering and security development of large language models. Summary of the Invention

[0003] This disclosure provides a model evaluation method, electronic device, storage medium, and program product.

[0004] One aspect of this disclosure provides a model evaluation method, comprising: determining at least one target security operation associated with a model to be evaluated; embedding at least one of the target security operations into an initial pipeline to obtain a target pipeline; and executing the target pipeline in response to a model iteration event associated with the model to be evaluated being triggered to obtain an evaluation result of the model to be evaluated.

[0005] Another aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the method as described above.

[0006] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed, are used to implement the method described above.

[0007] Another aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, are used to implement the method described above. Attached Figure Description

[0008] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0009] Figure 1 An exemplary system architecture for which model evaluation methods can be applied according to embodiments of this disclosure is shown.

[0010] Figure 2 A flowchart of a model evaluation method according to an embodiment of the present disclosure is shown.

[0011] Figure 3A A schematic diagram of an initial pipeline according to an embodiment of the present disclosure is shown.

[0012] Figure 3B A schematic diagram of a target pipeline according to an embodiment of the present disclosure is shown.

[0013] Figure 3C A schematic diagram of a target pipeline according to another embodiment of the present disclosure is shown.

[0014] Figure 3D A schematic diagram of a target pipeline according to another embodiment of the present disclosure is shown.

[0015] Figure 3E A schematic diagram of a target generation pipeline according to an embodiment of the present disclosure is shown.

[0016] Figure 4A A schematic diagram of an execution target pipeline according to an embodiment of the present disclosure is shown.

[0017] Figure 4B A schematic diagram of the execution interface of a target pipeline according to an embodiment of the present disclosure is shown.

[0018] Figure 5A A schematic diagram of an execution code review operation according to an embodiment of the present disclosure is shown.

[0019] Figure 5B A schematic diagram showing the result display interface of a code review job according to an embodiment of the present disclosure is illustrated.

[0020] Figure 5C A schematic diagram of an execution component vulnerability scanning operation according to an embodiment of the present disclosure is shown.

[0021] Figure 5D A schematic diagram showing the result display interface of a component vulnerability scanning operation according to an embodiment of the present disclosure is shown.

[0022] Figure 6A A schematic diagram of a computing resource application interface according to an embodiment of the present disclosure is shown.

[0023] Figure 6B A schematic diagram of a computing resource application according to an embodiment of the present disclosure is shown.

[0024] Figure 6C A schematic diagram of a service creation interface according to an embodiment of the present disclosure is shown.

[0025] Figure 6DA schematic diagram illustrating the creation of a service according to an embodiment of this disclosure is shown.

[0026] Figure 6E A schematic diagram of an environment instance configuration interface according to an embodiment of the present disclosure is shown.

[0027] Figure 6F A schematic diagram of an environment instance configuration interface according to an embodiment of the present disclosure is shown.

[0028] Figure 6G A schematic diagram of an iterative creation interface according to an embodiment of the present disclosure is shown.

[0029] Figure 6H A schematic diagram illustrating the iterative creation according to an embodiment of this disclosure is shown.

[0030] Figure 6I A schematic diagram of a security evaluation operation performed according to an embodiment of the present disclosure is shown.

[0031] Figure 7A A schematic diagram of a model management interface according to an embodiment of the present disclosure is shown.

[0032] Figure 7B A schematic diagram of an edit model window or a new model window according to an embodiment of the present disclosure is shown.

[0033] Figure 7C A schematic diagram of a model evaluation interface according to an embodiment of the present disclosure is shown.

[0034] Figure 7D A schematic diagram of a new model safety evaluation operation window according to an embodiment of the present disclosure is shown.

[0035] Figure 7E A schematic diagram of a first evaluation set interface according to an embodiment of the present disclosure is shown.

[0036] Figure 7F A schematic diagram of a second evaluation set interface according to an embodiment of the present disclosure is shown.

[0037] Figure 7G A schematic diagram of a first result display interface according to an embodiment of the present disclosure is shown.

[0038] Figure 7H A schematic diagram of a first result display interface according to an embodiment of the present disclosure is shown.

[0039] Figure 7I A schematic diagram of a visual model evaluation process according to an embodiment of the present disclosure is shown.

[0040] Figure 7J A schematic diagram of a large language model evaluation process according to an embodiment of the present disclosure is shown.

[0041] Figure 8 A schematic diagram of global security verification according to an embodiment of the present disclosure is shown.

[0042] Figure 9 A block diagram of an electronic device suitable for implementing a model evaluation method according to an embodiment of the present disclosure is shown. Detailed Implementation

[0043] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0044] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0045] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0046] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0047] With the rapid development of artificial intelligence technology, large language models have been applied in many fields. While providing convenience for code development, content production, and intelligent interaction, the security risks hidden in the development and application process are becoming increasingly prominent. Security risks run through the entire life cycle of model development, component integration, training and optimization, deployment and operation and maintenance, covering multiple dimensions such as code security, third-party component dependency security, and model security itself.

[0048] In related technologies, the development and maintenance of large language models often adopt the DevOps model. The security assessment process usually lags behind the development and deployment process. Security control does not intervene in the early stages of development, such as code writing and component integration. Security vulnerabilities such as code vulnerabilities, third-party component vulnerabilities, and model security risks are usually only discovered during the deployment stage or after the model goes online. Vulnerability remediation not only requires backtracking the entire development chain to roll back the code, replace components, and retrain the model, but also has high remediation costs and technical difficulties. It also leads to the extension of the model release cycle and affects the iterative development efficiency of large language models.

[0049] To address the issue of lagging security management in the DevOps model of related technologies, DevSecOps proposes embedding security capabilities into all stages of software development and operation, achieving a "left-hand" security approach to mitigate security risks from the development source. It has already seen some application in the general software field. However, DevSecOps solutions in related technologies are primarily designed for general software development scenarios and lack specific security evaluation processes and indicator systems tailored to the characteristics of large language models' development processes and security risks. This fails to meet the end-to-end security management needs of large language model scenarios and lacks targeted evaluation capabilities for risks unique to large language models, such as alignment security and adversarial robustness.

[0050] In view of this, the embodiments of this disclosure, based on the DevSecOps security left-shift concept, propose a model evaluation method for large language models. This method achieves full-link security control of large language models from computing power application to version release, improving security levels while ensuring model iteration efficiency and meeting compliant release requirements. The model evaluation method includes: identifying at least one target security job associated with the model to be evaluated; embedding the at least one target security job into an initial pipeline to obtain a target pipeline; and executing the target pipeline in response to a model iteration event related to the model to be evaluated, to obtain the evaluation result of the model to be evaluated.

[0051] In the embodiments disclosed herein, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0052] In the embodiments disclosed herein, user authorization or consent is obtained before acquiring or collecting user personal information.

[0053] Figure 1 An exemplary system architecture for applying model evaluation methods according to embodiments of this disclosure is shown. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0054] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0055] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays, including but not limited to smartphones, tablets, laptops, and desktop computers. Users can initiate evaluation control commands for the model through the first terminal device 101, the second terminal device 102, and the third terminal device 103. The first terminal device 101, the second terminal device 102, and the third terminal device 103 can display the evaluation results of the model through their front-end interactive interfaces on their displays.

[0056] Server 105 can be a server that provides various services, such as a server that supports model evaluation control commands initiated by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The server can perform model evaluation based on the received model evaluation control commands and feed back the evaluation results to the terminal devices.

[0057] It should be noted that the model evaluation method provided in this disclosure embodiment can generally be executed by server 105. The model evaluation method provided in this disclosure embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Alternatively, the model evaluation method provided in this disclosure embodiment can also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0058] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0059] Figure 2 A flowchart of a model evaluation method according to an embodiment of the present disclosure is shown.

[0060] like Figure 2 As shown, the method includes operations S201~S203.

[0061] In operation S201, at least one target safety operation is identified that is associated with the model to be evaluated.

[0062] In operation S202, at least one target safety operation is embedded into the initial pipeline to obtain the target pipeline.

[0063] In operation S203, in response to the triggering of a model iteration event related to the model to be evaluated, the target pipeline is executed to obtain the evaluation results of the model to be evaluated.

[0064] The model to be evaluated can be a large language model in the development and iteration stage, or it can be other types of models in the development and iteration stage, such as visual models, speech models, etc., without limitation.

[0065] A security job is an independently orchestratable and reusable atomic security task unit. In the embodiments of this disclosure, multiple security jobs can be pre-defined. These multiple security jobs can be pre-encapsulated into various security detection functions for large language models. These various security detection functions may include code detection functions, component detection functions, model robustness testing functions, sensitive data leakage detection functions, etc. Each security job can be configured with independent detection rules, execution logic, and output standards, and supports flexible selection and combination.

[0066] At least one target security job associated with the model to be evaluated can be represented as a set of security jobs that have a direct mapping relationship with the scope of changes, risk attributes, and release requirements of the model to be evaluated in this iteration.

[0067] At least one target safety operation can be determined based on the user's subjective choice. The user can subjectively determine whether the safety operation needs to be associated with the model to be evaluated, based on a series of factors such as risk level, compliance requirements, and historical risk status.

[0068] For example, the user terminal device can display a safety job configuration interface, which can display multiple safety jobs. The user can select and combine these multiple safety jobs based on the iteration type and safety level requirements of the model to be evaluated, in order to obtain at least one target safety job associated with the model to be evaluated.

[0069] Alternatively, a series of matching rules can be pre-defined. Each matching rule can be associated with a change at a certain level of the model to be evaluated, such as a change at the code level, a change at the model level, or a change at the deployment level. Each matching rule can be associated with one or more security jobs. When the iteration of the model to be evaluated matches the matching rule, the security job associated with that security job can be set as the target security job associated with the model to be evaluated.

[0070] For example, matching rule A can represent whether there is a change at the code level. Matching rule A can be related to code review and component vulnerability scanning jobs in multiple security jobs. If the current iteration of the model to be evaluated involves the addition or modification of inference service code, business logic code, etc., then the current iteration of the model to be evaluated can be considered to be related to a change at the code level, which matches matching rule A. The changed code may introduce injection vulnerabilities, privilege escalation vulnerabilities, hard-coded keys, and open-source protocol risks, and corresponding jobs need to cover code-level risks. In this case, code review and component vulnerability scanning jobs can be selected as target security jobs associated with the model to be evaluated.

[0071] For example, matching rule B can represent whether there is a change in the model layer. Matching rule B can be related to robustness detection, hallucination detection, privacy detection, and concept drift detection in multiple security tasks. If the current iteration of the model to be evaluated involves fine-tuning of model parameters, adjustment of pre-training corpus, or adjustment of prompt word templates, then the current iteration of the model to be evaluated can be considered related to a change in the model layer, which matches matching rule B. Changes to the model itself may introduce risks specific to large models, such as content violations, security bypass, and training data leakage, requiring model-specific tasks to cover model-layer risks. In this case, robustness detection, hallucination detection, privacy detection, and concept drift detection can be selected as target security tasks associated with the model to be evaluated.

[0072] For example, matching rule C can represent whether there is a change in the deployment level. Matching rule C can be related to interface security detection jobs, deployment configuration baseline detection jobs, etc., among multiple security jobs. If the current iteration of the model to be evaluated involves changes in deployment configuration, addition or modification of interfaces, adjustment of service permission policies, or adjustment of network exposure scope, then the current iteration of the model to be evaluated can be considered to be related to changes in the deployment level, which matches matching rule C. Changes in deployment configuration may introduce risks such as unauthorized access, configuration errors, and plaintext data transmission, and corresponding jobs need to cover deployment layer risks. In this case, interface security detection jobs, deployment configuration baseline detection jobs, etc., can be selected as target security jobs associated with the model to be evaluated.

[0073] Optionally, the current iteration of the model to be evaluated can also be matched with multiple matching rules. In this case, the security jobs associated with each of the multiple matching rules can be selected as the target security jobs associated with the model to be evaluated, which will not be elaborated here.

[0074] By selecting target security jobs associated with the model under evaluation, detection can be performed only on risk points related to changes, avoiding the repetitive testing of unchanged modules in traditional full-scale evaluation. This reduces computational consumption, improves evaluation efficiency, and adapts to the rapid iteration needs of large language models. A pipeline refers to an automated workflow from model or software development to deployment and maintenance, which can include multiple pipeline jobs that can be executed serially or in parallel. Accordingly, the initial pipeline can incorporate multiple general pipeline jobs designed for large language model development, including code retrieval, image building, image pushing, model deployment, model uploading, and API call jobs.

[0075] For each of the at least one target safety job, the target safety job can be embedded before or after any pipeline job in the initial pipeline to obtain the target pipeline. The target pipeline can be a customized pipeline instance adapted to the current iteration of the model to be evaluated, generated after embedding the selected target safety job, and includes a complete execution chain from environment preparation to safety detection and result output.

[0076] Embedding a target security job can be a process of severing dependencies between existing pipeline jobs in the initial pipeline and establishing new dependencies between the target security job and the existing pipeline jobs. For example, a target security job could be a code review job. In the initial pipeline, the dependencies between existing pipeline jobs can be represented as the code retrieval job and the image build job executing sequentially. If the code review job needs to be embedded between the code retrieval job and the image build job, then the sequential execution relationship between the code retrieval job and the image build job needs to be severed, and sequential execution relationships need to be established between the code retrieval job and the code review job, as well as between the code review job and the image build job, thus obtaining the target pipeline. In the target pipeline, the embedded target security job can be equivalent to the original pipeline jobs in the initial pipeline, both being components of the target pipeline. Model iteration events can be event-driven signals that trigger automatic pipeline execution. They can correspond to key nodes in the development of large language models that result in version changes and iterations, including submitting iteration versions to the code repository, updating third-party component versions, and adjusting model parameters.

[0077] The evaluation results of the model under evaluation can include security conclusions output after the target pipeline has been executed. For example, the evaluation results can include "evaluation passed" or "evaluation failed." These results can serve as the basis for determining whether a model version can be archived and deployed. For instance, if the evaluation result indicates "evaluation passed," a snapshot of the current iteration of the model under evaluation can be automatically saved and deployed to the production environment. If the evaluation result indicates "evaluation failed," the current iteration of the model under evaluation can be rejected. Developers can then fix vulnerabilities, optimize model security metrics, and resubmit a new iteration to the code repository to re-trigger pipeline execution and re-enter the quality verification and version confirmation process.

[0078] During the execution of both the assembly line operations and the target safety operations included in the target pipeline, there may be situations where the operation is abnormal. If any operation is abnormal, the execution of the target pipeline can be interrupted. After the user troubleshoots the abnormality, the execution of the target pipeline can be restarted.

[0079] If at least one target security job within the target pipeline executes normally, the results of at least one target security job can be obtained. For example, after the code review job is completed, the results can be output including security vulnerabilities, coding style issues, and logical problems present in the current iteration of the model under evaluation. As another example, after the component vulnerability scanning job is completed, the results can be output including known vulnerabilities present in the components involved in the current iteration of the model under evaluation.

[0080] The evaluation result of the model to be evaluated can be obtained based on the results of at least one target security task. For example, if at least one target security task includes a code review task and a component vulnerability scanning task, the number of high-risk security vulnerabilities and high-risk logic problems in the current iteration can be determined from the results of the code review task, and the number of high-risk known vulnerabilities in the current iteration can be determined from the results of the component vulnerability scanning task. If the total number of high-risk security vulnerabilities, high-risk logic problems, and high-risk known vulnerabilities is greater than a preset value, an evaluation result indicating that the evaluation has failed can be obtained; if the total number of high-risk security vulnerabilities, high-risk logic problems, and high-risk known vulnerabilities is less than or equal to the preset value, an evaluation result indicating that the evaluation has passed can be obtained. In this example, the preset value can be determined based on the security level of the model to be evaluated, for example, it can be set to 0. According to embodiments of this disclosure, after a user selects at least one target safety job through a front-end interface, the at least one target safety job can be embedded into the initial pipeline to form a target pipeline dedicated to the model to be evaluated. By integrating the target safety job into the pipeline, safety left shift can be achieved, supporting flexible configuration of safety jobs and adapting to the iteration requirements of large language models. After the model iteration event is triggered, the target pipeline can be executed automatically, which can effectively improve the efficiency of model evaluation.

[0081] The following describes specific embodiments with reference to the accompanying drawings. Figure 2 The method shown will be further explained.

[0082] After identifying at least one target safety operation, it can be embedded into the initial pipeline to obtain the target pipeline. The implementation method of embedding the target safety operation into the initial pipeline is not limited here.

[0083] Optionally, the initial pipeline includes multiple job embedding sites, which are located before or after the pipeline jobs included in the initial pipeline; a target job embedding site that matches the target safety job can be determined from the multiple job embedding sites; and at least one target safety job is embedded at the determined at least one target job embedding site to obtain the target pipeline.

[0084] Based on the arrangement principles such as the principle of satisfying prerequisite dependencies, the principle of prioritizing risk blocking, and the principle of topology optimization, the target job embedding site that matches the target safe job can be determined from multiple job embedding sites.

[0085] Optionally, the principle of satisfying prerequisite dependencies can be expressed as follows: all prerequisites for the execution of a security job must be completed before the job embedding point, and there must be no missing dependencies. For example, a code review job only needs code files to execute, so the target job embedding point matching the code review job must be after the code job is obtained; as another example, the security evaluation of a large language model requires calling a normally accessible model inference interface, so the target job embedding point matching the security evaluation job must be after the model deployment job.

[0086] The principle of prioritizing risk blocking can be expressed as embedding security tasks as early as possible in the process to detect risks early and avoid wasting computing power in subsequent stages. For example, when code scanning discovers high-risk vulnerabilities, it can directly block subsequent model building and deployment stages to avoid wasting computing power on training or deploying risky model versions. Therefore, the target task embedding point that matches the code review task should be the task embedding point that is as early as possible among multiple task embedding points.

[0087] The principle of topology optimization can be expressed as follows: security jobs with no dependencies should be embedded in parallel as much as possible to shorten the overall pipeline execution time. For example, model security evaluation jobs such as robustness detection, hallucination detection, and privacy detection have no sequential dependencies. Therefore, robustness detection, hallucination detection, and privacy detection jobs can be configured as jobs to be executed in parallel and embedded into the same job embedding site.

[0088] A workflow engine can be used to encapsulate target safety jobs as executable nodes of a pipeline, and embed at least one target safety job into the initial pipeline according to the at least one target job embedding site determined by the matching, to form a target pipeline.

[0089] For example, firstly, the target safety job can be instantiated to generate an independently executable safety job instance; secondly, the node registration interface of the workflow engine can be called to register the safety job instance as a pipeline node; finally, the process orchestration interface of the workflow engine can be called to embed at least one pipeline node into the link of the initial pipeline according to the target job embedding position that matches each pipeline node, so as to obtain the target pipeline.

[0090] Figure 3A A schematic diagram of an initial pipeline according to an embodiment of the present disclosure is shown.

[0091] like Figure 3AAs shown, the initial pipeline may include multiple pipeline jobs executed serially. These multiple pipeline jobs are: code retrieval job 301, image building job 302, image push job 303, model deployment job 304, model listing job 305, and API call job 306. Before or after each pipeline job, there may be job embedding points for the initial pipeline. For example, job embedding point D1 may be included before code retrieval job 301, job embedding point D2 may be included after code retrieval job 301, job embedding point D3 may be included after image building job 302, job embedding point D4 may be included after image push job 303, job embedding point D5 may be included after model deployment job 304, job embedding point D6 may be included after model listing job 305, and job embedding point D7 may be included after API call job 306.

[0092] In some embodiments, at least one target security job determined from multiple security jobs may include at least one code security evaluation job. These code security evaluation jobs may all be security jobs that perform detection at the code level. For example, at least one code security evaluation job may include at least one of a code review job and a component vulnerability scanning job. The execution prerequisite for a code security evaluation job may be obtaining code files. In the original pipeline job, the code acquisition job can be used to obtain the code files for the current iteration version of the model to be evaluated. Therefore, based on the principle of satisfying prerequisite dependencies, the target job embedding site matching the code security evaluation job should be located after the code acquisition job. Furthermore, based on the principle of prioritizing risk blocking, the code security evaluation job can be placed as early as possible. Therefore, the job embedding site after the code acquisition job, i.e., job embedding site D2, can be used as the target job embedding site matching the code security evaluation job.

[0093] At least one code security evaluation job can be executed in parallel or in sequence. For example, the determination of whether at least one code security evaluation job can be executed in parallel or in sequence can be based on the topology optimization principle as described above, according to whether there is a dependency relationship between at least one code security evaluation job. No limitation is made here.

[0094] Figure 3B A schematic diagram of a target pipeline according to an embodiment of the present disclosure is shown.

[0095] like Figure 3B After the target security job is embedded, the target pipeline can be represented as the serial execution of the following jobs: code acquisition job 301, code review job 307, component vulnerability scanning job 308, image building job 302, image push job 303, model deployment job 304, model uploading job 305, and interface call job 306.

[0096] In some embodiments, at least one target security job determined from multiple security jobs may include at least one model security evaluation job. These model security evaluation jobs may all be security jobs that perform detection at the model level. For example, at least one model security evaluation job may include at least one of robustness detection, illusion detection, privacy detection, and concept drift detection jobs. The execution prerequisite for a model security evaluation job may be the completion of model deployment and ensuring the model is in a callable state. In the original pipeline job, the interface call job can be used to provide a model call interface to the security platform. Therefore, based on the principle of satisfying prerequisite dependencies, the target job embedding site matching the model security evaluation job should be located after the interface call job. The job embedding site D7 after the interface call job is the last job embedding site. Therefore, job embedding site D7 can be used as the target job embedding site matching the model security evaluation job.

[0097] At least one model security evaluation job can be executed in parallel or in sequence. For example, the execution of at least one model security evaluation job in parallel or in sequence can be determined based on whether there is a dependency relationship between at least one model security evaluation job, according to the topology optimality principle mentioned above. No limitation is made here.

[0098] Figure 3C A schematic diagram of a target pipeline according to another embodiment of the present disclosure is shown.

[0099] like Figure 3C After the target security job is embedded, the target pipeline can be represented as the serial execution of the code acquisition job 301, the image building job 302, the image pushing job 303, the model deployment job 304, the model uploading job 305, and the interface call job 306, and then the robustness detection job 309, the illusion detection job 310, the privacy detection job 311, and the concept drift detection job 312 are executed in parallel.

[0100] In some embodiments, based on user selection, at least one target security job determined from a plurality of security jobs may include at least one code security evaluation job and at least one model security evaluation job. The at least one code security evaluation job may include at least one of a code review job and a component vulnerability scanning job. The at least one model security evaluation job may include at least one of a robustness detection job, an illusion detection job, a privacy detection job, and a concept drift detection job. The at least one code security evaluation job may be executed in parallel or sequentially, and the at least one model security evaluation job may be executed in parallel or sequentially; no limitation is made herein.

[0101] Figure 3DA schematic diagram of a target pipeline according to another embodiment of the present disclosure is shown.

[0102] like Figure 3D After the target security task is embedded, the target pipeline can be represented as follows: execute the code acquisition task 301, then execute the code review task 307 and the component vulnerability scanning task 308 in parallel, then execute the image building task 302, the image push task 303, the model deployment task 304, the model listing task 305 and the interface call task 306 in sequence, and then execute the robustness detection task 309, the illusion detection task 310, the privacy detection task 311 and the concept drift detection task 312 in parallel.

[0103] Optionally, the embedding of the target safety operation into the initial pipeline can also be achieved through the automatic orchestration function of the workflow engine. For example, the target safety operation can be embedded at any position in the initial pipeline, and the rationality of the resulting new pipeline flow can be determined. The rationality of the resulting new pipeline flow can be determined by checking whether it meets the orchestration principles mentioned above, such as the prerequisite dependency satisfaction principle, the risk blocking priority principle, and the topology optimization principle. If all orchestration principles are met, the resulting new pipeline flow is considered rational; if not, it is considered unreasonable. If the new pipeline flow is determined to be rational, it can be output as the target pipeline; if the new pipeline flow is determined to be unreasonable, the embedding of the target safety operation can be repeated.

[0104] Figure 3E A schematic diagram of a target generation pipeline according to an embodiment of the present disclosure is shown.

[0105] like Figure 3E As shown, after triggering a model iteration event, the system can load the service information and code repository branches associated with that iteration, and load and display the initial pipeline. For at least one target security job that needs to be embedded in the initial pipeline, the system can sort the target security job among the multiple pipeline jobs included in the initial pipeline according to the principle of "security first, development second, deployment third, and evaluation last," and determine whether the output sorting result is reasonable. If reasonable, the job sorting is confirmed, and the target pipeline is output. If unreasonable, the system can re-sort the jobs.

[0106] According to embodiments of this disclosure, when embedding safety operations into a pipeline, the safety operations can be automatically sorted with the original pipeline operations to ensure that the process is reasonable and efficient. This can adapt to the iterative needs of large language models, achieve full-process automation, reduce manual intervention, and improve development security and operational efficiency.

[0107] A target pipeline corresponding to the model to be evaluated can be generated before the development iteration of the model to be evaluated. During the development iteration of the model to be evaluated, if the developers submit an iterative version of the model to be evaluated, it can be considered that the model iteration event related to the model to be evaluated is triggered, and the target pipeline is executed.

[0108] When executing a target pipeline, each job can be executed sequentially or in parallel according to the pipeline's orchestration process. For target safety jobs, additional runtime data and output results for each target safety job can be collected. By executing the target pipeline, the results of at least one target safety job can be obtained.

[0109] Figure 4A A schematic diagram of an execution target pipeline according to an embodiment of the present disclosure is shown.

[0110] like Figure 4A As shown, when executing the target pipeline, the system can execute each pipeline job in the target pipeline serially or in parallel according to a predetermined orchestration order. During the execution of each pipeline job, the system can record the job's execution log in real time, including execution time, execution steps, execution status, log information, etc., and display it in real time through the display interface. If the currently executing job is determined to be a target security job, the system can additionally record the job's results, such as the number of vulnerabilities at each level and security assessment scores. After a pipeline job is successfully executed, the next pipeline job can be executed; conversely, if the pipeline job fails, the pipeline can be paused, a failure log can be output, and the user can retry the pipeline job after troubleshooting the problem.

[0111] Figure 4B A schematic diagram of the execution interface of a target pipeline according to an embodiment of the present disclosure is shown.

[0112] like Figure 4B As shown, the execution interface of the target pipeline can include the job identifier, job name, and status of each job in the target pipeline. For target security jobs, it can also display their job results, such as the number of high-risk, medium-risk, and low-risk vulnerabilities, and security assessment scores. The status indicates whether the corresponding job was executed successfully.

[0113] Each job can be configured with corresponding operation buttons, including a "View Log" button and a "View Report" button. Clicking the "View Log" button will pop up a log details window on the target pipeline's execution interface. Clicking the "View Report" button will pop up a report details window about the job results on the target pipeline's execution interface, or it can jump to the corresponding job's results display interface.

[0114] According to embodiments of this disclosure, by automating the execution of the target pipeline when a model iteration event is triggered, execution efficiency can be improved, and manual intervention and errors can be reduced. By collecting execution logs and the results of target safety operations in real time, data support can be provided for subsequent problem investigation, process tracing, and global security verification. Furthermore, by using the target pipeline to execute each operation, flexible control of operations such as pausing, continuing, and retrying operations can be achieved, thereby improving the flexibility of the model evaluation process.

[0115] The following description, in conjunction with the accompanying drawings and embodiments, illustrates the execution process and results of the target safety operation.

[0116] Optionally, at least one target security operation includes at least one code security evaluation operation; the code security evaluation operation can be a security detection operation targeting the model code, in which case the code security evaluation operation can be executed after the model code is obtained.

[0117] According to embodiments of this disclosure, executing a target pipeline to obtain the results of at least one target safety operation may include the following operations:

[0118] Obtain the model code related to the model iteration event; based on the model code, execute at least one code security evaluation job, and obtain the job results of at least one code security evaluation job, wherein the job results of at least one target security job include the job results of at least one code security evaluation job.

[0119] For example, code security assessment tasks include code review tasks. The process of performing a code review task can include: performing static scanning of the model code based on programming language security rules and the characteristics of the model source code to identify code defects; and obtaining the results of the code review task based on these defects. That is, for the source code of a large language model, static scanning can be used, combining programming language-specific security rules and the characteristics of the large language model code, to identify security vulnerabilities, coding style issues, logical defects, etc., and classify and statistically analyze these vulnerabilities according to their risk level, thereby obtaining the results of the code review task. By performing static scanning of the model code, security risks at the code level can be addressed, enabling source control of code security and preventing vulnerabilities from flowing into subsequent stages and affecting model security.

[0120] Figure 5A A schematic diagram of an execution code review operation according to an embodiment of the present disclosure is shown.

[0121] like Figure 5AAs shown, when performing a code review job, model code can be pulled from the code repository, and code review rules can be loaded. Code review rules can include language-specific security rules, large language model programming rules, etc. Both the model code and code review rules can be input into a code scanning tool to perform static scanning of the code, identifying security vulnerabilities, coding style issues, and logical defects. The code scanning tool can be one that can detect source code through lexical analysis, semantic analysis, and rule matching without running the code. Identified issues can be categorized and statistically analyzed according to their risk level. Risk levels can include high-risk, medium-risk, and low-risk. High-risk indicates a risk of sensitive information leakage or injection vulnerabilities; medium-risk indicates coding style issues or logical redundancy; and low-risk indicates incomplete comments. Based on the statistical results of defects at each risk level, the code review job results can be output. These results can include details such as the location, description, and risk level of defects, as well as the number of defects at each risk level and remediation suggestions. For example, the results of a code review assignment can be expressed as: There are A high-risk defects, B medium-risk defects, and C low-risk defects in the model code; among them, high-risk defect A1 is located on line E1 of file D1, with the description information XXX and the repair suggestion XXX; medium-risk defect B2 is located on line E2 of file D2, with the description information XXX and the repair suggestion XXX; low-risk defect C3 is located on line E3 of file D3, with the description information XXX and the repair suggestion XXX, and so on.

[0122] If an exception occurs during static scanning, such as inaccessibility of the code repository or code parsing failure, an exception log can be output, and the code review job can be paused until the problem is identified and then retried.

[0123] Figure 5B A schematic diagram showing the result display interface of a code review job according to an embodiment of the present disclosure is illustrated.

[0124] like Figure 5B As shown, the code review job results display interface can include job information, source code statistics, source code type information, and defect level statistics. Job information can include the detection language, detection method, code repository address, task type, source code source, and source code branch. Source code statistics can include package size, number of files, total number of lines, number of source code lines, number of comment lines, number of blank lines, and number of mixed lines. Source code type information can include the number of lines in source code of types such as .sh, .dockerfile, .config, .master, and .md. Defect level statistics can include the total number of defects, the number of high-risk defects, the number of medium-risk defects, and the number of low-risk defects.

[0125] According to embodiments of this disclosure, by performing code review tasks and adapting the code characteristics of the programming language and the large language model to perform static scanning of the code, code defects can be identified from the source of the code, improving the accuracy of code defect identification, reducing defect repair costs, and classifying and statistically analyzing the identified defects according to their risk levels. Users can prioritize the handling of high-risk defects and then handle low-risk defects, thereby improving defect repair efficiency and meeting the needs of rapid iteration.

[0126] For example, code security assessment tasks include component vulnerability scanning tasks. The process of performing a component vulnerability scanning task may include: based on the model code, identifying at least one target component from multiple components on which the model under assessment depends; and based on a vulnerability database, identifying vulnerabilities in the at least one target component to obtain the results of the vulnerability scanning task. That is, a comprehensive scan can be performed on the third-party components, libraries, and plugins on which the model under assessment depends, determined based on the model code. Combined with a security vulnerability database, known vulnerabilities in the components can be identified, and the identified known vulnerabilities can be classified and statistically analyzed according to their risk level to obtain the results of the component vulnerability scanning task. By performing component vulnerability scanning on the model code, security risks caused by the introduction of third-party components can be avoided, achieving source control of code security.

[0127] Figure 5C A schematic diagram of an execution component vulnerability scanning operation according to an embodiment of the present disclosure is shown.

[0128] like Figure 5C As shown, when performing a component vulnerability scan, the list of dependent components of the model to be evaluated can be read to identify multiple third-party components on which the model depends. Based on the model code, the scan scope can be determined, i.e., at least one target component needs to be scanned. For this target component, known vulnerabilities can be identified by comparing with a vulnerability database, and the risk level, impact scope, and remediation scheme of the vulnerabilities can be obtained. Simultaneously, the identified vulnerabilities can be categorized and statistically analyzed according to their risk level to output the results of the component vulnerability scan. For example, the results of the component vulnerability scan can be expressed as: among the third-party components on which the model to be evaluated depends, N1 third-party components have critical vulnerabilities, N2 third-party components have high-risk vulnerabilities, N3 third-party components have medium-risk vulnerabilities, N4 third-party components have low-risk vulnerabilities, and N5 third-party components have no vulnerabilities. As another example, the results of the component vulnerability scan can be expressed as: at least one third-party component on which the model to be evaluated depends has a total of M1 critical vulnerabilities, M2 high-risk vulnerabilities, M3 medium-risk vulnerabilities, and M4 low-risk vulnerabilities.

[0129] If an anomaly occurs during the scan, such as a network anomaly, an anomaly log can be output, the component vulnerability scan can be paused, and the scan can be retried after the problem has been investigated.

[0130] Figure 5D A schematic diagram showing the result display interface of a component vulnerability scanning operation according to an embodiment of the present disclosure is shown.

[0131] like Figure 5D As shown, the results display interface for a component vulnerability scan can include statistical graphs for at least one of the following statistical dimensions: component statistics, vulnerability statistics, and license statistics. The component statistics graph can represent the number of components with critical vulnerabilities, high-risk vulnerabilities, medium-risk vulnerabilities, low-risk vulnerabilities, or no vulnerabilities, as well as the number of open-source and proprietary components in at least one target component. The vulnerability statistics graph can represent the total number of low-risk, medium-risk, high-risk, and critical vulnerabilities in at least one dependent target component. The license statistics graph represents the number of low-risk, medium-risk, and high-risk licenses among the licenses used by at least one target component.

[0132] According to embodiments of this disclosure, by performing component vulnerability scanning, known vulnerabilities in third-party components can be comprehensively identified, and security risks introduced by components can be avoided; statistical classification by risk level facilitates prioritizing the handling of high-risk vulnerabilities; and in conjunction with code review operations, end-to-end security control can be achieved.

[0133] Optionally, at least one target security operation may include at least one model security evaluation operation; the model security evaluation operation may be a security detection operation for the model as a whole, and may be performed after the model is deployed and put on the shelf.

[0134] According to embodiments of this disclosure, executing a target pipeline to obtain the results of at least one target safety operation may include the following operations:

[0135] Obtain the model code related to the model iteration event; based on the model code, deploy and list the model service in the runtime environment; execute at least one model security evaluation job for the model service, and obtain the job results of at least one model security evaluation job, wherein the job results of at least one target security job include the job results of at least one model security evaluation job.

[0136] Optionally, the image build job, image push job, model deployment job, and model listing job can be executed sequentially to deploy and list the model service in the runtime environment. Specifically, by executing the image build job, a container image of the model service can be built based on the model code; by executing the image push job, the built container image can be pushed to a private image repository; when executing the model deployment job, the container image can be deployed from the private image repository to the runtime environment to deploy the model service in the runtime environment; by executing the model listing job, the model service can be registered with the gateway to list the model service in the runtime environment, making the model service ready to run.

[0137] In embodiments of this disclosure, the runtime environment can be any runtime environment involved in the development cycle of the large language model. For example, the runtime environment can be any of the development environment, testing environment, or production environment. In some embodiments, the runtime environment can also be a grayscale environment, a sandbox environment, etc., and is not limited thereto.

[0138] Optionally, by associating the applied computing resources with the created services, the computing resources can be configured in a fine-grained manner to generate the required operating environment and ensure that the specifications and parameters of the computing resources are adapted to the operating requirements of the large language model. This can solve the problem of model operation abnormalities caused by mismatch between environment configuration and model requirements or non-standard configuration, and provide a stable operating environment for model testing and deployment.

[0139] When developing and iterating the model to be evaluated, computing resources can be allocated to the model to be evaluated based on the computing power requirements of the large language model through a standardized computing resource application process. This will enable on-demand allocation of computing resources, address the issue of non-standard resource allocation, ensure that computing resources are adapted to the model's operating requirements, and improve the utilization rate of computing resources.

[0140] Figure 6A A schematic diagram of a computing resource application interface according to an embodiment of the present disclosure is shown.

[0141] like Figure 6A As shown, users can submit a computing resource application form by filling in information such as region, network zone, environment name, number of GPU cards, and usage time on the computing resource application interface.

[0142] Figure 6B A schematic diagram of a computing resource application according to an embodiment of the present disclosure is shown.

[0143] like Figure 6BAs shown, after a user submits a computing power resource application, the system can verify the application's validity, including checking if required fields are complete and if the computing power specifications are within the platform's available range. If the system verification is successful, the process can be transferred to the system administrator. After the system administrator approves the application, the system can allocate computing power resources of the corresponding specifications from the computing power resource pool and bind them to the model to be evaluated. If the review fails, the system administrator can return review comments, and the user can resubmit the computing power resource application.

[0144] After completing the allocation of computing resources, the results of the computing resource allocation can be output, including resource identifier, computing rules, allocation status, bound projects, effective time, and expiration time.

[0145] According to embodiments of this disclosure, the computing power resource application process enables on-demand application and precise allocation of computing power resources, ensuring the stability of model operation; dynamic scheduling of computing power resources improves resource utilization and reduces operation and maintenance costs; and standardized application process improves application efficiency and reduces manual intervention.

[0146] While applying for computing resources, basic service information related to the model to be evaluated can be entered, including service name, service abbreviation, service description, code repository address, terminal category, etc. This can clarify the core attributes of the service, provide basic identification and support for subsequent environment configuration, and solve the problems of chaotic service management and inconsistent information.

[0147] Figure 6C A schematic diagram of a service creation interface according to an embodiment of the present disclosure is shown.

[0148] like Figure 6C As shown, users can enter basic service information such as service name, service abbreviation, service description, code repository address, client category, model type, compilation language, compilation tool version, and file path on the service creation interface.

[0149] Figure 6D A schematic diagram illustrating the creation of a service according to an embodiment of this disclosure is shown.

[0150] like Figure 6D As shown, after a user enters basic service information, the system can validate this information, such as verifying whether required fields are complete, whether the code repository address is valid, and whether the service abbreviation is unique. If the validation passes, the system can create the service and generate a unique service identifier, which can be associated with the service's basic information. If the validation fails, an error message will be returned, allowing the user to modify and re-enter the service's basic information.

[0151] According to embodiments of this disclosure, standardized service information entry enables unified service management; clarifies core service attributes, ensuring that subsequent processes are adapted to service requirements; facilitates service iteration, maintenance, and traceability, and improves the level of engineering management.

[0152] Figure 6E A schematic diagram of an environment instance configuration interface according to an embodiment of the present disclosure is shown.

[0153] like Figure 6E As shown, users can enter instance configuration parameters such as port, CPU, memory, number of CPU cards, number of replicas, restart policy, environment variables, and mount directory through the environment instance configuration interface.

[0154] Figure 6F A schematic diagram of an environment instance configuration interface according to an embodiment of the present disclosure is shown.

[0155] like Figure 6F As shown, instance configuration parameters can be obtained based on the user's input. These parameters are then bound to computing resources associated with the service identifier of the model under evaluation, generating the runtime environment. Specifically, the user can associate allocated computing resources with the service identifier and select the required computing resource identifier. After the user enters the instance configuration parameters, the system can verify them, including checking for port conflicts and whether the CPU / memory / GPU configurations meet the computing resource specifications. If the verification passes, the system can bind the instance configuration parameters to the computing resources and generate instance configuration details to complete the runtime environment configuration. If the verification fails, the system can return an error message, allowing the user to modify and re-enter the instance configuration parameters.

[0156] According to embodiments of this disclosure, by accurately associating computing resources with services, it can be ensured that the environment configuration is adapted to the model requirements. By standardizing the instance configuration process, configuration errors can be reduced. Based on a stable operating environment, the smooth execution of subsequent pipeline operations can be guaranteed.

[0157] Optionally, each iteration of the large language model can correspond to a version update and be associated with a specified branch of the code repository. By clearly defining the update content of the iteration, version basis can be provided for pipeline configuration and job execution, solving the problems of chaotic versions and untraceable iterations in the large language model, and realizing standardized and traceable management of iterations.

[0158] Figure 6G A schematic diagram of an iterative creation interface according to an embodiment of the present disclosure is shown.

[0159] like Figure 6G As shown, users can enter the iteration name and iteration description through the iteration creation interface to complete the entry of iteration-related information.

[0160] Figure 6H A schematic diagram illustrating the iterative creation according to an embodiment of this disclosure is shown.

[0161] like Figure 6H As shown, users can select a service identifier and enter iteration-related information. The system can create an iteration, generate a unique iteration identifier, and bind the iteration identifier to the service identifier. After creating an iteration, the system can trigger a model iteration event.

[0162] According to embodiments of this disclosure, by standardizing the iterative creation process, the version, update content, and code branch of each iteration can be clearly defined, enabling the traceability of iterative versions, facilitating troubleshooting and version rollback, providing clear version basis for pipeline operations, and ensuring the accuracy of pipeline execution.

[0163] Optionally, after the creation of the operating environment and iterations is completed, a model security evaluation task can be performed to conduct targeted tests on the core security capabilities of the model to be evaluated, including adversarial sample robustness and privacy leakage protection capabilities, and output a quantitative evaluation score, which can reflect the security capabilities of the model to be evaluated.

[0164] Alternatively, model safety assessment can be performed through a security platform.

[0165] Figure 6I A schematic diagram of a security evaluation operation performed according to an embodiment of the present disclosure is shown.

[0166] like Figure 6I As shown, when the security platform triggers a model security evaluation task, it can connect to the deployed model service, load preset evaluation metrics and evaluation sets, and perform targeted testing on the model service. If the test is completed, each evaluation metric can be scored based on the test results. The security evaluation score of the model under evaluation can be calculated by weighting the scores of each metric according to their respective weights, thus obtaining the result of the model security evaluation task. If the test fails to complete, the execution of the model security evaluation task can be paused until the user investigates the problem and confirms that there are no errors, after which the model security evaluation task can be re-executed.

[0167] Specifically, in response to the deployment and listing of the model service in the runtime environment, the system can generate interface access credentials for the model service. Using these credentials, the model service can be loaded on the security platform; thereby, at least one model security evaluation job can be executed on the security platform for the model service, yielding the results of each of the at least one model security evaluation job.

[0168] With the large-scale deployment of artificial intelligence technology, model security has become a core bottom line for industrial applications. Large language models are capable of processing massive amounts of text, while multimodal models integrate diverse information inputs such as text, images, and audio. Without effective input security assessment and risk control mechanisms, there will be many security risks in actual interactions.

[0169] For example, in real-world applications, input information often carries complex noise. In industrial production line image recognition, input images may suffer from uneven lighting or partial product occlusion. In smart financial identity verification, issues such as image blurring or image tampering may arise. If the model lacks sufficient anti-interference capability and robustness, it may result in missed or incorrect judgments in smart factory defect detection and security monitoring, directly leading to inaccurate production management and security failure.

[0170] For example, users increasingly demand reliability from model output information. However, due to limitations in their working principles, large language models and visual models may generate incorrect knowledge answers or output descriptions unrelated to images, resulting in "illusions" in their output. These factual errors directly mislead users' judgments. In client-facing service scenarios such as smart education and intelligent customer service, erroneous output not only affects user experience but also directly damages the quality of corporate services and brand reputation.

[0171] For example, medical images, personal identification information, and biometric data generated during user interaction with models are all considered private data. If these are leaked due to security vulnerabilities, they will not only directly infringe upon the legitimate rights and interests of users, but also expose companies to legal compliance risks and brand reputation crises.

[0172] Therefore, in the embodiments of this disclosure, multiple model safety evaluation jobs can be set up. When building the target pipeline, one or more model safety evaluation jobs can be selected and added to the original pipeline to achieve a comprehensive evaluation of model safety.

[0173] The various model security evaluation tasks can include robustness testing, hallucination testing, privacy testing, and concept drift testing.

[0174] Robustness detection, hallucination detection, privacy detection, and concept drift detection jobs can be configured with priorities. Robustness detection can have the highest priority, followed by hallucination detection, then privacy detection, and concept drift detection can be performed only when performance degrades after long-term operation, thus having a lower priority. In some embodiments, model safety evaluation jobs can also be selected according to this priority order.

[0175] A model's robustness directly determines its reliability in practical applications. Robustness testing jobs can support various data modalities, including text, images, audio, and video as input data. They are compatible with both general and custom evaluation sets and cover abnormal input scenarios. By executing robustness testing jobs, the cost of troubleshooting after model deployment can be reduced, and the model can be adapted to specific business scenarios, such as the personalized evaluation needs of medical imaging and industrial inspection.

[0176] Whether a model exhibits hallucinations directly impacts the credibility of its output. Visual inspection tasks can quantify the presence of hallucinations or the severity of such issues using a dedicated evaluation set. Furthermore, through result sharing and collaborative optimization, these tasks can be adapted to specific scenarios for large language models, addressing their core shortcomings and preventing legal risks and business trust crises caused by false information output. Coupled with compliance reviews, this can expand the application scenarios of large language models, such as into advertising and financial information generation.

[0177] A model's privacy performance can affect users' sense of security when using the model. Privacy testing jobs can be configured with an evaluation set containing test data on account permission tier protection. This evaluation set supports custom configuration for privacy scenarios. By utilizing this evaluation set, combined with two-way privacy protection assessments, compliance risk costs associated with model use can be reduced, and product credibility can be enhanced.

[0178] After prolonged use, models may experience concept drift, leading to decreased accuracy in their output. Concept drift detection supports multimodal model adaptation, monitors real-time changes in the data stream, and works in conjunction with an optimization-re-detection closed loop to address performance degradation after long-term model operation. This ensures the model remains effective as business data iterates, preventing evaluation failure due to data changes, thereby extending the model's lifespan and reducing retraining costs.

[0179] The following section uses a large language model or visual model as an example, and the model security evaluation tasks include robustness detection, illusion detection, privacy detection, and concept drift detection. It explains the security platform and the process of using the security platform to perform model security evaluation tasks. The security platform may include a model management interface. After receiving interface access credentials, the security platform loads the interface access credentials and lists the model service on the model management interface, thereby enabling the execution of model security evaluation tasks on the model service.

[0180] Figure 7A A schematic diagram of a model management interface according to an embodiment of the present disclosure is shown.

[0181] like Figure 7AAs shown, the model management interface can include a query window and a management window. The management window of the model management interface can maintain information on multiple model services. Each model service's information can include the model name, model type, application, creator, creation time, and last modification time. In the query window of the model management interface, users can enter information such as the model name, model type, application, creator, and creation time to retrieve the desired model service from the multiple model services included in the management window.

[0182] Each model service in the model management interface's management window can be configured with corresponding operation buttons, including edit and delete buttons. Clicking the edit button will bring up the edit model window in the model management interface. In this window, users can edit and modify information such as the model name, model type, application, and interface parameters of the model service.

[0183] The model management interface can also be configured with a "Create New Model" button. Clicking the "Create New Model" button will open a new model window in the model management interface. Figure 7B A schematic diagram of an edit model window or a new model window according to an embodiment of the present disclosure is shown.

[0184] like Figure 7B As shown, in the edit model window or create model window, users can enter information such as model name, model type, application, and interface parameters to modify or create model services.

[0185] According to embodiments of this disclosure, a model management interface can be used to track model modification records and prevent the model from being tampered with.

[0186] The security platform may also include a model evaluation interface. After loading the interface access credentials, the model service can also be loaded on the model evaluation interface. Users can create model security evaluation jobs corresponding to the model service on the model evaluation interface.

[0187] Robustness detection, hallucination detection, privacy detection, and concept drift detection can all be treated as independent detection tasks, differing significantly in their defect causes and detection logic. For example, robustness issues stem from insufficient model resistance to input interference, with the core detection focus on "output stability after input variation"; hallucination issues arise from biases in model knowledge reserves or flaws in generation logic, with the core detection focus on "verifying the authenticity of output information"; privacy issues arise from data training leaks or missing recognition rules, with the core detection focus on "the ability to protect and avoid privacy information"; and concept drift issues arise from changes in data flow distribution, with the core detection focus on "the model's adaptability to data changes." Treating robustness detection, hallucination detection, privacy detection, and concept drift detection as independent tasks ensures the accuracy and specificity of the detection logic.

[0188] Furthermore, different business scenarios have different requirements for detection priorities. For example, content creation models focus more on illusion detection, financial models focus more on privacy detection, and industrial detection models rely more on robustness detection. Treating robustness detection, illusion detection, privacy detection, and concept drift detection as independent detection jobs allows users to select them as needed, avoiding the waste of resources caused by full detection and aligning with the lightweight design concept.

[0189] Figure 7C A schematic diagram of a model evaluation interface according to an embodiment of the present disclosure is shown.

[0190] like Figure 7C As shown, similar to the model management interface, the model evaluation interface can also include a query window and a management window. The management window of the model evaluation interface can maintain job information for multiple model security evaluation jobs. Each job's information can include job name, model type, application, evaluation model, evaluation set, perturbation evaluation set, job status, creation time, and creator. In the query window of the model evaluation interface, users can enter information such as job name, model type, application, evaluation model, evaluation set, perturbation evaluation set, job status, creation time, and creator to retrieve the desired model security evaluation job from the multiple model security evaluation jobs included in the management window.

[0191] Each model safety evaluation job in the management window of the model evaluation interface can be configured with corresponding operation buttons. When the job status is "successful," the corresponding operation buttons can include a "Result" button and a "Delete" button; when the job status is "Terminated," the corresponding operation buttons can include a "Continue" button and a "Delete" button; when the job status is "Evaluating," the corresponding operation buttons can include a "Terminate" button. Clicking the "Result" button will redirect the user to the results display interface; clicking the "Delete" button will delete the corresponding model safety evaluation job from the model evaluation interface; clicking the "Continue" button will change the job status from "Terminated" to "Evaluating" and continue executing the job; clicking the "Terminate" button will change the job status from "Evaluating" to "Terminated" and stop executing the job.

[0192] The management window of the model evaluation interface can also be configured with a button to create a new model safety evaluation job. After the user clicks the button to create a new model safety evaluation job, a new model safety evaluation job window will pop up on the model evaluation interface.

[0193] Figure 7D A schematic diagram of a new model safety evaluation operation window according to an embodiment of the present disclosure is shown.

[0194] like Figure 7D As shown, in the "Create New Model Security Evaluation Job" window, users can enter information such as the job name, application, and model type to complete the creation of a new model security evaluation job. At this time, the management window of the model evaluation interface can maintain the job information of this newly created model security evaluation job.

[0195] The security platform may also include an evaluation set interface. After creating a model security evaluation task corresponding to the model service, users can select the evaluation sets to participate in the evaluation in the evaluation set interface. For example, the first evaluation set interface can display evaluation sets related to visual models, and the second evaluation set interface can display evaluation sets related to large language models.

[0196] Figure 7E A schematic diagram of a first evaluation set interface according to an embodiment of the present disclosure is shown.

[0197] like Figure 7E As shown, the first evaluation set interface can maintain multiple evaluation sets. Each evaluation set's information can include the evaluation set name, the application it belongs to, the number of images, its status, the creator, the creation time, and any notes. Furthermore, users can search for evaluation sets using information such as the evaluation set name, the application it belongs to, its status, the creator, and the creation time.

[0198] Each evaluation set in the first evaluation set interface can be configured with corresponding operation buttons, which may include edit buttons, details buttons, delete buttons, etc.

[0199] Figure 7F A schematic diagram of a second evaluation set interface according to an embodiment of the present disclosure is shown.

[0200] like Figure 7F As shown, the second evaluation set interface can maintain multiple evaluation sets. The evaluation set information for each evaluation set can include the evaluation set name, latest version, number of questions, creator, creation time, and remarks.

[0201] Each evaluation set in the second evaluation set interface can be configured with corresponding operation buttons, which may include edit buttons, version management buttons, delete buttons, etc.

[0202] According to embodiments of this disclosure, based on a user's selection of multiple evaluation sets on an evaluation set interface, at least one target evaluation set related to the model safety evaluation task is determined; sample data included in at least one target evaluation set is provided to the model service to obtain output data; and the task result of the model safety evaluation task is obtained based on the output data and the label data included in at least one target evaluation set.

[0203] Optionally, the evaluation set interface can include multiple evaluation sets related to the user's identification information and model safety evaluation tasks. That is, based on the user's identification information, the user's role, permissions, and other information can be determined. Each evaluation set can be configured with a corresponding permission level and usable role information. Based on the user's role, permissions, and other information, evaluation sets available to the user can be filtered from the evaluation set library. Then, a secondary filtering of evaluation sets can be performed based on the specific task represented by the model safety evaluation task to obtain multiple evaluation sets related to the user's identification information and model safety evaluation task. Optionally, the same evaluation set can be adapted to multiple model safety evaluation tasks. For example, a general text interference evaluation set can support both text noise testing in robustness detection tasks and basic question-and-answer accuracy verification in hallucination detection tasks, etc. A single model safety evaluation task can also be associated with multiple evaluation sets. For example, robustness detection can select the system's built-in general interference evaluation set, or upload a medical image-specific dirty data evaluation set, an industrial audio-visual anomaly evaluation set, etc. By employing a many-to-many relationship design between evaluation sets and model security evaluation jobs, evaluation sets can be reused, effectively improving flexibility and scenario adaptability. For example, users do not need to repeatedly create similar evaluation sets for different model security evaluation jobs, reducing data preparation and storage costs. For complex business scenarios, such as multimodal model detection scenarios that combine text generation and image recognition, multi-dimensional evaluations can be quickly completed by combining different evaluation sets, avoiding the need to create multiple model security evaluation jobs, significantly shortening the evaluation cycle, and meeting the efficient operation needs of non-technical personnel.

[0204] The security platform can be configured with different result display interfaces corresponding to different model types. For example, the security platform can be configured with a first result display interface corresponding to the visual model and a second result display interface corresponding to the large language model.

[0205] Each model security evaluation task generates an independent evaluation report, which clearly identifies the specific manifestations of each defect, such as image rotation interference in robustness testing and ID number leakage in privacy detection. The evaluation report can be displayed through either a first or second results display interface to help developers quickly identify optimization directions and improve model iteration efficiency.

[0206] Figure 7G A schematic diagram of a first result display interface according to an embodiment of the present disclosure is shown.

[0207] like Figure 7GAs shown, the first result display interface shows the job information for the corresponding model security evaluation task, including job name, model type, application, evaluation model, original evaluation set, and perturbation evaluation set. It also displays the job results, which may include sub-results for evaluation metrics such as noise contamination, fuzziness distortion, and compression distortion. Each sub-result for a specific evaluation metric may include the number of samples, accuracy, the number of predictions that match between the original and perturbation evaluation sets, and the number of predictions that do not match between the original and perturbation evaluation sets. Based on the sub-results for each evaluation metric, the evaluation score for that metric can be output. By weighted summing of the evaluation scores for multiple evaluation metrics, the job result corresponding to this model security evaluation task can be obtained.

[0208] Optionally, the results display interface can also show data details of the disturbance evaluation set, including the disturbance type, disturbance intensity, original data, prediction label, and disturbance data of each disturbance data in the disturbance evaluation set, and supports querying the disturbance data by disturbance type and prediction result.

[0209] For example, when performing a robustness evaluation task on a visual model, the output results may include a robustness evaluation score; when performing an illusion evaluation task on a visual model, the output results may include an illusion evaluation score, etc.

[0210] Figure 7H A schematic diagram of a second result display interface according to an embodiment of the present disclosure is shown.

[0211] like Figure 7H As shown, in the second results display interface, the results of the model security assessment can be displayed using at least one of the following methods: statistical graphs, detailed evaluation tables, and comprehensive evaluation tables. Statistical graphs, detailed evaluation tables, and comprehensive evaluation tables can all be used to display the same assessment result, but their presentation formats differ. For example, in statistical graphs, model scores for multiple evaluation dimensions can be displayed in the form of a star chart. These multiple evaluation dimensions may include human values, personal and property protection, information accuracy and reliability, legal norms, and prison break and abuse. In the detailed evaluation table, for each evaluation dimension, there may be corresponding sub-dimensions, detailed explanations for each sub-dimension, the model score for that evaluation dimension, and its security level. The comprehensive evaluation table can provide the security level obtained from the comprehensive evaluation, the comprehensive score, and model optimization suggestions. The comprehensive score in the comprehensive evaluation table can be obtained by weighted summation of the model scores for multiple evaluation dimensions; the result of the model security assessment can be represented as this comprehensive average score.

[0212] Optionally, the results of the model safety evaluation task may include the comprehensive score for robustness detection, the comprehensive score for hallucination detection, the comprehensive score for privacy detection, and the comprehensive score for concept drift detection.

[0213] According to embodiments of this disclosure, the security platform can integrate four core evaluation functions: robustness detection, illusion detection, privacy protection detection, and concept drift detection, forming four layers of protection covering the full-dimensional security of the model. It can also perform detection on multimodal interactive information such as text, images, and audio / video, and is compatible with various types of models such as large language models and visual models, as well as general cross-domain business scenarios. This improves the accuracy of model evaluation and solves the problem of performance degradation during long-term model operation.

[0214] Figure 7I A schematic diagram of a visual model evaluation process according to an embodiment of the present disclosure is shown.

[0215] like Figure 7I As shown, through the model management interface provided by the security platform, users can create new visual models and configure the necessary information. Through the first evaluation set interface provided by the security platform, corresponding to model security evaluation tasks such as robustness detection, illusion detection, and privacy detection, users can create original evaluation sets, including uploading evaluation data and configuring relevant information, to generate the original evaluation set. After generating the original evaluation set, the security platform can call the perturbation service to perform image perturbation on the original evaluation set to generate a perturbed evaluation set. Based on the user-configured visual model, the original evaluation set, and the perturbed evaluation set, evaluation data can be assembled. The security platform can call the model inference service, using the original evaluation set and the perturbed evaluation set to evaluate the visual model, obtain inference results, and based on the inference results and the labels of the original and perturbed evaluation sets, generate the job results for the model security evaluation task.

[0216] Figure 7J A schematic diagram of a large language model evaluation process according to an embodiment of the present disclosure is shown.

[0217] like Figure 7J As shown, through the model management interface provided by the security platform, users can create new large language models and configure the necessary information. Through the second evaluation set interface provided by the security platform, users can create evaluation sets for model security evaluation tasks such as robustness detection, illusion detection, and privacy detection. Based on the user-configured large language model and evaluation set, evaluation data can be assembled. The security platform can then call the model inference service, use the evaluation set to evaluate the large language model, obtain inference results, and generate the job results for the model security evaluation task based on the inference results and the labels of the evaluation set.

[0218] According to embodiments of this disclosure, the evaluation can be initiated in just three steps—"selecting model type → checking security assessment task → selecting / uploading assessment set"—without requiring any professional technical background, thanks to a low-threshold interactive design. It is compatible with both private and general models, avoiding complex interface adaptation processes. The many-to-many assessment set reuse mechanism reduces redundant data storage, and the decoupled front-end and back-end design allows for parallel processing of assessment tasks, effectively shortening the evaluation time. It can generate online reports in real time, supports exporting incorrect questions and triggering re-inspection with one click, eliminating the need for offline data organization and secondary uploading, and streamlining the entire "assessment-optimization-re-inspection" process, effectively improving team collaboration efficiency.

[0219] Optionally, at least one target security operation includes at least one code security evaluation operation and at least one model security evaluation operation; the code security evaluation operation can be a security detection operation targeting the model code, in which case the code security evaluation operation can be executed after the model code is obtained; the model security evaluation operation can be a security detection operation targeting the model as a whole, in which case the model security evaluation operation can be executed only after the model is deployed and put on the shelf.

[0220] According to embodiments of this disclosure, executing a target pipeline to obtain the results of at least one target safety operation may include the following operations:

[0221] Obtain the model code related to the model iteration event; based on the model code, execute at least one code security evaluation job and obtain the job results of each of the at least one code security evaluation job; based on the model code, deploy and list the model service in the runtime environment; execute at least one model security evaluation job for the model service and obtain the job results of each of the at least one model security evaluation job; wherein, the job results of each of the at least one target security job include the job results of each of the at least one code security evaluation job and the job results of each of the at least one model security evaluation job.

[0222] According to embodiments of this disclosure, by integrating security assessment tasks such as code review, component vulnerability scanning, and model security evaluation into the pipeline, security assessment can be shifted to the left, enabling security control from the code development and component integration stages, thereby reducing vulnerability remediation costs.

[0223] After obtaining the results of at least one target security task, a global security verification can be performed based on these results to obtain the evaluation results of the model to be evaluated. This process can archive and lock large model iteration versions, determine release access, and execute a final version review according to the platform's global security access rules. If the verification passes, the version is solidified and allowed to be released into the production environment; if the verification fails, the version is rejected and forced to undergo iterative rectification, preventing non-compliant and high-risk model versions from going live.

[0224] Specifically, multiple access conditions can be verified based on the operational results of at least one target safety operation, and the number of access conditions that pass the verification can be determined; based on the number of access conditions that pass the verification, the evaluation result of the model to be evaluated can be obtained.

[0225] Figure 8 A schematic diagram of global security verification according to an embodiment of the present disclosure is shown.

[0226] like Figure 8 As shown, at least one target security task can include code review, component vulnerability scanning, and multiple model security evaluation tasks. These multiple model security evaluation tasks can be robustness detection, illusion detection, privacy protection detection, and concept drift detection, respectively. The admission criteria can include three items: the number of high-risk vulnerabilities identified by the code review task is 0, the number of high-risk vulnerabilities identified by the component vulnerability scanning task is 0, and the overall score of the multiple model security evaluation tasks is ≥60 points. If all three admission criteria pass the verification, version confirmation and locking can be performed, the version can be marked as compliant and usable, and production release permissions can be granted. After version confirmation, version snapshots, security evaluation reports, vulnerability reports, and end-to-end pipeline logs can be automatically saved. Conversely, if any admission criterion fails the verification, the version confirmation application can be rejected, a rejection reason and rectification list can be generated, and version archiving and deployment can be prohibited. Rejected versions require developers to fix vulnerabilities and optimize model security metrics before re-executing the pipeline and re-entering the quality verification and version confirmation process.

[0227] According to embodiments of this disclosure, by rigidly constraining quality access thresholds, high-risk models can be completely blocked from entering production from the process, eliminating the need for subjective manual review, avoiding human oversights and unauthorized releases, and the version lock is irreversible, ensuring that the production model version is unique, stable, and secure.

[0228] Optionally, in the above example, the obtained model code can be the incremental code for this iteration, and the target security operation can be executed based on the incremental code to obtain the operation result of the target security operation. Accordingly, a global security verification can be performed based on the operation results of at least one target security operation to obtain the initial evaluation result of the model to be evaluated; based on the initial evaluation result and the historical evaluation results of the historical iteration versions of the model to be evaluated, the initial evaluation result of the model to be evaluated can be obtained.

[0229] According to the embodiments of this disclosure, unlike full-scale repeated evaluation, targeted security detection can be performed only on incremental code generated in the current iteration due to code changes, new dependent components, and model parameter updates. Historical stable item evaluation data can be reused, effectively reducing computing power consumption and detection time, while ensuring that security coverage is not lacking.

[0230] Figure 9 A block diagram of an electronic device suitable for implementing a model evaluation method according to an embodiment of the present disclosure is shown. Figure 9 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0231] like Figure 9 As shown, an electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0232] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0233] According to embodiments of this disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the input / output (I / O) interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0234] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by processor 901, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0235] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0236] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0237] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 902 and / or RAM 903 described above and / or one or more memories other than ROM 902 and RAM 903.

[0238] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the model evaluation methods provided in the embodiments of this disclosure.

[0239] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0240] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0241] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0242] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of the present disclosure can be combined and / or combined in various ways, even if such combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0243] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A model evaluation method, comprising: Identify at least one target safety operation associated with the model to be evaluated; By embedding at least one of the target safety operations into the initial pipeline, a target pipeline is obtained; as well as In response to the triggering of a model iteration event associated with the model to be evaluated, the target pipeline is executed to obtain the evaluation result of the model to be evaluated.

2. The method according to claim 1, wherein, The execution of the target pipeline to obtain the evaluation results of the model under evaluation includes: Execute the target pipeline to obtain the operation results of at least one of the target safety operations; and A global security verification is performed based on the results of at least one of the target security operations to obtain the evaluation result of the model to be evaluated.

3. The method according to claim 2, wherein, At least one of the target security operations includes at least one code security assessment operation; The execution of the target pipeline to obtain the operation results of at least one of the target safety operations includes: Obtain the model code associated with the model iteration event; Based on the model code, at least one code security evaluation job is executed to obtain the job results of each of the at least one code security evaluation jobs, and the job results of each of the at least one target security job include the job results of each of the at least one code security evaluation jobs.

4. The method according to claim 2, wherein, At least one of the target safety operations includes at least one model safety evaluation operation; The execution of the target pipeline to obtain the operation results of at least one of the target safety operations includes: Obtain the model code associated with the model iteration event; Based on the model code, deploy and list the model service in the runtime environment; At least one model security evaluation job is executed for the model service to obtain the job results of each of the at least one model security evaluation jobs, and the job results of each of the at least one target security job include the job results of each of the at least one model security evaluation jobs.

5. The method according to claim 2, wherein, At least one of the target security operations includes at least one code security assessment operation and at least one model security assessment operation; The execution of the target pipeline to obtain the operation results of at least one of the target safety operations includes: Obtain the model code associated with the model iteration event; Based on the model code, execute at least one of the code security assessment jobs to obtain the job results of each of the at least one code security assessment jobs; Based on the model code, deploy and list the model service in the runtime environment; Execute at least one model security evaluation job on the model service to obtain the job results of each of the at least one model security evaluation jobs; The results of at least one of the target security operations include the results of at least one code security evaluation operation and the results of at least one model security evaluation operation.

6. The method according to claim 3 or 5, wherein, The code security assessment task includes a code review task; The step of executing at least one code security evaluation job based on the model code to obtain the job results of each of the at least one code security evaluation job includes: Based on programming language security rules and model source code characteristics, static scanning is performed on the model code to identify code defects; and Based on the aforementioned code defects, the results of the code review task are obtained.

7. The method according to claim 3 or 5, wherein, The code security assessment task includes a component vulnerability scanning task; The step of executing at least one code security evaluation job based on the model code to obtain the job results of each of the at least one code security evaluation job includes: Based on the model code, at least one target component is determined from the multiple components that the model to be evaluated depends on; Based on the vulnerability database, vulnerabilities are identified in at least one target component to obtain the results of the vulnerability scanning operation.

8. The method according to claim 4 or 5, further comprising: In response to the deployment and listing of the model service in the runtime environment, an interface access credential for the model service is generated; The step of performing at least one model security evaluation job on the model service and obtaining the job results of each of the at least one model security evaluation job includes: Access credentials through the interface to load the model service on the security platform; and At least one model security evaluation job is executed on the model service on the security platform to obtain the job results of each of the at least one model security evaluation job.

9. The method according to claim 8, wherein, At least one of the model security evaluation tasks includes at least one of robustness detection task, illusion detection task, privacy detection task, and concept drift detection task. The security platform includes an evaluation set interface, which includes multiple evaluation sets related to the user's identity and permission information and the model security evaluation task. The step of executing at least one model security evaluation job on the model service on the security platform to obtain the job results of each of the at least one model security evaluation job includes: Based on the user's selection of multiple evaluation sets on the evaluation set interface, at least one target evaluation set related to the model safety evaluation task is determined; Provide the model service with sample data comprising at least one target evaluation set to obtain output data; and Based on the output data and the label data included in at least one target evaluation set, the results of the model safety evaluation task are obtained.

10. The method according to claim 8, wherein, The security platform also includes a results display interface; The method further includes: The results of the model safety evaluation operation are displayed on the results display interface using at least one of the following methods: statistical graphs, evaluation details tables, and comprehensive evaluation tables.

11. The method according to claim 4 or 5, wherein, The process of deploying and listing the model service in the runtime environment based on the model code includes: Based on the model code, construct a container image of the model service; The container image is pushed to a private image repository to facilitate its deployment to the runtime environment, thereby deploying the model service within that environment. Register the model service with the gateway to list the model service in the operating environment.

12. The method according to claim 4 or 5, wherein, The operating environment includes any one of the development environment, testing environment, and production environment; The method further includes: Based on the user's input, the instance configuration parameters are obtained; The instance configuration parameters are bound to the computing resources associated with the service identifier of the model to be evaluated, thereby generating the runtime environment.

13. The method according to claim 2, wherein, The step of performing a global security verification based on the results of at least one of the target security operations to obtain the evaluation result of the model to be evaluated includes: Based on the operational results of at least one of the target safety operations, multiple access conditions are verified to determine the number of access conditions that pass the verification; and The evaluation result of the model to be evaluated is obtained based on the number of valid admission criteria.

14. The method according to any one of claims 2 to 13, wherein, The model code associated with the model iteration event includes the incremental code of the model to be evaluated; The step of performing a global security verification based on the results of at least one of the target security operations to obtain the evaluation result of the model to be evaluated includes: A global security verification is performed based on the results of at least one of the target security operations to obtain the initial evaluation results of the model to be evaluated; and Based on the initial evaluation results and the historical evaluation results of the historical iterations of the model to be evaluated, the initial evaluation results of the model to be evaluated are obtained.

15. The method according to claim 1, wherein, The initial pipeline includes multiple job embedding sites, which are located before or after the pipeline jobs included in the initial pipeline; Wherein, embedding at least one of the target safety operations into the initial pipeline to obtain the target pipeline includes: Determine a target job embedding site that matches the target safety job from among the multiple job embedding sites; and The target pipeline is obtained by embedding at least one of the target safety operations at at least one determined target operation embedding site.

16. An electronic device comprising: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 15.

17. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 15.

18. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 15.

Images