Continuous evaluation method, apparatus, device, and storage medium

Abstract

This invention discloses a continuous evaluation method, apparatus, device, and storage medium, relating to the field of artificial intelligence model evaluation technology. The method includes: acquiring pipeline event data from a continuous integration and continuous deployment pipeline based on a preset monitoring cycle; identifying event types in the pipeline event data to obtain event type identification results; when the event type identification results indicate the existence of a new version tag release event, extracting target evaluation corpus from a preset safety corpus, inputting the target evaluation corpus into an intelligent agent model for batch evaluation, and obtaining offline evaluation results; writing the offline evaluation results into a quality database to obtain quality status records. By embedding the evaluation program into the development process and automatically triggering the safety corpus batch evaluation based on pipeline release events, real-time automatic evaluation after the intelligent agent model is released is achieved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence model evaluation technology, and in particular to a continuous evaluation method, apparatus, device and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, intelligent agents are being used more and more widely in various business scenarios, and the frequency of model iteration is constantly accelerating. During the development of intelligent agents, in order to ensure deployment quality, timely evaluation of model changes is necessary to obtain quality feedback. However, with the popularization of agile development models, traditional evaluation methods are no longer sufficient to meet the urgent need for evaluation efficiency in rapid iteration. There is a pressing need for a solution that can deeply integrate the evaluation process with the development workflow to shorten the cycle from model optimization to quality feedback.

[0003] In existing technologies, agent evaluation is typically run independently as a downstream step in the development process, relatively isolated from the model development process. When developers optimize the agent model or release a new version, the initiation of the evaluation task often relies on manual operation and coordination, resulting in a significant delay in evaluation triggering. Furthermore, because the evaluation process is not effectively integrated with the development process, the quality status of the model after changes cannot be perceived in a timely manner. Developers struggle to quickly ascertain the optimized performance, leading to a lengthy overall cycle for model optimization, evaluation, and feedback, severely hindering the efficiency of agent deployment.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a continuous evaluation method, apparatus, device, and storage medium, aiming to solve the technical problem of how to shorten the evaluation feedback cycle after the release of an intelligent agent model.

[0006] To achieve the above objectives, the present invention provides a continuous evaluation method, which includes the following steps: Based on a preset monitoring cycle, pipeline event data is obtained from the continuous integration and continuous deployment pipeline; The pipeline event data is subjected to event type identification to obtain the event type identification result; When the event type identification result indicates that a new version tag release event exists, the target evaluation corpus is extracted from the preset backup corpus, and the target evaluation corpus is input into the intelligent agent model for evaluation and batch processing to obtain offline evaluation results. The offline evaluation results are written into the quality database to obtain quality status records.

[0007] In one embodiment, the step of performing event type identification on the pipeline event data to obtain the event type identification result includes: Extract the event identifier field and branch identifier field from the pipeline event data; The event identifier field is matched according to a preset event type mapping table to obtain a first matching result and a second matching result, wherein the first matching result corresponds to a merge request event and the second matching result corresponds to a new version tag release event. The branch status of the first matching result is verified based on the branch identifier field to obtain the branch verification result. When the branch verification result indicates that the merge request event exists in the main branch, an event type identification result is generated based on the first matching result and the second matching result.

[0008] In one embodiment, the step of extracting target evaluation data from a preset safety corpus when the event type identification result indicates the existence of a new version tag release event, and inputting the target evaluation data into an intelligent agent model for evaluation and batch processing to obtain offline evaluation results includes: When the event type identification result indicates the existence of a new version tag release event, a candidate corpus set is extracted from a preset backup corpus. Perform business scenario coverage verification on the candidate corpus set to obtain the coverage verification result; When the coverage verification result indicates that the coverage of key business scenarios meets the preset coverage threshold, the target evaluation corpus is selected from the candidate corpus set. The target evaluation corpus is input into the intelligent agent model for inference calculation to obtain model response data; The model response data is compared with the preset standard response for consistency to obtain the offline evaluation results.

[0009] In one embodiment, the step of extracting a candidate corpus set from a preset safety corpus when the event type identification result indicates the existence of a new version tag release event includes: When the event type identification result indicates that a new version tag release event exists, the production behavior distribution model is read. The production behavior distribution model is used to characterize the frequency of occurrence of online business scenarios. Based on the production behavior distribution model, the preset full corpus is sorted by scene frequency to obtain the corpus frequency sorting result. High-frequency data is extracted from the preset full corpus based on the preset corpus size threshold and the corpus frequency sorting results; The high-frequency corpus is subjected to key business scenario coverage screening to obtain a candidate corpus set.

[0010] In one embodiment, the step of comparing the model response data with a preset standard response to obtain offline evaluation results includes: The response content is extracted from the model response data to obtain the actual response text; The standard content of the preset standard response is extracted to obtain the standard response text; The semantic similarity between the actual response text and the standard response text is calculated to obtain a semantic similarity value; When the semantic similarity value is greater than a preset similarity threshold, the model response data is marked as a pass case; When the semantic similarity value is less than or equal to the preset similarity threshold, the model response data is marked as a failure case, and a failure case summary is generated based on the failure case. Offline evaluation results are generated based on the number of successful cases, the number of failed cases, and the summaries of the failed cases.

[0011] In one embodiment, the step of writing the offline evaluation results into a quality database to obtain a quality status record includes: Extract version tag identifiers, offline accuracy rates, and failure case information from the offline evaluation results; A quality record index is generated based on the version tag identifier; The data storage partition is determined based on the quality record index; Write the offline accuracy rate, the failure case information, and the version tag identifier into the data storage partition; The written data is stored and verified. If the storage verification passes, a quality status record is obtained.

[0012] In one embodiment, the step of obtaining pipeline event data from the continuous integration and continuous deployment pipeline based on a preset monitoring period includes: Scanning tasks are generated based on a preset monitoring cycle; Send the scanning task to the continuous integration and continuous deployment pipeline; Receive the raw event logs returned by the continuous integration and continuous deployment pipeline; The original event records are parsed to obtain pipeline event data.

[0013] Furthermore, to achieve the above objectives, the present invention also proposes a continuous evaluation device, the device comprising: The event monitoring module is used to obtain pipeline event data from the continuous integration and continuous deployment pipeline based on a preset monitoring period; The event identification module is used to identify the event type of the pipeline event data and obtain the event type identification result; The offline evaluation module is used to extract target evaluation data from a preset backup corpus when the event type identification result indicates that a new version tag release event exists, input the target evaluation data into the intelligent agent model for evaluation and batch processing, and obtain offline evaluation results. The data storage module is used to write the offline evaluation results into the quality database to obtain quality status records.

[0014] Furthermore, to achieve the above objectives, the present invention also proposes a continuous evaluation device, the device comprising: a memory, a processor, and a continuous evaluation program stored in the memory and executable on the processor, the continuous evaluation program being configured to implement the steps of the continuous evaluation method as described above.

[0015] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a continuous evaluation program, which, when executed by a processor, implements the steps of the continuous evaluation method as described above.

[0016] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the continuous evaluation method described above.

[0017] One or more technical solutions proposed in this application have at least the following technical effects: By deeply integrating the evaluation process with the development workflow, the evaluation process can be automatically triggered when the agent model is modified and a new version is released, eliminating the need for manual coordination and intervention. This significantly shortens the time interval between model release and obtaining quality feedback. Furthermore, because the evaluation trigger is closely linked to the version release event, the quality status of the model after changes can be perceived in a timely manner. Developers can quickly learn about the optimized performance, effectively improving the iteration efficiency and deployment speed of the agent model. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the continuous evaluation method of this application in Implementation Example 1. Figure 2 The overall flowchart provided for Embodiment 1 of the continuous evaluation method of this application; Figure 3 This is a flowchart illustrating Embodiment 2 of the continuous evaluation method of this application; Figure 4 This is a schematic diagram of the module structure of the continuous evaluation device according to an embodiment of this application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the continuous evaluation method in this application embodiment.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or continuous evaluation device capable of performing the above functions. The following description uses a continuous evaluation device as an example to illustrate this embodiment and the subsequent embodiments.

[0025] Based on this, the embodiments of this application provide a continuous evaluation method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the continuous evaluation method of this application.

[0026] In this embodiment, the continuous evaluation method includes steps S10 to S40: Step S10: Obtain pipeline event data from the continuous integration and continuous deployment pipeline based on a preset monitoring cycle; The application scenario for this step is the automatic evaluation triggering stage after the release of the intelligent agent model. The purpose is to proactively sense the operational dynamics of the development pipeline at fixed time intervals, capture key event information such as version release, and provide data input for subsequent automatic evaluation triggering.

[0027] This solution proposes a CI / CD+CE (Continuous Evaluation) architecture, which integrates evaluation procedures into the development model optimization process, achieving deep integration of evaluation and development, and building an automated pipeline of "evaluation upon release, and risk awareness upon discovery." Figure 2 As shown, Figure 2 The overall flowchart is divided into three main modules: development program, pipeline platform, and evaluation program. The development program initiates a development program release, triggering a pipeline label sequence number change and updating the pipeline label list. The pipeline platform queries the interface every 30 minutes, entering the pipeline-triggered CI / CD continuous evaluation process: first, it starts a pipeline monitoring task to query the pipeline version release status, then determines if it's the first call; if it's the first call, it records the current pipeline number and the process ends; if it's not the first call, it compares the current and previous pipeline numbers to see if they match; if they match, the process ends directly; if they don't match, it determines that a pipeline label change has occurred, triggering automatic evaluation of flow control cases; after the automatic evaluation of flow control cases is completed, it notifies the development and testing personnel and updates the test results to the evaluation result storage database. The evaluation process is divided into two paths. The first path involves users uploading evaluation sets on the evaluation page, which are then processed by the evaluation program. Afterward, users can view the evaluation results details, and the process ends. The second path allows users to query the evaluation results, view the evaluation result list by label, and then view the evaluation result details. Test developers can query the results through the front-end page. This path returns to the query evaluation results stage. The output of the automatic evaluation of the flow control case is also connected to the query evaluation results stage of the evaluation program.

[0028] It should be noted that the preset monitoring period refers to a pre-configured time interval parameter, specifically thirty minutes in this embodiment, used to control the frequency of status checks. It should also be noted that a Continuous Integration (CI) / Continuous Deployment (CD) pipeline refers to an automated process that links code integration, building, testing, and deployment. Furthermore, pipeline event data refers to the operation records generated during the operation of the continuous integration / continuous deployment pipeline, covering development operation information such as code commits, version tag releases, and branch changes.

[0029] Understandably, obtaining pipeline event data from the continuous integration and continuous deployment pipeline based on a preset monitoring cycle involves periodically querying the current status of the development pipeline at fixed time intervals, receiving operation record information generated on the pipeline, and thus promptly sensing key development events such as version tag releases.

[0030] Furthermore, by periodically acquiring pipeline event data, it is possible to automatically capture release dynamics on the development pipeline without manual monitoring, thus establishing a data foundation for subsequent identification of version release events and automatic triggering of evaluations.

[0031] In one feasible implementation, step S10 includes steps A11 to A14: Step A11: Generate a scanning task based on a preset monitoring cycle; This step is used in the preparation phase of periodic status checks. Its purpose is to create instruction data for querying the status of the continuous integration and continuous deployment pipeline, providing an operational platform for subsequent status queries to the pipeline.

[0032] It should be noted that a scanning task refers to instruction data used to query the current status of a continuous integration and continuous deployment pipeline. Specifically, it can take the form of a status query request or a timed trigger instruction.

[0033] Understandably, generating scan tasks based on a preset monitoring cycle involves creating status query instructions at 30-minute intervals, enabling status check actions to be generated periodically at a preset frequency.

[0034] Furthermore, by periodically generating scanning tasks, the regularity and continuity of status check actions are ensured, providing an operational basis for the stable monitoring of pipeline dynamic releases.

[0035] Step A12: Send a scan task to the continuous integration and continuous deployment pipeline; This step is used in the transmission phase of status query instructions. Its purpose is to pass the generated status query request to the development pipeline and trigger the pipeline to return the operation record within the current cycle.

[0036] Understandably, sending a scan task to the continuous integration and continuous deployment pipeline is to transmit the generated status query command to the development pipeline, and through this transmission action, request the pipeline to provide feedback on the current operation record information.

[0037] Furthermore, by sending scanning tasks to the pipeline, a data interaction channel with the development pipeline is established, enabling the pipeline to respond and return current operation status information.

[0038] Step A13: Receive the raw event logs returned by the continuous integration and continuous deployment pipeline; This step is used in the response receiving phase of a status query. Its purpose is to obtain the operation log data returned by the continuous integration and continuous deployment pipeline after responding to the status query, and use it as input for subsequent parsing and processing.

[0039] It should be noted that raw event logs refer to the unprocessed operation log data returned after the continuous integration and continuous deployment pipeline responds to the status query. Its format is usually the pipeline's native output format.

[0040] Understandably, receiving raw event records returned by the continuous integration and continuous deployment pipeline is to obtain the raw, unprocessed operation logs from the pipeline, which serve as the basis for subsequent format parsing and event identification.

[0041] Furthermore, by receiving raw event records, complete operation log information from the pipeline was obtained, providing raw materials for extracting key fields from the raw data and identifying event types.

[0042] Step A14: Parse the original event records to obtain pipeline event data.

[0043] This step is used in the preprocessing stage of raw log data. Its purpose is to convert the pipeline's native output format into a standardized data structure that can be used for subsequent identification and processing, and to extract the key field information required for event identification.

[0044] Understandably, parsing the original event records to obtain pipeline event data involves converting the original operation logs returned by the continuous integration and continuous deployment pipeline from their native format into a standardized data structure, extracting key fields such as event identifiers and branch identifiers, and forming standardized data that can be used for subsequent event type identification.

[0045] Furthermore, by parsing the format of the raw event records, the unusable raw logs are converted into structured pipeline event data, enabling the accurate extraction of event identifier fields and branch identifier fields for event type determination.

[0046] Step S20: Perform event type identification on the pipeline event data to obtain the event type identification result; This step is applied in the classification and judgment stage of pipeline events. Its purpose is to accurately distinguish different types of development operation events such as code commits, merge requests, and version tag releases from the obtained pipeline data, and to filter out the valid release events that can trigger evaluation.

[0047] It should be noted that event type identification refers to the data processing procedure of classifying and judging pipeline event data. The event type identification result refers to the conclusive data output after event type identification processing, which indicates the specific operational event category currently occurring on the pipeline.

[0048] Understandably, performing event type identification on pipeline event data and obtaining event type identification results involves classifying and judging the acquired structured pipeline data, distinguishing different operation types such as merge request events and new version tag release events, and outputting identification results containing event category judgment conclusions to provide a basis for subsequent decisions on whether to trigger evaluation.

[0049] Furthermore, by identifying event types in the pipeline event data, valid release operations such as new version tag release events can be accurately filtered out, avoiding non-release events from mistakenly triggering subsequent evaluation processes and ensuring that evaluation is only initiated when a valid version is released.

[0050] In one feasible implementation, step S20 includes steps A21 to A24: Step A21: Extract the event identifier field and branch identifier field from the pipeline event data; This step is used in the basic data preparation stage for event type identification. Its purpose is to extract key data items from structured pipeline event data to determine the event type and location.

[0051] It should be noted that the event identifier field is a specific data item in the pipeline event data used to mark the event type, and its content includes the event type's code or name information. Similarly, the branch identifier field is a specific data item in the pipeline event data used to mark the name of a code branch, distinguishing the source of operations on different code branches such as the main branch, development branch, and test branch.

[0052] Understandably, extracting event identifier and branch identifier fields from pipeline event data involves separating data items that mark event types and data items that mark branch sources from structured pipeline operation records, providing basic data for subsequent type matching and branch validation.

[0053] Furthermore, by extracting the event identifier field and the branch identifier field, two core data dimensions required for event type identification were obtained: the event type dimension and the branch source dimension. This provides data support for accurately identifying published events and confirming that they occurred on core branches.

[0054] Step A22: Perform type matching on the event identifier field according to the preset event type mapping table to obtain the first matching result and the second matching result, where the first matching result corresponds to the merge request event and the second matching result corresponds to the new version tag release event; This step is applied in the automatic classification stage of event types. Its purpose is to map the value of the event identifier field to the specific operation event type through a preset correspondence, and to identify code merging operations and version release operations on the pipeline.

[0055] It should be noted that the preset event type mapping table refers to a pre-configured table of event type correspondences, which establishes a mapping relationship between the values ​​of the event identifier field and specific event type definitions such as merge request events and new version tag release events. It should be noted that type matching refers to the data processing process of comparing the values ​​of the event identifier field with the definitions in the preset event type mapping table one by one. It should be noted that the first matching result refers to the correspondence conclusion obtained after comparing the event identifier field with the merge request event type. It should be noted that the second matching result refers to the correspondence conclusion obtained after comparing the event identifier field with the new version tag release event type. It should be noted that the merge request (MR) event refers to the pipeline operation record triggered when a developer merges a code branch into the main branch. It should be noted that the new version tag release event refers to the pipeline operation record triggered when a version tag is added to the new version code and released during the development process.

[0056] Understandably, the event identifier field is matched against the preset event type mapping table to obtain the first matching result and the second matching result. The first matching result corresponds to the merge request event, and the second matching result corresponds to the new version tag release event. The extracted event identifier field value is compared with the pre-configured event type mapping relationship item by item to determine whether the current event contains merge request operation and version tag release operation, and output two independent matching conclusions.

[0057] Furthermore, by using a preset event type mapping table for type matching, the original event identifiers in the pipeline can be automatically converted into understandable event type definitions, accurately identifying two key development operations: merge requests and version tag releases, providing a classification basis for subsequent comprehensive judgment of release status.

[0058] Step A23: Perform branch status verification on the first matching result based on the branch identifier field to obtain the branch verification result; This step is used in the branch source verification stage of merge request events. Its purpose is to confirm whether the detected merge request event occurred on the core code branch, so as to avoid the subsequent evaluation process being triggered by code merging operations on non-main branches.

[0059] It should be noted that branch status verification refers to the process of determining whether a code operation occurred on a specified branch based on the branch identifier field. The branch verification result refers to the conclusive data output after branch status verification, indicating whether a merge request event exists on the target branch. The main branch refers to the core code branch in the code repository that carries the official release version code; in this embodiment, the code branch serving as the main branch is the target branch of the merge request and the source of the official release.

[0060] Understandably, the branch status is verified based on the branch identifier field of the first matching result to obtain the branch verification result. This is done by using the data item that marks the source of the branch to verify whether the identified merge request event has occurred on the main branch, and outputting a verification conclusion indicating whether there is a merge request event on the main branch.

[0061] Furthermore, by verifying the branch status of the first matching result, non-core merge operations occurring on development or test branches can be filtered out, ensuring that the release attribute of the event is recognized only when there is a merge request on the main branch, thus improving the accuracy of event identification.

[0062] Step A24: When the branch verification result indicates that there is a merge request event in the main branch, generate an event type identification result based on the first matching result and the second matching result.

[0063] This step is applied in the comprehensive generation stage of event type identification conclusions. Its purpose is to comprehensively output pipeline event type identification results that include merge requests and version release judgment conclusions, provided that code merging has been confirmed in the main branch.

[0064] Understandably, when the branch verification result indicates that there is a merge request event in the main branch, the event type identification result is generated based on the first matching result and the second matching result. This is done on the premise that there is a code merge operation in the core branch. The matching conclusion of the merge request event and the matching conclusion of the version tag release event are integrated to generate a comprehensive pipeline event type identification result.

[0065] Furthermore, by combining the first and second matching results to generate an identification conclusion under the condition that there is a merge request in the main branch, it is ensured that the event type identification result not only reflects the existence of the version release event, but also confirms that the release is based on the code merge of the main branch, providing a reliable event judgment basis for subsequent accurate triggering of evaluation.

[0066] Step S30: When the event type identification result indicates that there is a new version tag release event, extract the target evaluation data from the preset backup corpus, input the target evaluation data into the intelligent agent model for evaluation and batch processing, and obtain the offline evaluation results. This step is applied in the automatic quality verification phase after version release. Its purpose is to automatically start regression testing when a valid version release is detected, and obtain quality assessment data of the new version of the agent model by executing tests in batches.

[0067] It should be noted that the preset baseline corpus refers to a pre-constructed set of corpora used for basic regression testing. In this embodiment, it includes two hundred highly representative corpora selected based on production behavior distribution modeling. The target evaluation corpus refers to the specific set of test data selected from the preset baseline corpus for this batch evaluation run, covering key business scenarios. The intelligent agent model refers to the artificial intelligence dialogue model used to handle user interaction requests; in this embodiment, it is the object under test. Batch evaluation refers to the automated test execution process of batch inputting test corpora into the model under test and collecting the model's responses. Offline evaluation results refer to the quality assessment data obtained after executing the batch evaluation run in a non-production environment, including accuracy and failure case information.

[0068] Understandably, when the event type identification result indicates the existence of a new version tag release event, the target evaluation corpus is extracted from the preset backup corpus. The target evaluation corpus is then input into the intelligent agent model for batch testing to obtain offline evaluation results. After a valid version release is identified, test data covering key business scenarios is selected from the pre-built test corpus. The selected test data is then input into the intelligent agent model under test in batches to perform inference. The model's response performance on batch test cases is collected to obtain offline evaluation results including quality indicators and abnormal cases.

[0069] Furthermore, by automatically triggering a minimum corpus evaluation run when a new version tag release event is detected, an automated process of evaluation upon release is achieved. Quality verification can be started immediately after version release without manual intervention, reducing the cycle from model optimization to evaluation feedback from several days to minutes.

[0070] Step S40: Write the offline evaluation results into the quality database to obtain the quality status record.

[0071] This step is used for the persistent storage of evaluation results. Its purpose is to structure and save the version identifiers, quality indicators, and abnormal case information obtained from this evaluation so that the development team can query and trace them later.

[0072] It should be noted that the quality database refers to the data storage area used for persistently storing evaluation results. It should also be noted that the quality status record refers to the traceable data entries formed by saving the evaluation results in a structured form; its function is to record the quality performance of the agent model corresponding to a specific version label.

[0073] Understandably, writing offline evaluation results into a quality database to obtain quality status records involves persistently saving the obtained quality assessment data, which includes accuracy, failure case information, and version identifiers, to a dedicated data storage area, forming a structured quality archive that can be queried and traced later.

[0074] Furthermore, by writing offline evaluation results into the quality database, persistent storage and versioned archiving of evaluation data are achieved, enabling the development team to query historical evaluation results by version tag and supporting comparison and traceability of quality performance of different versions.

[0075] In one feasible implementation, step S40 includes steps A31 to A35: Step A31: Extract version tag identifier, offline accuracy rate, and failure case information from the offline evaluation results; This step is applied in the preparation stage of quality record data. Its purpose is to extract core data items such as version identifiers, quality indicators, and anomaly cases required for building quality records from the evaluation results.

[0076] It should be noted that the version tag identifier refers to the number or name information used to uniquely identify a software version, which in this embodiment corresponds to the version tag released in the pipeline. It should also be noted that offline accuracy refers to the proportion of corpus data in which the model response matches the preset standard response in the offline batch testing run, out of the total number of test corpus data. Furthermore, failure case information refers to a detailed record of test cases in which the model response does not match the preset standard response in the offline batch testing run, including comparative information such as input corpus, actual response, and expected response.

[0077] Understandably, extracting version tags, offline accuracy, and failure case information from offline test results involves separating data items that mark the software version, proportions that measure the model's quality level, and data items that record test failure details from the obtained offline test results. This provides core data content for subsequently building a quality record index and writing it into the data storage partition.

[0078] Furthermore, by extracting version tag identifiers, offline accuracy, and failure case information from offline evaluation results, three core data dimensions required to construct a complete quality status record were obtained: version identifier dimension, quality indicator dimension, and anomaly details dimension, providing a content foundation for subsequent version-based archiving of evaluation data.

[0079] Step A32: Generate a quality record index based on the version tag identifier; This step is applied in the construction phase of quality record retrieval identifiers. The purpose is to create a data retrieval identifier that uniquely corresponds to the software version, so as to facilitate the quick location of evaluation records by version tag in the future.

[0080] It should be noted that the quality record index refers to the data identifier generated based on the version tag identifier, which is used to quickly locate and retrieve the evaluation results of a specific version.

[0081] Understandably, generating a quality record index based on version tags involves using data items that mark software versions to construct a unique retrieval identifier that points to the data storage location of the evaluation data for a specific version tag.

[0082] Furthermore, by generating a quality record index based on version tags, a direct mapping relationship between version tags and evaluation records is established, enabling the development team to quickly retrieve the corresponding quality assessment data based on the version number, thereby improving the query efficiency of historical evaluation data.

[0083] Step A33: Determine the data storage partition based on the quality record index; This step is applied in the planning stage of quality record storage location. Its purpose is to clarify the specific storage location of the current quality record in the quality database and realize partitioned management of different versions of data.

[0084] It should be noted that data storage partitions refer to independent storage areas in the quality database that are divided according to specific rules. Their function is to organize and manage quality record data corresponding to different version tags, and to avoid data mixing.

[0085] Understandably, determining the data storage partition based on the quality record index involves analyzing the corresponding storage rules based on the generated retrieval identifiers to clarify the specific storage area in the quality database for data such as the current version's offline accuracy and failure case information.

[0086] Furthermore, by determining data storage partitions based on the quality record index, the physical or logical isolation of different versions of evaluation data is achieved, avoiding query chaos caused by mixed storage of multiple versions of data, and providing a storage foundation for independent management of quality records by version.

[0087] Step A34: Write the offline accuracy, failure case information, and version tag identifier to the data storage partition; This step is applied in the actual writing stage of quality record data. Its purpose is to persist the core evaluation data to the designated storage area and form a complete version quality archive.

[0088] Understandably, writing offline accuracy, failure case information, and version tag identifiers into the data storage partition is to save the three core data items that have been extracted—quality indicators, abnormal case details, and version identifiers—into a designated independent storage area, thus completing the physical writing of quality record data.

[0089] Furthermore, by writing offline accuracy, failure case information, and version tag identifiers into the data storage partition, the evaluation data was transformed from temporary results to persistent archives, enabling the quality performance of specific versions to be preserved for a long time, providing a data foundation for subsequent quality traceability and trend analysis.

[0090] Step A35: Perform storage verification on the written data. If the storage verification passes, determine that a quality status record has been obtained.

[0091] This step is used in the data write integrity verification stage. Its purpose is to confirm that data such as offline accuracy and failure case information have been correctly persisted to the quality database, ensuring the integrity and reliability of quality records.

[0092] It should be noted that storage verification refers to the integrity verification operation performed after data is written to the storage area. Its purpose is to confirm the integrity and accuracy of the written data.

[0093] Understandably, after the data is written, storage verification is performed. If the storage verification passes, the quality status record is confirmed. This is done after the preceding write operation has completed the data writing. The integrity of the saved offline accuracy, failure case information and version tag is checked. After confirming that the data has been saved correctly and completely, the quality status record is officially confirmed to have been generated.

[0094] Furthermore, by performing storage verification on the written data, the integrity and accuracy of the quality record data are ensured, avoiding the loss of quality archives due to write failures or data loss, and providing the development team with a reliable basis for querying historical evaluation data.

[0095] This embodiment provides a continuous evaluation method that deeply integrates the evaluation program into the continuous integration and continuous deployment pipeline, building an automated process of evaluation upon release. The development pipeline is scanned according to a preset monitoring cycle. When a new version tag release event is detected, a guaranteed corpus evaluation batch run is automatically triggered. Quality verification can be started immediately after version release without manual intervention, significantly shortening the cycle from model optimization to evaluation feedback. Simultaneously, by using a highly representative guaranteed corpus selected based on production behavior distribution modeling as evaluation input, the computational resource consumption of a single evaluation is significantly reduced while ensuring coverage of key business scenarios. Furthermore, by structurally storing offline evaluation results according to version tags and establishing a quality record index, the development team can quickly query historical evaluation results by version tag, supporting the traceability and comparison of quality performance across different versions, forming a complete quality data archive. Further, through the deep integration of the event-driven triggering mechanism with the continuous integration and continuous deployment pipeline, the evaluation stage is embedded as a quality gate in the code release path of the development process. This automatically detects quality issues introduced by model changes early in the development phase, achieving quality shifting to the left and preventing unqualified versions from flowing into subsequent stages.

[0096] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 Step S30 includes steps S301 to S305: Step S301: When the event type identification result indicates that a new version tag release event exists, extract a candidate corpus set from the preset backup corpus; This step is applied in the evaluation corpus preparation stage after the version release. Its purpose is to select a set of candidate test data for this evaluation from the preset backup corpus, so as to provide a data foundation for determining the final evaluation corpus.

[0097] It should be noted that the candidate corpus set refers to a subset of test corpora that has been initially selected from the preset baseline corpus and is yet to be further verified. Its content has not yet been verified for coverage of business scenarios.

[0098] Understandably, when the event type identification result indicates the existence of a new version tag release event, the candidate corpus set is extracted from the preset backup corpus. After confirming that a valid version release has occurred in the pipeline, a batch of test data is initially extracted from the pre-built test corpus according to the screening rules to form a subset of candidate corpus to be verified.

[0099] Furthermore, by extracting a candidate corpus set from a pre-set baseline corpus, the initial corpus range for this evaluation can be quickly identified, providing data preparation for subsequent business scenario coverage verification and determination of the final evaluation corpus.

[0100] In one feasible implementation, step S301 includes steps A41 to A44: Step A41: When the event type identification result indicates that a new version tag release event exists, read the production behavior distribution model. The production behavior distribution model is used to characterize the frequency of occurrence of online business scenarios. This step is applied in the modeling basis reading stage of candidate corpus selection. Its purpose is to obtain model data that represents the distribution pattern of online business scenarios, so as to provide a sorting basis for subsequent corpus selection by scenario frequency.

[0101] It should be noted that online business scenarios refer to the real-world business processing situations faced by the intelligent agent model when handling user requests in an actual production environment, such as specific business types like account inquiries, fund transfers, and credit card applications. It should also be noted that frequency of occurrence refers to the percentage of times a specific business scenario is triggered or processed by users in a real production environment, used to measure how frequently that business scenario occurs.

[0102] Understandably, when the event type identification result indicates the existence of a new version tag release event, the production behavior distribution model is read. The production behavior distribution model is used to characterize the frequency of occurrence of online business scenarios. After confirming the version release, the model data that is pre-built to describe the frequency of occurrence of real business scenarios is loaded to obtain the distribution of each business scenario in the production environment.

[0103] Furthermore, by reading the production behavior distribution model, the frequency distribution pattern of real business scenarios was obtained, enabling subsequent corpus screening to prioritize test data covering high-frequency business scenarios, ensuring that the evaluation corpus matches the actual online business distribution.

[0104] Step A42: Sort the scene frequency of the preset full corpus according to the production behavior distribution model to obtain the corpus frequency ranking result; This step is applied in the frequency ranking stage of the full corpus. The purpose is to rank all test corpora according to the frequency of occurrence of scenarios based on the production behavior distribution model, so as to provide a sequential basis for extracting high-frequency corpora.

[0105] It should be noted that the pre-built full corpus refers to a complete set of all available test data, which is much larger than the pre-built minimum corpus and covers a wider range of business scenario test data. It should also be noted that scenario frequency sorting refers to the data processing process of arranging test data according to the frequency of occurrence of the corresponding business scenario in the production environment. Finally, it should be noted that the corpus frequency sorting result is the output list of corpora arranged in descending order of the frequency of occurrence of the business scenario after scenario frequency sorting.

[0106] Understandably, the frequency ranking of the entire corpus based on the production behavior distribution model is obtained by sorting the frequency of the scenarios. This is done by using model data that represents the frequency patterns of business scenarios to label the frequency level of each corpus in the complete test corpus set, and then arranging them from high to low according to the frequency level, outputting a list of corpus sorted by frequency.

[0107] Furthermore, by sorting the pre-set full corpus by scenario frequency, the test corpus corresponding to high-frequency business scenarios is given priority in the sorting results, providing a clear order guide for the subsequent extraction of corpus covering high-frequency scenarios.

[0108] Step A43: Extract high-frequency data from the preset full corpus based on the preset corpus size threshold and corpus frequency sorting results; This step is applied in the high-frequency corpus extraction stage, and its purpose is to select test data covering high-frequency business scenarios from the sorted full corpus according to a preset scale.

[0109] It should be noted that high-frequency corpus refers to test corpus data that ranks high in the scenario frequency ranking results and whose corresponding business scenarios appear frequently in the production environment. Its content reflects the business types that online users trigger most often.

[0110] Understandably, extracting high-frequency corpus from the preset full corpus based on the preset corpus size threshold and corpus frequency ranking results involves selecting the top-ranked high-frequency scene corpus from the complete test corpus set based on the preset upper limit of the corpus quantity and the obtained corpus list sorted by frequency, forming a corpus subset with controlled size.

[0111] Furthermore, by extracting high-frequency data based on preset corpus size thresholds and corpus frequency ranking results, it is possible to prioritize the retention of test data covering high-frequency business scenarios while controlling the total amount of corpus, thus laying the foundation for constructing a concise and representative evaluation corpus.

[0112] Step A44: Perform key business scenario coverage screening on the high-frequency corpus to obtain a candidate corpus set.

[0113] The application scenario of this step is the business coverage verification stage of the candidate corpus. The purpose is to perform a key business scenario integrity check on the extracted high-frequency corpus to ensure that the candidate corpus set can cover all core business types.

[0114] It should be noted that coverage screening refers to the screening process of checking the completeness of business scenarios on the extracted corpus. Its purpose is to verify whether the extracted corpus contains all key business scenario types and to supplement any missing scenarios.

[0115] Understandably, the process of screening high-frequency corpora for coverage of key business scenarios to obtain a candidate corpus set involves checking each extracted high-frequency scenario corpus to confirm whether it covers all predefined key business scenario types. When missing scenarios are found, the corresponding corpus is supplemented from the pre-set full corpus, ultimately forming a candidate corpus set with complete business coverage.

[0116] Furthermore, by screening high-frequency corpora for coverage of key business scenarios, it is ensured that the candidate corpus set not only covers high-frequency business scenarios, but also includes all key business scenario types, thus avoiding the omission of important but low-frequency business scenarios due to selection based solely on frequency.

[0117] Step S302: Perform business scenario coverage verification on the candidate corpus set to obtain the coverage verification result; This step is used in the quality control stage of candidate corpora. Its purpose is to verify whether the candidate corpus set meets the completeness requirements of business scenario coverage, and to provide a basis for deciding whether to select the final evaluation corpus from the candidate corpora.

[0118] Understandably, performing business scenario coverage verification on the candidate corpus set and obtaining the coverage verification result involves a comprehensive business scenario coverage check on the formed candidate corpus set, statistically analyzing the degree of matching between the number of business scenario types it contains and the total number of predefined key business scenario types, and outputting a verification conclusion indicating whether the coverage meets the standard.

[0119] Furthermore, by verifying the business scenario coverage of the candidate corpus set, the business integrity of the test corpus can be confirmed before the formal evaluation, preventing the evaluation results from failing to reflect the model's true performance in key business scenarios due to incomplete corpus coverage.

[0120] Step S303: When the coverage verification result indicates that the coverage of key business scenarios meets the preset coverage threshold, the target evaluation corpus is selected from the candidate corpus set. This step is applied in the final evaluation corpus determination stage. Its purpose is to select the target corpus for formal evaluation after confirming that the candidate corpus meets the coverage requirements.

[0121] Understandably, when the coverage verification result indicates that the coverage of key business scenarios meets the preset coverage threshold, the target evaluation corpus is selected from the candidate corpus set. This is done after confirming that the business scenario coverage of the candidate corpus meets the preset integrity standard, and the candidate corpus that has passed the verification is determined as the test data set to be officially used in this evaluation.

[0122] Furthermore, by selecting target evaluation corpora from the candidate corpus set after the coverage verification meets the standards, it is ensured that the corpora entering the evaluation stage are both representative of high-frequency scenarios and meet the requirements of full coverage of key business, thus providing corpus quality assurance for obtaining reliable evaluation results.

[0123] Step S304: Input the target evaluation corpus into the intelligent agent model for inference calculation to obtain the model response data; This step is applied in the batch inference execution stage of the intelligent agent model. Its purpose is to input the determined target evaluation corpus into the model under test and obtain the actual output of the model on each test case.

[0124] It should be noted that inference computation refers to the process of feeding the input corpus into the agent model for forward propagation and generating output results; that is, the computational operation in which the model generates corresponding responses based on the input content. It should also be noted that model response data refers to the output data generated by the agent model after receiving the input corpus and performing inference computation; its content is usually the text response or structured answer generated by the model.

[0125] Understandably, inputting the target evaluation corpus into the agent model for inference calculation and obtaining the model response data involves sending the determined test corpus one by one or in batches into the agent model under test, triggering the model to perform inference calculation, and collecting the output results generated by the model for each test corpus.

[0126] Furthermore, by inputting the target evaluation corpus into the intelligent agent model for inference calculation, the actual response data of the model on test cases covering key business scenarios were obtained in batches, providing raw output materials for subsequent comparison with standard responses to determine the evaluation results.

[0127] Step S305: Compare the model response data with the preset standard response to obtain the offline evaluation results.

[0128] This step is used to determine the quality of the model output. The purpose is to compare the actual response generated by the model with the expected standard response, evaluate the model's performance on each test case, and summarize the results into an offline evaluation.

[0129] It should be noted that the preset standard response refers to the pre-configured reference response data used to measure the correctness of the model output, that is, the expected model output content set for each test corpus. It should also be noted that consistency comparison refers to the data processing procedure of comparing and analyzing the actual response generated by the model with the preset standard response to determine whether the two are consistent or similar.

[0130] Understandably, comparing the model response data with the preset standard response to obtain offline evaluation results involves comparing the collected actual output of the model with the pre-set expected output item by item, analyzing the degree of consistency between the two in terms of content or semantics, and summarizing the offline evaluation results containing quality indicators and anomaly details based on the comparison results to count the number of successful and failed cases.

[0131] Furthermore, by comparing the model response data with the preset standard response for consistency, the quality of the model output is automatically judged, and the performance of the model on batch test cases is quantified into structured evaluation results, enabling the development team to intuitively grasp the quality level of the new version model.

[0132] In one feasible implementation, step S305 includes steps A51-A56: Step A51: Extract the response content from the model response data to obtain the actual response text; This step is applied in the stage of extracting the actual output of the model, and its purpose is to separate the text content for comparison from the model response data.

[0133] Understandably, extracting the response content from the model's response data to obtain the actual response text involves parsing the original output data generated by the model, removing irrelevant formatting information or metadata, and separating the core text content generated by the model for the test corpus.

[0134] Furthermore, by extracting the response content from the model response data, we obtained clean actual response text, providing standardized input data for subsequent accurate semantic comparison with standard response text.

[0135] Step A52: Extract standard content from the preset standard response to obtain the standard response text; This step is used in the standard expected output content extraction stage, and its purpose is to separate the reference text content for comparison from the preset standard response.

[0136] Understandably, extracting standard content from the preset standard response to obtain the standard response text involves parsing the pre-configured expected output data, removing irrelevant format information or metadata, and separating the core reference text content set for the test corpus.

[0137] Furthermore, by extracting standard content from the preset standard response, a clean standard response text was obtained, which established a reference benchmark for subsequent semantic similarity calculation with the actual response text.

[0138] Step A53: Calculate the semantic similarity between the actual response text and the standard response text to obtain the semantic similarity value; The application scenario of this step is the semantic similarity quantification stage between the model output and the expected output. The purpose is to calculate the degree of similarity between the two at the semantic level, so as to provide a quantitative basis for determining whether the case passes.

[0139] It should be noted that semantic similarity calculation refers to the process of numerically evaluating the degree of semantic similarity between two texts. This can be achieved using text matching algorithms based on word vectors or pre-trained language models. The semantic similarity value is the numerical result output after semantic similarity calculation, representing the degree of semantic similarity between the two texts; a higher value indicates a closer semantic connection.

[0140] Understandably, calculating the semantic similarity between the actual response text and the standard response text to obtain a semantic similarity value involves inputting the actual text content generated by the model and the pre-defined standard text content into a semantic similarity evaluation algorithm, calculating the degree of closeness between the two at the semantic level, and outputting a quantitative value representing the degree of similarity.

[0141] Furthermore, by calculating the semantic similarity between the actual response text and the standard response text, the degree of conformity between the model output and the expected output is converted into a quantifiable numerical indicator, providing an objective quantitative standard for determining whether a case passes based on a threshold.

[0142] Step A54: When the semantic similarity value is greater than the preset similarity threshold, mark the model response data as a pass case; The application scenario of this step is the judgment and marking stage of the test case. The purpose is to mark the test case as passed when the semantics of the model output and the standard output are highly consistent.

[0143] It should be noted that the preset similarity threshold refers to a pre-configured semantic similarity cutoff value used to determine whether the model response is acceptable. When the calculated semantic similarity value is higher than this cutoff value, the model output is considered to meet expectations. It should also be noted that the test cases refer to test cases where the model response and the preset standard response meet the acceptable standard in semantic similarity calculation; that is, test instances where the model output meets expectations.

[0144] Understandably, when the semantic similarity value is greater than the preset similarity threshold, the model response data is marked as a pass case. This means that when the semantic similarity between the model's actual output and the expected output exceeds the preset passing standard, the test case is classified as a case type where the model output is correct.

[0145] Furthermore, by setting a preset similarity threshold and marking successful cases accordingly, the system achieves automated identification and classification of the model's correct output, enabling qualified cases to be accurately counted and providing basic data for calculating offline accuracy.

[0146] Step A55: When the semantic similarity value is less than or equal to the preset similarity threshold, mark the model response data as a failure case and generate a failure case summary based on the failure cases; This step is applied in the stage of marking and generating summaries of failed cases. Its purpose is to mark the test case as a failed case and generate an abnormal summary when the semantics of the model output are inconsistent with the standard output.

[0147] It should be noted that a failure case refers to a test case in which the model response and the preset standard response do not meet the passing standard in semantic similarity calculation, that is, a test instance in which the model output does not meet expectations.

[0148] Understandably, when the semantic similarity value is less than or equal to the preset similarity threshold, the model response data is marked as a failure case, and a failure case summary is generated based on the failure case. When the semantic similarity between the actual output and the expected output of the model does not exceed the preset qualified standard, the test case is classified as a case type of model output error. At the same time, key information such as the input corpus, actual response and expected response of the case are extracted to generate summary data to describe the details of the failure.

[0149] Furthermore, by marking failed cases and generating failure case summaries, the system achieves automated identification and recording of abnormal model outputs, enabling the development team to quickly locate specific cases where the model is performing poorly, and providing clear directions for improvement in subsequent model optimization.

[0150] Step A56: Generate offline evaluation results based on the number of successful cases, the number of failed cases, and the summary of failed cases.

[0151] This step is used in the offline assessment results aggregation stage, and its purpose is to integrate the statistical information of successful and failed cases, as well as the failure details, into complete assessment result data.

[0152] Understandably, generating offline evaluation results based on the number of successful cases, the number of failed cases, and the summaries of failed cases involves counting the total number of cases where the model outputs correctly and the total number of cases where it outputs incorrectly during this evaluation. This, combined with the generated summary information of failed cases, forms a structured offline evaluation result that includes overall quality indicators and details of abnormal cases.

[0153] Furthermore, by generating offline evaluation results based on the number of successful cases, the number of failed cases, and the summary of failed cases, the execution conclusions of batch evaluations are integrated into unified result data, enabling the development team to grasp the overall quality level and specific defect distribution of the new version model through a single result data.

[0154] This embodiment provides a continuous evaluation method. By sorting the pre-set full corpus according to scenario frequency based on a production behavior distribution model and combining it with key business scenario coverage filtering, a target evaluation corpus matching the online business frequency and fully covering key business types is obtained. This effectively controls the corpus size and reduces computational resource consumption. The target evaluation corpus is input into an intelligent agent model for inference calculation, and the model response data is compared with the pre-set standard response for semantic similarity. Pass and failure cases are automatically determined, and offline evaluation results containing summaries of failure cases are generated. This allows the development team to grasp the overall quality level of the new version model and quickly locate defects.

[0155] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the continuous evaluation method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0156] This application also provides a continuous evaluation device; please refer to... Figure 4 The continuous evaluation device includes: Event monitoring module 10 is used to obtain pipeline event data from the continuous integration and continuous deployment pipeline based on a preset monitoring period; Event recognition module 20 is used to identify the event type of pipeline event data and obtain the event type recognition result; The offline evaluation module 30 is used to extract target evaluation data from a preset backup corpus when the event type identification result indicates that a new version tag release event exists, input the target evaluation data into the intelligent agent model for evaluation and batch processing, and obtain offline evaluation results. The data storage module 40 is used to write the offline evaluation results into the quality database to obtain quality status records.

[0157] The continuous evaluation apparatus provided in this application, employing the continuous evaluation method in the above embodiments, can solve the technical problem of how to shorten the evaluation feedback cycle after the release of the intelligent agent model. Compared with the prior art, the beneficial effects of the continuous evaluation apparatus provided in this application are the same as those of the continuous evaluation method provided in the above embodiments, and other technical features in the continuous evaluation apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0158] In one embodiment, the event identification module 20 is further configured to extract an event identifier field and a branch identifier field from the pipeline event data; The event identifier field is matched according to the preset event type mapping table to obtain the first matching result and the second matching result. The first matching result corresponds to the merge request event, and the second matching result corresponds to the new version tag release event. The branch status of the first matching result is validated based on the branch identifier field to obtain the branch validation result. When the branch verification result indicates that there is a merge request event in the main branch, an event type identification result is generated based on the first matching result and the second matching result.

[0159] In one embodiment, the offline evaluation module 30 is further configured to extract a set of candidate corpora from a preset safety corpus when the event type identification result indicates that a new version tag release event exists; Perform business scenario coverage verification on the candidate corpus set to obtain the coverage verification results; When the coverage verification results indicate that the coverage of key business scenarios meets the preset coverage threshold, the target evaluation corpus is selected from the candidate corpus set. The target evaluation corpus is input into the intelligent agent model for inference calculation to obtain the model response data; The model response data is compared with the preset standard response for consistency to obtain offline evaluation results.

[0160] In one embodiment, the offline evaluation module 30 is further configured to read the production behavior distribution model when the event type identification result indicates that a new version tag release event exists. The production behavior distribution model is used to characterize the frequency of occurrence of online business scenarios. Based on the production behavior distribution model, the pre-set full corpus is sorted by scene frequency to obtain the corpus frequency sorting results; High-frequency data is extracted from the preset full corpus based on the preset corpus size threshold and corpus frequency sorting results; The high-frequency corpus is screened to cover key business scenarios, resulting in a candidate corpus set.

[0161] In one embodiment, the offline evaluation module 30 is further used to extract response content from the model response data to obtain the actual response text; Extract standard content from the preset standard response to obtain the standard response text; Semantic similarity is calculated between the actual response text and the standard response text to obtain a semantic similarity value; When the semantic similarity value is greater than the preset similarity threshold, the model response data is marked as a pass case; When the semantic similarity value is less than or equal to the preset similarity threshold, the model response data is marked as a failure case, and a failure case summary is generated based on the failure cases. Offline evaluation results are generated based on the number of successful cases, the number of failed cases, and summaries of failed cases.

[0162] In one embodiment, the data storage module 40 is further configured to extract version tag identifiers, offline accuracy rates, and failure case information from the offline evaluation results; Generate a quality record index based on version tag identifiers; Determine the data storage partition based on the quality record index; Write offline accuracy, failure case information, and version tag identifiers to the data storage partition; The written data is stored and verified. If the storage verification passes, a quality status record is obtained.

[0163] In one embodiment, the event monitoring module 10 is further configured to generate scanning tasks based on a preset monitoring period; Send scan tasks to the continuous integration and continuous deployment pipeline; Receive raw event logs returned by the continuous integration and continuous deployment pipeline; The original event records are parsed to obtain pipeline event data.

[0164] This application provides a continuous evaluation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the continuous evaluation method in the first embodiment described above.

[0165] The following is for reference. Figure 5The diagram illustrates a structural schematic suitable for implementing the continuous evaluation device in the embodiments of this application. The continuous evaluation device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The continuous evaluation device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0166] like Figure 5 As shown, the continuous evaluation device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in ROM (Read Only Memory) 1002 or a program loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the continuous evaluation device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touch screens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the continuous evaluation device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show continuous evaluation devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0167] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0168] The continuous evaluation device provided in this application, employing the continuous evaluation method described in the above embodiments, can solve the technical problem of how to shorten the evaluation feedback cycle after the release of an intelligent agent model. Compared with the prior art, the beneficial effects of the continuous evaluation device provided in this application are the same as those of the continuous evaluation method provided in the above embodiments, and other technical features of the continuous evaluation device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0169] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0170] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0171] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the continuous evaluation method in the above embodiments.

[0172] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory or Flash Memory), optical fibers, CD-ROM (CD-Read Only Memory), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0173] The aforementioned computer-readable storage medium may be included in the continuous evaluation device; or it may exist independently and not be assembled into the continuous evaluation device.

[0174] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the continuous evaluation device, the continuous evaluation device: acquires pipeline event data from the continuous integration and continuous deployment pipeline based on a preset monitoring cycle; identifies the event type of the pipeline event data to obtain the event type identification result; when the event type identification result indicates the existence of a new version tag release event, extracts the target evaluation corpus from a preset safety corpus, inputs the target evaluation corpus into the intelligent agent model for batch evaluation, and obtains the offline evaluation result; and writes the offline evaluation result into the quality database to obtain a quality status record.

[0175] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including LAN (Local Area Network) or WAN (Wide Area Network)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0176] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0177] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0178] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described continuous evaluation method, thereby solving the technical problem of how to shorten the evaluation feedback cycle after the release of an intelligent agent model. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the continuous evaluation method provided in the above embodiments, and will not be repeated here.

[0179] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the continuous evaluation method described above.

[0180] The computer program product provided in this application can solve the technical problem of how to shorten the evaluation and feedback cycle after the release of an intelligent agent model. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the continuous evaluation method provided in the above embodiments, and will not be repeated here.

[0181] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A continuous evaluation method, characterized in that, The method includes: Based on a preset monitoring cycle, pipeline event data is obtained from the continuous integration and continuous deployment pipeline; The pipeline event data is subjected to event type identification to obtain the event type identification result; When the event type identification result indicates that a new version tag release event exists, the target evaluation corpus is extracted from the preset backup corpus, and the target evaluation corpus is input into the intelligent agent model for evaluation and batch processing to obtain offline evaluation results. The offline evaluation results are written into the quality database to obtain quality status records.

2. The method as described in claim 1, characterized in that, The step of identifying the event type of the pipeline event data and obtaining the event type identification result includes: Extract the event identifier field and branch identifier field from the pipeline event data; The event identifier field is matched according to a preset event type mapping table to obtain a first matching result and a second matching result, wherein the first matching result corresponds to a merge request event and the second matching result corresponds to a new version tag release event. The branch status of the first matching result is verified based on the branch identifier field to obtain the branch verification result. When the branch verification result indicates that the merge request event exists in the main branch, an event type identification result is generated based on the first matching result and the second matching result.

3. The method as described in claim 1, characterized in that, The step of extracting target evaluation data from a preset safety corpus when the event type identification result indicates the existence of a new version tag release event, and inputting the target evaluation data into an intelligent agent model for evaluation and batch processing to obtain offline evaluation results includes: When the event type identification result indicates the existence of a new version tag release event, a candidate corpus set is extracted from a preset backup corpus. Perform business scenario coverage verification on the candidate corpus set to obtain the coverage verification result; When the coverage verification result indicates that the coverage of key business scenarios meets the preset coverage threshold, the target evaluation corpus is selected from the candidate corpus set. The target evaluation corpus is input into the intelligent agent model for inference calculation to obtain model response data; The model response data is compared with the preset standard response for consistency to obtain the offline evaluation results.

4. The method as described in claim 3, characterized in that, The step of extracting a candidate corpus set from a preset safety corpus when the event type identification result indicates the existence of a new version tag release event includes: When the event type identification result indicates that a new version tag release event exists, the production behavior distribution model is read, which is used to characterize the frequency of occurrence of online business scenarios; Based on the production behavior distribution model, the preset full corpus is sorted by scene frequency to obtain the corpus frequency sorting result. High-frequency data is extracted from the preset full corpus based on the preset corpus size threshold and the corpus frequency sorting results; The high-frequency corpus is screened for coverage of key business scenarios to obtain a candidate corpus set.

5. The method as described in claim 3, characterized in that, The step of comparing the model response data with the preset standard response to obtain the offline evaluation result includes: The response content is extracted from the model response data to obtain the actual response text; The standard content of the preset standard response is extracted to obtain the standard response text; The semantic similarity between the actual response text and the standard response text is calculated to obtain a semantic similarity value; When the semantic similarity value is greater than a preset similarity threshold, the model response data is marked as a pass case; When the semantic similarity value is less than or equal to the preset similarity threshold, the model response data is marked as a failure case, and a failure case summary is generated based on the failure case. Offline evaluation results are generated based on the number of successful cases, the number of failed cases, and the summaries of the failed cases.

6. The method as described in claim 1, characterized in that, The step of writing the offline evaluation results into the quality database to obtain quality status records includes: Extract version tag identifiers, offline accuracy rates, and failure case information from the offline evaluation results; A quality record index is generated based on the version tag identifier; The data storage partition is determined based on the quality record index; Write the offline accuracy rate, the failure case information, and the version tag identifier into the data storage partition; The written data is stored and verified. If the storage verification passes, a quality status record is obtained.

7. The method as described in claim 1, characterized in that, The step of obtaining pipeline event data from the continuous integration and continuous deployment pipeline based on a preset monitoring period includes: Scanning tasks are generated based on a preset monitoring cycle; Send the scanning task to the continuous integration and continuous deployment pipeline; Receive the raw event logs returned by the continuous integration and continuous deployment pipeline; The original event records are parsed to obtain pipeline event data.

8. A continuous evaluation device, characterized in that, The device includes: The event monitoring module is used to obtain pipeline event data from the continuous integration and continuous deployment pipeline based on a preset monitoring period; The event identification module is used to identify the event type of the pipeline event data and obtain the event type identification result; The offline evaluation module is used to extract target evaluation data from a preset backup corpus when the event type identification result indicates that a new version tag release event exists, input the target evaluation data into the intelligent agent model for evaluation and batch processing, and obtain offline evaluation results. The data storage module is used to write the offline evaluation results into the quality database to obtain quality status records.

9. A continuous evaluation device, characterized in that, The device includes: a memory, a processor, and a continuous evaluation program stored in the memory and executable on the processor, the continuous evaluation program being configured to implement the steps of the continuous evaluation method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a continuous evaluation program, which, when executed by a processor, implements the steps of the continuous evaluation method as described in any one of claims 1 to 7.

Images