A model verification method

Abstract

This application relates to the field of artificial intelligence, and more particularly to a model verification method. The method includes: acquiring a model to be verified and its configuration information, the configuration information including at least the model's runtime environment information, parameter configuration information, and target task information; based on the configuration information, invoking the model to be verified to execute model training tasks and / or model inference tasks; generating a verification report for the model to be verified; the verification report including the configuration information, the training and / or inference process of the model to be verified, and the execution results of the target task. This method, by integrating the model verification process into a standardized online process from configuration acquisition and task execution to report generation, lowers the operational threshold for model verification, ensures the consistency and reproducibility of the verification process, and enables the verification results to serve as a strong basis for model capability evaluation, solution credibility assessment, or project delivery materials.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model verification method. Background Technology

[0002] With the rapid development of large-scale model technology, the demand for research and deployment of various intelligent models is increasing. During model development, it is typically necessary to verify the model's performance capabilities on specific tasks to assess whether it meets the expected performance requirements.

[0003] However, in the existing verification process, model selection, environment configuration and other steps usually need to be completed manually by developers in a distributed environment. The lack of management between the steps makes the verification process difficult to reproduce and the results difficult to trace, and it cannot effectively support the subsequent model evaluation or solution delivery needs. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a model validation method that integrates the model validation process into a standardized online workflow from configuration acquisition and task execution to report generation, thereby lowering the operational threshold for model validation.

[0005] In a first aspect, embodiments of this application provide a model verification method, the method comprising: obtaining a model to be verified and configuration information of the model to be verified, the configuration information including at least the runtime environment information, parameter configuration information, and target task information of the model to be verified; based on the configuration information, calling the model to be verified to execute a model training task and / or a model inference task; generating a verification report of the model to be verified; the verification report including the configuration information, the training and / or inference process of the model to be verified, and the execution result of the target task.

[0006] In one possible implementation, the method further includes: displaying an interactive interface, wherein the interactive interface includes a first display area and a second display area, the first display area displays at least one model and at least one configuration item, and the second display area is used to display information corresponding to the model or configuration item selected by the user; obtaining the model to be verified and the configuration information of the model to be verified, including: obtaining the model to be verified and the configuration information in response to the user's operation on the interactive interface.

[0007] In one possible implementation, the method further includes: displaying a running video of the target task on an interactive interface; playing the running video in response to user operations on the interactive interface; and controlling the playback state of the running video in response to user operations on video control controls.

[0008] In one possible implementation, the second display area also includes an interactive guidance control, which is used by the user to control the verification process. The first and second display areas are used to display content corresponding to the nodes in the verification process.

[0009] In one possible implementation, the interactive guidance control includes a trigger run control and a cancel task control. The method further includes: in response to a user's trigger operation on the trigger run control in the second display area, displaying the task running progress and / or the current running stage of the task in the first display area and the second display area, and displaying a cancel task control in the second display area, the cancel task control being used to cancel the currently executing target task.

[0010] In one possible implementation, the interactive guidance control further includes a re-execution control and a reconfiguration control; the method further includes: displaying the re-execution control and the reconfiguration control in a second display area; wherein the re-execution control is used to control the re-execution of the model training task and / or the model inference task based on the configuration information, and the reconfiguration control is used to control the display of multiple models and multiple configuration items in the first display area to re-acquire the new model to be verified selected by the user and / or the new configuration information.

[0011] In one possible implementation, based on configuration information, the model to be verified is invoked to perform model training tasks and / or model inference tasks, including: generating a training and inference task package based on the configuration information; in response to a user's trigger operation on the trigger run control, adding the training and inference task package to a task queue, and allocating computing resources to the training and inference task package based on the task queue; according to the training and inference task package, using the allocated computing resources to load the model to be verified and its runtime environment, and invoking the model to be verified to perform model training tasks and / or model inference tasks.

[0012] In one possible implementation, after calling the model to be verified to perform model training and / or model inference tasks based on configuration information, the method further includes: recording the running video of the target task; extracting at least one key frame based on the running video; wherein different key frames correspond to different running stages of the target task; and determining whether the target task has been achieved based on at least one key frame to obtain the execution result of the target task.

[0013] In one possible implementation, generating a verification report for the model to be verified includes: obtaining operational metrics and task backtracking information for the model training task and / or model inference task; wherein the operational metrics include at least the execution time, resource consumption, and task success rate of the model training task and / or model inference task, and the task backtracking information includes at least the task execution logs of the model training task and / or model inference task; and generating a verification report based on the configuration information, the execution results of the target task, the operational metrics, and the task backtracking information.

[0014] In one possible implementation, the method further includes: determining the failure record of the target task based on the task execution log when the target task is not achieved; or, the method further includes: obtaining historical failure records related to the target task and associating the historical failure records with the verification report; the historical failure records are failure records generated when the target task is not achieved due to differences in the model to be verified, the running environment and / or parameter configuration during the historical verification process.

[0015] To achieve the above objectives, in a second aspect, embodiments of this application provide a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform a method as provided in any of the optional implementations of the second aspect described above.

[0016] To achieve the above objectives, in a third aspect, embodiments of this application provide a computer program product that, when a computer reads and executes the computer program product, causes the method provided by any of the optional implementations of the second aspect above to be executed.

[0017] As described above, this application provides a model verification method. The method includes: acquiring a model to be verified and its configuration information, wherein the configuration information includes at least the model's runtime environment information, parameter configuration information, and target task information; based on the configuration information, invoking the model to be verified to execute model training tasks and / or model inference tasks; generating a verification report for the model to be verified; the verification report includes the configuration information, the training and / or inference process of the model to be verified, and the execution result of the target task. This method, by integrating the model verification process into a standardized online process from configuration acquisition and task execution to report generation, lowers the operational threshold for model verification, ensures the consistency and reproducibility of the verification process, and enables the verification results to serve as a strong basis for model capability evaluation, solution credibility assessment, or project delivery materials. Attached Figure Description

[0018] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram illustrating the application of the embodied intelligent model and robotic device provided in the embodiments of this application; Figure 2 This is a schematic diagram of the application scenario architecture of the model verification system provided in the embodiments of this application; Figure 3 A schematic flowchart illustrating the model verification method provided in this application embodiment; Figure 4 A schematic diagram of a first type of user interface provided in an embodiment of this application; Figure 5 A schematic diagram of a second user interface provided in an embodiment of this application; Figure 6 A schematic diagram of a third user interface provided in an embodiment of this application; Figure 7 A schematic diagram of a fourth user interface provided in an embodiment of this application; Figure 8 This is a schematic diagram of the state transition of a model training task or a model inference task provided in the embodiments of this application.

[0020] Among them, 10-Embodied Intelligent Model; 20-Robot Equipment; 100-Verification Platform; 200-User Terminal; 300-Backend Server; 101-Third Display Area; 102-First Display Area; 103-Second Display Area; 1021-Model Card; 1022-Running Environment Card; 1023-Task Card; 1024-Video Control Control; 1031-Interactive Guidance Control; 1031a-First Jump Control; 1031b-Second Jump Control; 1031c-Trigger Run Control; 1031d-Cancel Task Control; 1031e-Re-execute Control; 1031f-Reconfigure Control. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are all within the protection scope of this application.

[0022] In the following description, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0023] Furthermore, in this application, directional terms such as "upper," "lower," "inner," and "outer" are defined relative to the indicated placement of the components in the accompanying drawings. It should be understood that these directional terms are relative concepts, used for relative description and clarification, and can change accordingly depending on the placement of the components in the accompanying drawings.

[0024] The intelligent model in this application refers to an artificial intelligence model capable of receiving input data and generating output results through computation. An embodied AI model is a type of intelligent model that refers to an intelligent agent (such as a robot or robotic arm) with a body capable of interacting with the physical world. Its goal is to bring Artificial General Intelligence (AGI) from the digital world into the physical world, enabling robots to interact with and perceive their environment and perform tasks like humans. Embodied AI models integrate perception, cognition, decision-making, and action capabilities, enabling intelligent agents to perceive, learn, and dynamically interact with their environment in the physical environment.

[0025] To enable intelligent models to perform tasks, model validation is necessary. Model validation can specifically include model training and / or model inference to evaluate the model's performance on specific tasks, such as robotic arm grasping, dual-arm collaboration, mobile chassis inspection, humanoid robot service, and language command execution.

[0026] Existing model validation processes typically involve developers manually selecting models, configuring simulation environments, preparing task data, and performing inference or training in their local environments. However, in existing validation processes, steps such as model selection and environment configuration usually require developers to manually complete these tasks in distributed environments. The lack of management between these steps makes the validation process difficult to reproduce, results difficult to trace, and fails to effectively support subsequent model evaluation or solution delivery requirements.

[0027] In real-world embodied intelligence projects, enterprise needs or business scenarios often require service providers to offer verifiable model execution results as evidence of solution feasibility. If the model verification process and results cannot be fully traced and reproduced, service providers will find it difficult to demonstrate the model's true performance in specific tasks to the client; conversely, the client will struggle to objectively evaluate the model's capabilities based on fragmented evidence. This lack of unified management and traceability in verification methods makes verification results difficult to accept in practice.

[0028] Based on this, this application provides a model verification method that can be applied to the verification process of intelligent models, especially embodied intelligent models.

[0029] Figure 1 This is a schematic diagram illustrating the application of the embodied intelligent model and robotic device provided in the embodiments of this application.

[0030] like Figure 1As shown, the embodied intelligent model 10 is an intelligent model applied to the robot device 20. For example, the embodied intelligent model 10 can be used to receive sensor data (such as visual images, depth information, tactile signals, force feedback, etc.) from the robot device 20 to perceive and understand the physical environment; and to perform task planning and action decisions based on the perception results and target task instructions, generating action sequences or control instructions; it can also be used to convert the control instructions generated by the decision module into motion instructions executable by the robot device 20, controlling the robot device 20 to complete specific actions.

[0031] The robot device 20 serves as the physical carrier of the embodied intelligent model 10, executing the action commands generated by the embodied intelligent model 10 and interacting with the physical environment. In practical applications, the robot device 20 can be a robotic arm, a mobile chassis, a humanoid robot, an unmanned aerial vehicle (UAV), a smart home device, etc. Among these, the robotic arm can perform tasks such as grasping, placing, and assembling; the mobile chassis can perform humanoid tasks such as service, companionship, and operation; and the bionic robot can perform tasks such as detection and operation in special environments.

[0032] Furthermore, the embodied intelligent model 10 and the robot device 20 can interact via a communication interface. The embodied intelligent model 10 receives sensor data (such as images, point clouds, force sensing, pose, etc.) from the robot device 20, calculates it, and outputs control commands (such as joint angles, end effector positions, movement speed, etc.). After executing the control commands, the robot device 20 feeds back the execution results and new perception data to the embodied intelligent model 10.

[0033] In practical applications, the embodied intelligent model 10 can be deployed on the local computing unit of the robot device 20 (such as an embedded system, edge computing device, etc.), or on a cloud server or remote computing node, and interact with the robot device 20 through a communication network. This application embodiment does not specifically limit the deployment method of the embodied intelligent model 10.

[0034] Furthermore, the model verification method provided in this application embodiment can be used to verify the task execution capability of the above-mentioned embodied intelligent model 10 in the robot device 20 or simulation environment online.

[0035] Figure 2 This is a schematic diagram of the application scenario architecture of the model verification system provided in the embodiments of this application.

[0036] like Figure 2 As shown in the figure, this application embodiment can provide a model verification system, which can be used to implement a model verification method. The model verification system may include a verification platform 100, a user terminal 200, and a backend server 300.

[0037] User terminal 200: Users can access the verification platform through user terminal 200. User terminal 200 can be a personal computer (PC), laptop, tablet, smartphone, or other electronic device with network communication capabilities. User terminal 200 runs a browser or client application for interacting with verification platform 100.

[0038] Verification platform 100: Deployed on a cloud server or a local server. For example, verification platform 100 can provide an interactive interface to user terminal 200, receive user configuration operations and instructions, and display verification progress and results to the user. It can also invoke computing nodes (specifically, backend server 300) to enable backend server 300 to execute core logic such as model loading, environment initialization, task scheduling, resource allocation, training and push execution, and result evaluation. Backend server 300 can be a cloud server or a local server; this embodiment does not specifically limit its use.

[0039] In some implementations, the model verification system of this application embodiment may further include a robotic device. The robotic device can serve as a physical environment for executing the target task, used to verify the embodied intelligent model to be verified in a real physical scene, so as to verify the model's task execution capability in a real environment.

[0040] The configuration of the robot device can be adapted to the supporting ontology type of the model to be verified. For example, if the model to be verified supports a robotic arm, the robot device is a robotic arm; if the model to be verified supports a mobile chassis, the robot device is a mobile chassis; if the model to be verified supports a humanoid robot, the robot device is a humanoid robot. The backend server 300 can establish a data connection with the robot device through a communication network, send motion commands to the robot device and receive sensor feedback data, realizing online training and verification in a real ontology environment.

[0041] It should be understood that Figure 2 The application scenario architecture shown is merely an exemplary illustration of the embodiments of this application and does not constitute a limitation on the application scenarios of the embodiments of this application.

[0042] Figure 3 This is a flowchart illustrating the model verification method provided in an embodiment of this application.

[0043] like Figure 3 As shown, the model verification method provided in this application embodiment may include the following steps S100-S300.

[0044] S100: Obtain the model to be verified and its configuration information.

[0045] The configuration information includes at least the runtime environment information of the model to be verified, the parameter configuration information, and the task information of the target task.

[0046] Specifically, a model to be validated can refer to an artificial intelligence model that can receive input data and generate output results through computation. The specific type of model to be validated can be selected according to the actual application scenario, including but not limited to: computer vision models, natural language processing models, speech processing models, multimodal models, reinforcement learning models, large language models, diffusion models, recommendation models, time series prediction models, scientific computing models, or embodied intelligence models, etc.

[0047] In some implementations, the model to be verified can be an embodied intelligence model or an intelligent system containing an embodied intelligence model; however, this application does not specifically limit this.

[0048] Furthermore, the model to be validated can include model information such as model identifier, model version, model type, supported tasks, supported simulation environments, download address, and model capability tags. The model identifier uniquely identifies the model to be validated, and the model version identifies the iterative version of the model. For example, the model to be validated could be an embodied intelligence model with the model identifier "DM0-base" and the model version v1.0. The model type indicates the model's architecture category (such as a Vision-Language-Action Model (VLA), a Large Language Model (LLM), etc.). Supported tasks refer to the types of tasks the model to be validated can perform. Supported simulation environments can identify the runtime environment to which the model to be validated is adapted. The download address can be used to obtain the weight file or code repository of the model to be validated. Model capability tags can summarize the core capabilities or characteristics of the model to be validated.

[0049] Furthermore, the runtime environment information in the configuration includes environment identifier, environment version, supported ontology, task set, interface specification, and resource requirements. The environment identifier is used to uniquely identify the runtime environment, the environment version indicates the iterative version of the runtime environment, the supported ontology indicates the types of robot devices supported by the runtime environment, the task set indicates specific tasks built into the runtime environment, the interface specification defines the data interaction protocol between the model to be verified and the runtime environment, and the resource requirements identify the specifications of computing resources required by the runtime environment.

[0050] It is worth noting that the operating environment referred to in the embodiments of this application can be either a simulation environment or an on-the-ground environment. A simulation environment refers to a virtual simulation environment running on computing nodes, used to simulate the physical world and perform target tasks without a physical robot. Examples include embodied intelligence simulation benchmark environments such as LIBERO, CALVIN, SimplerEnv, ManiSkill2, and RoboTwin. An on-the-ground environment refers to a real physical device environment, including but not limited to robotic arms, mobile chassis, and humanoid robots.

[0051] The configuration information can include training epochs, inference threshold, random seed, resource specifications, input data summary, and running mode. Training epochs control the number of rounds in which the model performs a complete task interaction in the simulation environment during training. The inference threshold controls the sensitivity of the model's action output during inference. The random seed ensures the reproducibility of the training or inference process. Resource specifications identify the configuration of computing nodes required to perform the target task. The input data summary identifies the dataset information used for training or inference. The running mode distinguishes between training mode, inference mode, and fine-tuning mode.

[0052] The task information in the configuration includes the task name, task number, task instructions, target result, decision criteria, and the dataset or benchmark set to which the task belongs. It should be noted that the target task is a specific task instance selected by the user to evaluate the performance capabilities of the model to be validated. The task name is used to represent the Chinese description of the target task, the task number is used to uniquely identify the target task, the task instructions describe the specific execution requirements of the target task, the target result defines the expected state when the target task is achieved, the decision criteria define the rules for determining the completion state of the target task, and the dataset or benchmark set to which the task belongs identifies the source of the target task.

[0053] In this embodiment, the methods for obtaining the model to be verified and its configuration information include manual selection by the user or automatic recommendation based on the user's needs profile. The specific acquisition steps will be detailed below and will not be repeated here.

[0054] S200: Based on the configuration information, call the model to be verified to perform model training tasks and / or model inference tasks.

[0055] This application embodiment can respond to user-triggered operations by invoking the model to be verified to perform model training and / or model inference tasks. Specifically, this application embodiment can bind the user-selected model information, runtime environment information, task information, and parameter configuration information to generate a training and inference task package, and submit the training and inference task package to a task queue. Based on the task queue, computing resources are allocated to the training and inference task package according to the resource requirements in the training and inference task package and the current availability of computing resources. After successful allocation of computing resources, this application embodiment can load the model to be verified from the model repository according to the model identifier, model version, and download address in the training and inference task package; then, according to the environment identifier and environment version in the training and inference task package, the simulation environment is initialized or the ontology environment is connected; according to the task information in the training and inference task package, the corresponding task data and judgment conditions are loaded; and according to the parameter configuration information in the training and inference task package, the training rounds, inference thresholds, random seeds, and other runtime parameters are set. After completing the above preparations, at least one round of model training and / or model inference tasks is executed.

[0056] During the execution of model training or inference tasks, embodiments of this application can also record process data such as running logs, running videos, trajectory files, and running metrics for use in generating subsequent verification reports. Running videos can be used to record the visual performance of the model under test during model training and / or model inference tasks to achieve the target task. For example, when the target task is a robotic arm grasping an object, the running video can record the complete action process of the robotic arm moving from its initial position to the target object, grasping the object, moving the object to the target position, and releasing it, allowing users to intuitively understand the actual execution of the model. Trajectory files can be used to record the pose change path during the model's action execution. For example, when the target task is a robotic arm grasping an object, the trajectory file can record the sequence data of the robotic arm's end-effector position coordinates and attitude angles changing over time, which can be used for subsequent trajectory analysis, path planning, and collision detection. Running metrics can be used to quantitatively evaluate the model's task execution performance, such as task execution time, resource utilization, and action success rate, serving as a basis for performance evaluation when generating subsequent verification reports.

[0057] It should also be noted that, when the model to be verified is an embodied intelligence model, the model training task may include the process of the embodied intelligence model performing actions in a simulation environment or ontology environment, receiving environmental feedback, accumulating interaction experience, and adjusting model parameters accordingly; the model inference task may include the process of the embodied intelligence model perceiving the environmental state, generating action instructions, and executing actions in a simulation environment or ontology environment based on the learned strategy to achieve the target task. In the embodiments of this application, both the model training task and the model inference task cover the complete interactive closed loop of environment initialization, state perception, action generation, environment execution, and result determination, rather than just referring to the narrow parameter update or forward calculation steps.

[0058] In some implementations, the model training task and the model inference task can be collectively referred to as the verification task.

[0059] S300: Generate a validation report for the model to be validated; the validation report includes configuration information, the training and / or inference process of the model to be validated, and the execution results of the target task.

[0060] Specifically, embodiments of this application can determine whether a target task has been achieved based on operation logs, operation videos, and operation metrics, thus obtaining a success or failure result for the target task. For example, for target tasks involving object manipulation, embodiments of this application can identify keyframes in the operation video to determine whether the target object has reached the target position and whether the operating components are in the expected state, thereby determining whether the target task has been successful or failed.

[0061] After determining the execution result of the target task, this embodiment of the application can further integrate the configuration information, the training and / or inference process of the model to be verified, and the execution result of the target task to generate a verification report of the model to be verified. The training and / or inference process of the model to be verified can be specifically represented in the form of running videos, trajectory files, running logs, and running metrics.

[0062] As can be seen, the verification report contains complete information about this verification. This application embodiment can archive all information generated during the verification process into a structured verification report, so that the verification results not only include the final success or failure status, but also include a complete chain of evidence that is traceable and reproducible, which is convenient for subsequent model capability evaluation, scheme credibility assessment or project delivery.

[0063] In practical applications, validation reports can be used as structured data for external systems, including at least a model capability evaluation system, a solution generation system, a requirements matching system, or a delivery management system. For example, a model capability evaluation system can quantitatively score the model using indicators such as task success rate, runtime, and resource consumption from the validation report; a solution generation system can assess the feasibility of solutions based on the model version, environment configuration, and task execution results in the validation report; a requirements matching system can match suitable business requirements based on the model capability tags and task success rate in the validation report; and a delivery management system can incorporate the runtime videos and validation conclusions from the validation report as part of the delivery and acceptance materials. This allows for the acquisition of the required model performance data without re-executing the validation, achieving one-time generation and multiple reuse of validation results.

[0064] As described above, this application provides a model verification method. The method includes: acquiring a model to be verified and its configuration information, wherein the configuration information includes at least the model's runtime environment information, parameter configuration information, and target task information; based on the configuration information, invoking the model to be verified to execute model training tasks and / or model inference tasks; generating a verification report for the model to be verified; the verification report includes the configuration information, the training and / or inference process of the model to be verified, and the execution result of the target task. This method, by integrating the model verification process into a standardized online process from configuration acquisition and task execution to report generation, lowers the operational threshold for model verification, ensures the consistency and reproducibility of the verification process, and enables the verification results to serve as a strong basis for model capability evaluation, solution credibility assessment, or project delivery materials.

[0065] The user interaction process will be described in detail below with reference to the accompanying diagram.

[0066] Specifically, the method provided in this application embodiment further includes the following step S401.

[0067] S401: Display the interactive interface. The interactive interface may include a third display area 101, which can display the current node of the verification process.

[0068] Understandably, the interactive interface is a graphical interface that interacts with the user, used to receive configuration operations and command inputs from the user regarding the model validation process, and to display the validation progress and results to the user.

[0069] It is worth noting that, from the perspective of user interaction or model verification process, the model verification method provided in this application embodiment may include four process nodes (or stages): Node 1: Selecting the model and environment (i.e., specifying the model to be verified and the running environment); Node 2: Selecting the task (i.e., specifying the target task); Node 3: Running inference (corresponding to the process of executing the model training task or the model inference task); Node 4: Viewing the results. These four nodes can be triggered by the user.

[0070] Figure 4 This is a schematic diagram of a first user interface provided in an embodiment of this application.

[0071] like Figure 4 As shown, in this embodiment of the application, the current node of the verification process can be displayed in a highlighted form in the third display area 101. Figure 4 The verification process shown in the example is at node one.

[0072] For further details, please refer to [link / reference]. Figure 4 The interactive interface may also include a first display area 102 and a second display area 103. The first display area 102 displays at least one model and at least one configuration item, including at least environment configuration items, task configuration items, and parameter configuration items. The second display area is used to display information corresponding to the model or configuration item selected by the user, such as model description information, environment description information, task description information, or parameter description information. The model displayed in the first display area may include models pre-configured by the system or models submitted by the user.

[0073] In this embodiment, both the first display area 102 and the second display area 103 can serve as interactive areas to respond to user operations. Users can browse and select models or configuration items in the first display area 102, and the second display area 103 will simultaneously display detailed information corresponding to the selected content to guide the user in obtaining configuration information.

[0074] Furthermore, step S100 may also include the following step S101.

[0075] S101: Respond to user actions on the interactive interface to obtain the model to be verified and configuration information.

[0076] In this embodiment, the user can select the model to be verified and its configuration information by performing a selection operation in the first display area 102. The following is in conjunction with... Figure 4 The process of users completing configuration through an interactive interface is illustrated with an example.

[0077] like Figure 4As shown, the first display area 102 can display multiple model cards 1021. Users can click on the model card 1021 to select the model to be verified. In response to the user's click operation, the model card 1021 presents the visual effect of being selected (such as a highlighted border, filled background color, or check mark), and the second display area 103 simultaneously displays the introductory information of the selected model, including the model name, model description, model characteristics, etc.

[0078] Furthermore, the first display area 102 can also display multiple environment configuration items, which can specifically be runtime environment cards 1022. In other words, a card is a form of representation of an environment configuration item. Each runtime environment card 1022 can correspond to an available simulation environment or ontology environment. The runtime environment card 1022 can display brief information such as the environment name, environment version, supported task types, and compatible ontology. Users can select the required runtime environment for verification from multiple candidate environments by clicking on the runtime environment card 1022. In response to the user's click operation, the runtime environment card 1022 displays the selected visual effect (such as a highlighted border, filled background color, or a checkmark).

[0079] In some implementations, in response to a user's click on the runtime environment card 1022, embodiments of this application can simultaneously display a detailed description and configuration information of the runtime environment in the second display area 103.

[0080] It should also be noted that the first display area 102 can display multiple parameter configuration items, which are used by the user to configure various operating parameters of the model to be validated. In some implementations, the user can select a parameter configuration item in the first display area 102 to determine various operating parameters. In other implementations, the values ​​of the parameter configuration items can be bound to the model to be validated, that is, different models to be validated are pre-configured with a set of parameter values ​​that are suitable for them (such as recommended training epochs, inference thresholds, and random seeds). When the user selects a model, the parameter configuration items are automatically filled with the recommended configuration values ​​for that model, and the user can directly trigger the run without manual adjustment, reducing configuration complexity.

[0081] In some implementations, embodiments of this application may also display specific values ​​or details of parameters such as training rounds, inference thresholds, random seeds, resource specifications, and operating modes in the second display area 103.

[0082] For further details, please refer to [link / reference]. Figure 4 The second display area 103 may also include an interactive guidance control 1031, which can be used by the user to control the verification process, such as switching between four process nodes. The first display area 102 and the second display area 103 can be used to display content corresponding to the nodes of the verification process.

[0083] Specifically, the interactive guidance control 1031 may include a first jump control 1031a, the display text of which may be "Next step: Select task". In response to the user's triggering operation on the first jump control 1031a, this embodiment of the application may display the content corresponding to the next process node (select task node).

[0084] Figure 5 This is a schematic diagram of a second user interface provided in an embodiment of this application.

[0085] like Figure 5 As shown, in response to the user's triggering operation on the jump control 1031a, this embodiment of the application can display the jumped process node in the third display area 101, that is, display the display content corresponding to node two (e.g., Figure 5 (The text includes phrases such as "select task" and "choose task scenario"). Furthermore, in this embodiment, multiple task configuration items for configuring a target task can be displayed in the first display area 102. Specifically, each task configuration item can be a task card 1023, with each task card 1023 corresponding to an available target task. The task card 1023 can display brief information such as the task name, task number, task instructions, and / or target result, allowing users to quickly understand the basic content of each task.

[0086] Users can select a target task from multiple candidate tasks by clicking on task card 1023. In response to the user's click on task card 1023, the task card 1023 displays a selected visual effect (such as a highlighted border, highlighted background, or checkmark), and the second display area 103 can simultaneously display the task information of the target task. For example, Figure 5 The task card currently selected, 1023, has the task name "Put the yellow and white mug in the microwave and close it", the task number is "libero_10·task 9", the task instruction is "put the yellow and white mug in the microwave and close it", and the target result is "The yellow and white mug is inside the microwave oven and the microwave oven door is closed".

[0087] It should be noted that task card 1023 specifically refers to the task that is compatible with the selected model to be verified and the operating environment. That is, the first display area 102 displays task configuration items that are compatible with the currently selected model and environment, so as to avoid the user selecting an incompatible task configuration.

[0088] It should be added that, Figure 5The textual descriptions such as "select inference task" shown are merely exemplary interface expressions used to facilitate user understanding of the current operation node, prompting the user to select the embodied intelligent task (i.e., the target task) to be executed at this node. These textual descriptions are not conceptually equivalent to "model training task" or "model inference task" in the embodiments of this application. "Select inference task" describes the operation goal of the current node from the user's perspective, while "model training task / model inference task" describes the actual type and operating mode of the task from a technical implementation perspective. Figure 5 The interface text is for illustrative purposes only and does not constitute a limitation on the technical solution of this application.

[0089] In some implementations, the target task can be manually selected by the user or automatically recommended by the verification platform 100 based on the user's requirement profile. The requirement profile may include the user's historical verification records, model capability tags, etc. In this embodiment, the target task can be matched and recommended to the user based on the requirement profile, thereby reducing the configuration threshold and improving verification efficiency.

[0090] For further details, please refer to [link / reference]. Figure 5 The interactive guidance control 1031 may include a second jump control 1031b, and the display text of the second jump control 1031b may be "Reselect Model". In this embodiment, the second jump control 1031b can be displayed in the second display area 103. In response to the user's trigger operation on the second jump control 1031b, this embodiment can display the content corresponding to the previous process node (select model and environment node) in the first display area 102, the second display area 103 and the third display area 101.

[0091] Furthermore, the interactive guidance control 1031 may also include a trigger run control 1031c, the display text of which may be "Run Inference". In this embodiment, the trigger run control 1031c can be displayed in the second display area 103. In response to a user's trigger operation on the trigger run control 1031c, this embodiment can display the content corresponding to the next process node (run inference node).

[0092] Figure 6 This is a schematic diagram of a third user interface provided in an embodiment of this application.

[0093] like Figure 6 As shown, in response to the user's trigger operation on the trigger control 1031c, this embodiment of the application can display the process node after the jump in the third display area 101, that is, display the content corresponding to the third display node (e.g., Figure 6 The text includes phrases such as "run the reasoning" and "view the reasoning process".

[0094] Furthermore, in response to the user's trigger operation on the trigger control 1031c, this embodiment of the application can issue a task execution instruction to the backend server 300. The backend server 300 loads the model to be verified, initializes the simulation environment or connects to the ontology environment according to the instruction, and calls the model to be verified to execute the task.

[0095] Furthermore, the method provided in this application embodiment may also include the following step S501.

[0096] S501: In response to the user's trigger operation on the trigger run control 1031c in the second display area 103, the task running progress and / or the current running stage of the task are displayed in the first display area 102 and the second display area 103.

[0097] The "running phase" refers to the specific stage in the execution of a model training or inference task, which may include, but is not limited to, model loading, environment initialization, data loading, action inference, simulation execution, and target determination. This embodiment can obtain the running phase through interaction with the backend server 300 and process it into user-side display information. The current running phase and task progress can be simplified information after a general description of the running phase. By viewing the current running phase and task progress, users can quickly understand the specific stage of the model training or inference task being executed, facilitating the determination of any anomalies or stuttering. In some implementations, the current running phase can be represented by information such as "system scheduling" or "action inference," and the task progress can be displayed as a progress bar, which can be a continuous progress bar or a segmented progress bar, allowing users to clearly understand which specific stage the task is currently executing. The task progress may also include information such as progress percentage and estimated remaining time (countdown), but this embodiment does not specifically limit this.

[0098] For example, such as Figure 6 As shown, the backend sequentially goes through several sub-stages, including model loading, environment initialization, and data loading. The frontend doesn't need to display each stage individually; instead, it can be uniformly summarized as "System scheduling in progress, estimated completion in 4 seconds" or "Inference in progress, results will be automatically displayed in 4 seconds," while presenting the overall completion progress as 20% of the inference progress. As the backend progresses to the action inference stage, the frontend's current task execution stage can be updated to "System scheduling in progress, estimated completion in 2 seconds" or "Inference in progress, results will be automatically displayed in 2 seconds," with the task progress gradually increasing from 20% to 40%. This simplifies the information presented about the complex backend execution stages, making the frontend display more concise and intuitive.

[0099] In some implementations, the content displayed in the first display area 102 and the second display area 103 corresponds to the same task running status, but the presentation format can be different, such as text or graphics, so that users can compare and view between different display areas and fully grasp the task execution status.

[0100] Furthermore, the interactive guidance control 1031 may also include a cancel task control 1031d, the display text of which may be "Cancel Task", and the cancel task control 1031d is used to cancel the currently executing model training task or model inference task.

[0101] Furthermore, the method provided in this application embodiment may also include the following step S502.

[0102] S502: In response to the user's trigger operation on the trigger run control 1031c in the second display area 103, the cancel task control 1031d is displayed in the second display area.

[0103] Furthermore, the method provided in this application embodiment may also include the following step S503.

[0104] S503: In response to the user's trigger operation on the cancel task control 1031d in the second display area 103, a prompt message indicating that the task has been canceled is displayed in the first display area 102 and the second display area 103.

[0105] In this way, users can actively cancel tasks at any time during the execution of the target task, improving the flexibility and controllability of the verification process.

[0106] Figure 7 This is a schematic diagram of a fourth user interface provided in an embodiment of this application.

[0107] like Figure 7 As shown, in response to the completion or termination of the model training task or model inference task, this embodiment of the application can display the content corresponding to node four (i.e., the view result node) in the third display area 101 (e.g., ...). Figure 7 The text includes phrases such as "View Results" and "View Reasoning Results" to indicate to the user that the verification process has entered the results viewing stage.

[0108] And, see also Figure 7 The method provided in this application embodiment may further include the following steps S504-S505.

[0109] S504: Display the running video of the target task on the interactive interface.

[0110] Specifically, in this embodiment, the running video can be displayed in the first display area 102. Specifically, after the model training task or model inference task is completed or terminated, this embodiment can display the running video in the first display area 102 (such as displaying a video cover or the first frame), allowing the user to watch the running video. By watching the running video, the user can intuitively understand the model's actual performance on the target task, including the continuity of actions, the efficiency of task execution, and whether any abnormal behavior occurs.

[0111] S505: Responds to user operations on the interactive interface to play and run the video, and responds to user operations on the video control control 1024 to control the playback status of the running video.

[0112] Specifically, users can control the playback and pause of the running video by clicking the play / pause button on the running video screen in the first display area 102. The video control control 1024 may also include a progress bar, a speed control button, and a full-screen button. Users can drag the progress bar to jump to any point in time in the video to view the model's execution status at a specific moment; they can adjust the video playback speed (such as 0.5x, 1x, 2x, etc.) using the speed control button to quickly browse or carefully examine key actions; and they can zoom in to full-screen display using the full-screen button for close observation of the model's execution details. Through the video control control 1024, users can flexibly view various segments of the running video and gain a comprehensive understanding of the model's performance during the target task execution process.

[0113] It is understood that embodiments of this application may also display other forms of result visualization, such as trajectory playback, keyframes, action sequences, indicator tables, or report summaries, or one or more of these. Users can choose different visualization formats according to their needs, such as viewing the complete execution process by running a video, analyzing action strategies through trajectory playback, comparing multiple verification results through indicator tables, or quickly understanding verification conclusions through report summaries.

[0114] See also Figure 7 The interactive guidance control 1031 also includes a re-execution control 1031e and a reconfiguration control 1031f. The display text of the re-execution control 1031e can be "Try again", and the display text of the reconfiguration control 1031f can be "Reselect". The method provided in this application embodiment may further include the following step S506.

[0115] S506: Display the re-execution control 1031e and the reconfiguration control 1031f in the second display area 103; wherein, the re-execution control 1031e is used to control the re-execution of the model training task and / or model inference task based on the configuration information, and the reconfiguration control 1031f is used to control the display of multiple models and multiple configuration items in the first display area to re-acquire the new model to be verified selected by the user and / or the new configuration information.

[0116] It is understood that step S506 can be executed after the model training task or model inference task is completed or actively canceled by the user. That is to say, as long as the target task is terminated, this embodiment of the application can provide the user with two entry points in the second display area 103: re-execution and reconfiguration, so that the user can perform the next round of operations without leaving the current interface.

[0117] Furthermore, the method provided in this application embodiment may also include the following step S507.

[0118] S507: In response to the user's trigger operation on the re-execution control 1031e, the display content corresponding to the current process node is displayed in the third display area 101; and the task running progress and / or the current running stage of the task are displayed in the first display area 102 and the second display area 103; and the task cancellation control 1031d is displayed in the second display area 103.

[0119] It is understandable that the user's triggering operation of re-execution control 1031e can cause the process node to jump to node three. In this embodiment of the application, the display content corresponding to node three (such as text content such as "run reasoning, view reasoning process") can be displayed in the third display area 101.

[0120] Furthermore, the method provided in this application embodiment may also include the following step S508.

[0121] S508: In response to the user's trigger operation on the reconfiguration control 1031f, the display content corresponding to the current process node is displayed in the third display area 101, and at least one model and at least one configuration item are displayed in the first display area 102.

[0122] Understandably, in response to the user's triggering operation on the reconfiguration control 1031f, the verification process jumps to node one. In this embodiment, the display content corresponding to node one (such as text content like "Select Model, Model / Simulation") can be displayed in the third display area 101. This embodiment can clear the currently configured information and wait for the user to reconfigure the model, environment, task, and parameters before triggering execution again.

[0123] In some implementations, the method provided in this application embodiment may further include the following step S509.

[0124] S509: In response to the user's trigger operation on the cancel task control 1031d, a re-execution control 1031e and a reconfiguration control 1031f are provided in the second display area 103.

[0125] In this way, users can choose to re-execute the target task based on the current configuration, or return to node one to retrieve the configuration information again.

[0126] It should be further noted that the various user interface diagrams provided in the embodiments of this application are merely illustrative examples, used to intuitively demonstrate the layout and functional area division of the interactive interface. The text content, control names, model names, layout proportions, and visual styles displayed therein do not limit the solution. In actual applications, the user interface can be adaptively adjusted according to product requirements, user habits, and platform specifications.

[0127] As can be seen, based on the method provided in the embodiments of this application, users can complete operations such as configuration adjustment, re-execution and result comparison in the interactive interface without having to switch repeatedly between different pages or tools, making the verification process smoother and more efficient, and improving the continuity and operational efficiency of the verification process.

[0128] Furthermore, in the embodiments of this application, step S200 may specifically include the following steps S201-S203.

[0129] S201: Generate training and push task packages based on configuration information.

[0130] This application embodiment can bind the model to be verified and its corresponding configuration information to generate a training and push task package. Furthermore, this application embodiment can generate a unique training and push task identifier (e.g., training and push task ID) for the training and push task package, used for unique traceability and status management of this verification task in subsequent processes.

[0131] In some implementations, the training and push task package may also include input data configuration and output data configuration. The input data configuration specifies the data sources required for the execution of the target task, such as training datasets, validation datasets, simulation scenario configuration files, or specific test samples uploaded by the user, ensuring that the input data used in each validation is traceable and reproducible. The output data configuration specifies the output content and storage format to be generated after validation, such as running videos, trajectory files, running logs, running metrics, and validation reports, to meet the needs of result viewing and archiving in different scenarios.

[0132] Furthermore, the training and push task package can also include external reference IDs, such as requirement IDs, solution IDs, or evaluation IDs, to associate this verification task with specific requirements, solutions, or evaluation items in external business systems. This facilitates subsequent data linkage and result traceability between different systems. Through this approach, the training and push task package not only contains all the configuration information required to execute the target task but also includes input / output specifications and external association information, making the execution, reproduction, and cross-system traceability of the verification task possible.

[0133] For example, the field structure and exemplary values ​​of the training and push task package are shown in Table 1.

[0134] Table 1. Field Structure Table of Training and Push Task Package

[0135] S202: In response to the user's trigger operation on the trigger run control, add the training push task package to the task queue, and allocate computing resources to the training push task package based on the task queue.

[0136] Specifically, the user triggers the running control (such as...) by clicking on the second display area 103 in the interactive interface. Figure 6 After triggering the execution control 1031c shown, this embodiment of the application can respond to the triggering operation by submitting the generated training push task package to the task queue. The task queue is used to manage all verification tasks that are pending, in execution, or completed. It can adopt a first-in-first-out strategy or be dynamically scheduled according to task priority. The task queue and resource scheduling module can perform scheduling decisions, select computing nodes (also known as backend servers 300) that meet the resource requirements based on the availability and specifications of the current computing resource pool, and allocate the training push task package to that node for execution. When computing resources are sufficient, the training push task package can be scheduled for execution immediately; when computing resources are insufficient, the training push task package waits in the task queue.

[0137] S203: Based on the training and inference task package, use the allocated computing resources to load the model to be verified and the running environment, and call the model to be verified to execute the model training task and / or model inference task.

[0138] After successful allocation of computing resources, the backend server 300 can perform the following steps: Load the weight file and configuration file of the model to be verified from the model repository according to the model identifier, model version, and download address in the training and inference task package; initialize the simulation environment (such as LIBERO, CALVIN, etc.) or connect to the ontology environment (such as a robotic arm, mobile chassis, etc.) according to the environment identifier and environment version in the training and inference task package; load the corresponding task data and judgment conditions according to the task information in the training and inference task package; and set the training rounds, inference threshold, random seed, and other running parameters according to the parameter configuration information in the training and inference task package. After completing the above preparations, the backend server can call the model to be verified to perform at least one round of model training and / or model inference tasks in the simulation environment or ontology environment. During execution, the backend server 300 can provide real-time feedback on the running status and process data to the verification platform 100, which can monitor the running status and display the running progress and stages through an interactive interface.

[0139] It is worth noting that during the process of the model to be verified performing model training tasks or model inference tasks (collectively referred to as verification tasks) in a simulation environment or ontology environment, the embodiments of this application can use a task state machine to uniformly manage the lifecycle of the model training task or model inference task. A task state machine is a computational model used to manage and control the various state transitions within the task lifecycle. In the embodiments of this application, the task state machine can specifically define all possible states of a verification task from creation to archiving, as well as the triggering conditions and transition rules between states.

[0140] Figure 8 This is a schematic diagram of the state transition of a model training task or a model inference task provided in the embodiments of this application.

[0141] like Figure 8 As shown, the status of a verification task can include pending configuration, configured, queued, running, running successfully, running failed, canceled, and archived.

[0142] When a user enters the verification process but has not yet completed the configuration, the verification task is in the "pending configuration state". After the user completes the model selection, environment selection, task selection and parameter configuration in the interactive interface, the verification task transitions from the pending configuration state to the configured state, indicating that the preparation work before verification is complete.

[0143] After the user clicks to trigger the run control, the verification task transitions from the "configured" state to the "queued" state, awaiting the allocation of computing resources. After the task queue and resource scheduling module successfully allocates computing resources to the training and push task package, the verification task transitions from the "queued" state to the "running" state, and the background server 300 begins loading the model, initializing the environment, and executing the target task.

[0144] In the "Running" state, when the target task achieves the expected execution result, the verification task transitions from the "Running" state to the "Running Successfully" state. If problems arise such as the target task failing to achieve the expected execution result, model training or inference anomalies, or performance metrics not meeting targets, the verification task transitions from the "Running" state to the "Running Failed" state. During the process, the user can click the cancel task control at any time, and the verification task transitions from the "Running" state to the "Cancelled" state. After generating and storing the verification report (also known as the training and inference report), the verification task can transition from the "Running Successfully" state to the "Archived" state.

[0145] In some implementations, the embodiments of this application can add states such as pause, resume, retry, timeout, and review to the task state machine. The pause state indicates a temporary interruption of the verification task to release resources; the resume state indicates the resumption of a paused verification task; the retry state indicates automatic re-execution after a verification task fails; the timeout state indicates forced termination when the verification task exceeds a preset time limit; and the review state indicates waiting for the execution result of the target task to be manually reviewed. By extending the state machine states, the flexibility of resource scheduling and the fault tolerance of task execution can be improved.

[0146] This application embodiment can display some or all of the status in the interactive interface. Through the above status management method, unified tracking and control of the entire lifecycle of verification tasks, from configuration, queuing, and execution to completion or cancellation, is achieved. This allows users to understand the current task status in real time through the interactive interface, facilitating diagnosis and recovery when task execution is abnormal, thereby improving the transparency and controllability of the verification process.

[0147] Furthermore, the method provided in this application embodiment may also include the following steps S601-S603.

[0148] S601: Records the running video of the target task.

[0149] In this embodiment, the running video can be generated in real-time by the backend server 300 during the execution of the target task. Specifically, in a simulation environment, the backend server 300 can call the simulation environment's rendering engine to render and encode the images of each simulation step into a video stream; in an on-premises environment, the backend server 300 can capture real-time images of the execution process using its onboard camera and encode them into a video stream. The video streams generated in the above manner form a running video in time sequence. This running video records the images of the model to be verified performing the target task in the simulation environment or on-premises environment, including visual information such as action execution and final state. As an important component of the verification results, the running video allows users to intuitively replay and view the actual performance of the model.

[0150] S602: Extract at least one key frame based on the running video; wherein different key frames correspond to different running stages of the target task.

[0151] This application embodiment can perform frame extraction processing on running video to extract at least one keyframe. A keyframe refers to an image frame in the running video that reflects key nodes in the execution process of the target task, such as an initial state frame, a capture action frame, a placement action frame, a door closing action frame, and a completion state frame. Different keyframes correspond to different stages in the execution process of the target task. By extracting and saving keyframes, on the one hand, long-duration video can be compressed into several representative frames, making it easy for users to quickly understand the key nodes of task execution; on the other hand, keyframes can serve as an important basis for subsequent judgment on whether the task has been completed.

[0152] S603: Determine whether the target task has been achieved based on at least one keyframe, and obtain the execution result of the target task.

[0153] After extracting keyframes, embodiments of this application can determine whether the target task has been achieved based on at least one keyframe. Specifically, embodiments of this application can analyze the content of the keyframes using image recognition algorithms to determine whether the target object has reached the target position, whether the operating components are in the expected state, etc., thereby determining whether the expected execution result has been achieved. For example, for a target task involving object handling, target detection can be performed on the keyframes corresponding to the completion state to determine whether the target object has been placed inside the target container and whether the operating components (such as container doors, drawers, etc.) are in the state required by the target result. If all the above conditions are met, the task completion state is determined to be successful; if any condition is not met, the task completion state is determined to be unsuccessful.

[0154] It should be noted that the method for determining whether a task has been completed is not limited to image recognition using keyframes. Other alternative implementations include directly obtaining the task completion flag returned by the simulation engine or robot device by parsing the structured result fields in the runtime log. These various determination methods can be used individually or in combination, and this application does not specifically limit their application.

[0155] Furthermore, in this embodiment of the application, step S300 includes the following steps S301-S302.

[0156] S301: Obtain the operational metrics and task backtracking information of the model training task and / or model inference task; wherein, the operational metrics include at least the running time, resource consumption and task success rate of the model training task and / or model inference task, and the task backtracking information includes at least the task running log of the target task.

[0157] Specifically, in this embodiment, the runtime metrics and task backtracking information recorded during the execution of model training and / or model inference tasks can be obtained from the backend server 300. The runtime metrics are used to quantitatively evaluate the model's task performance and include at least the execution time of the model training and / or model inference tasks (total time consumed from task start to task end), resource consumption (GPU memory usage, CPU utilization, and memory usage during the execution of the model training and / or model inference tasks), and task success rate. The task backtracking information records detailed data about the task execution process and includes at least the task execution logs for the model training and / or model inference tasks, which record timestamps, event types, state changes, and exception information for each stage of the model training and / or model inference task execution.

[0158] In some implementations, the task success rate can be adaptively adjusted based on the content of the target task, for example, using the ratio of the number of subtasks completed to the total number of subtasks as the task success rate. In other implementations, the task success rate can be the ratio of the number of times the target task is completed to the total number of times during multiple verification processes for the target task; in still other implementations, the task success rate can be the ratio of the number of successful rounds to the total number of rounds during multiple executions of the model training task or model inference task. This application does not limit the specific calculation method for the task success rate in its embodiments.

[0159] S302: Generate a verification report based on configuration information, execution results of the target task, operational metrics, and task backtracking information.

[0160] Specifically, this application embodiment can integrate configuration information (including model information, runtime environment information, task information, and parameter configuration information), the execution result (success or failure) of the target task, runtime metrics (running time, resource consumption, and task success rate), and task backtracking information (task runtime logs, etc.) to generate a verification report for the model to be verified. This verification report contains complete information about this verification, providing a complete and traceable basis for subsequent model capability evaluation, solution credibility assessment, or project delivery.

[0161] When generating the verification report, this embodiment of the application can also add a reproducible identifier to the verification report. The reproducible identifier is used to uniquely identify the configuration information, task execution process, and task execution result corresponding to this verification. The reproducible identifier can be a universally unique identifier (UUID) or a hash value generated by a combination of model version, environment version, parameter configuration, random seed, and input data digest. Through this reproducible identifier, the user can retrieve the complete configuration information and process data of this verification at any subsequent point in time and re-execute the target task under the same conditions, achieving accurate reproduction of the verification process. The user can also provide this reproducible identifier to a third party, who can then retrieve the corresponding verification report and evidence data using this identifier without having to re-execute the verification task.

[0162] In some implementations, embodiments of this application may also perform a verification report archiving step. Archiving specifically refers to storing the verification report and related data in a preset format in an archiving database or file system, establishing index relationships, and ensuring the traceability of relationships between data items. Simultaneously, structured verification result retrieval services are provided to model capability evaluation systems, solution generation systems, demand matching systems, or delivery management systems through external interface modules, supporting external systems in performing model evaluation, solution assessment, or project acceptance based on the verification report.

[0163] In some implementations, the method provided in this application embodiment may further include the following step S303.

[0164] S303: If the target task is not completed, determine the failure record of the target task based on the task execution log.

[0165] Specifically, when a target task is completed but not achieved, embodiments of this application can parse task execution logs and execution videos to extract diagnostic information that led to task failure and determine the failure record of the target task. In embodiments of this application, the failure record may include the failure stage and / or the uncompleted target.

[0166] A failure phase refers to the stage in a model training or inference task where an anomaly occurs during execution. For example, the execution phases of a model training or inference task may include, but are not limited to: model loading, environment initialization, data loading, action inference, simulation execution, and target determination. If the task log shows that the model failed to execute instructions during action inference, causing the task to be interrupted, then the failure phase can be identified as the "task inference phase." By recording failure phases, users can quickly pinpoint which stage of the task encountered the problem, allowing for targeted troubleshooting and optimization.

[0167] An unfulfilled goal, also known as an unfulfilled subtask, refers to one or more subtasks within a target task that have not been achieved. For example, when a target task is a combination of multiple subtasks, if all subtasks are achieved, the target task is considered accomplished; if only some subtasks are achieved, the target task is not accomplished, and the unfulfilled subtasks are considered unfulfilled subtasks. For instance, if the expected result is "the object is inside the container, and the container door is closed," but the actual result is "the object is inside the container, but the container door is not closed," then "the container door is not closed" constitutes an unfulfilled goal. By recording unfulfilled goals, users can clearly understand on which specific conditions the target task failed to meet, thus providing a precise basis for subsequent targeted improvements.

[0168] In some implementations, the method provided in this application embodiment may further include the following step S304.

[0169] S304: Obtain historical failure records related to the target task and associate the historical failure records with the verification report; the historical failure records are the failure records generated when the target task was not achieved due to differences in the model to be verified, the operating environment and / or parameter configuration during the historical verification process.

[0170] In practical applications, embodiments of this application can retrieve all failure records of the current target task during historical verification based on the task number of the current target task, and associate the retrieved historical failure records with the current verification report. The historical failure records contain information such as the failure stages and unfulfilled objectives of the target task in previous verifications. By associating historical failure records with the current verification report, users can trace the complete verification history of the same target task under different models, environments, or parameter configurations, and understand the evolution of the target task from failure to success, providing a more comprehensive decision-making basis for model capability evaluation and scheme optimization.

[0171] For example, when the target task fails, the user can return to the model selection or task selection interface, change the model to be verified, the runtime environment, and / or parameter configuration, and then re-execute the target task. After re-execution, this embodiment of the application can associate and archive the failure record with the execution result after re-execution. In this way, multiple verification results of the same target task (including failed attempts and successful attempts) can be linked together to form a complete verification chain, making it easy for users to trace the entire process of model iterative optimization.

[0172] As described above, this application provides a model verification method. After obtaining the user-selected model, environment, task, and parameter configuration, the method generates a training and inference task package and submits it to the task queue. The resource scheduling module allocates computing resources, the backend server loads the model and initializes the environment, performs online training or inference, evaluates the results, and archives the running video, running trajectory, and running logs. Finally, a structured verification report is generated and provided to downstream systems through an external system interface. This method integrates the verification process of models, especially embodied intelligence models, into a unified online guided process, lowering the operational threshold. Furthermore, by binding the model, environment, task, parameters, and data through the training and inference task package, the verification process becomes reproducible. Additionally, the method uses a task state machine to uniformly manage the entire lifecycle of the verification task, improving the transparency of the operation process. Finally, by generating and uniformly archiving the verification report from the running video, running trajectory, task running logs, running indicators, and reproducibility markers, the verification process can be retrieved, referenced, and reused. This allows the verification results to be used as input for model capability evaluation, solution credibility assessment, or project delivery materials, expanding the application value of the verification results.

[0173] This application also provides a computer program product, which, when read and executed by a computer, causes the aforementioned model verification method to be performed. The computer program product can be a software package or SDK (Software Development Kit), or a computer program stored on a computer-readable storage medium (such as a USB flash drive, solid-state drive, optical disc, or cloud storage).

[0174] This application also provides a computer-readable storage medium storing a computer program, which, when part or all of the computer program is executed by a computer, causes the above-described model verification method to be executed.

[0175] It should be noted that, after considering the specification and practicing the content disclosed in this application, those skilled in the art will readily conceive of other embodiments of this application. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary technical means in the art that are not disclosed in this application.

[0176] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The true scope is indicated by this application.

Claims

1. A model validation method, characterized in that, The method includes: Obtain the model to be verified and its configuration information, wherein the configuration information includes at least the runtime environment information, parameter configuration information, and target task information of the model to be verified; Based on the configuration information, the model to be verified is invoked to perform model training tasks and / or model inference tasks. Generate a verification report for the model to be verified; the verification report includes the configuration information, the training and / or inference process of the model to be verified, and the execution result of the target task.

2. The model validation method according to claim 1, characterized in that, The method further includes: The interactive interface includes a first display area and a second display area. The first display area displays at least one model and at least one configuration item, and the second display area is used to display information corresponding to the model or configuration item selected by the user. The process of obtaining the model to be verified and its configuration information includes: The system obtains the model to be verified and the configuration information in response to the user's operation on the interactive interface.

3. The model verification method according to claim 2, characterized in that, The method further includes: The running video of the target task is displayed on the interactive interface; The system plays the running video in response to user actions on the interactive interface, and controls the playback state of the running video in response to user actions on the video control controls.

4. The model verification method according to claim 2, characterized in that, The second display area also includes an interactive guidance control, which is used by the user to control the verification process. The first display area and the second display area are used to display content corresponding to the nodes of the verification process.

5. The model verification method according to claim 4, characterized in that, The interactive guidance control includes a trigger run control and a cancel task control; the method further includes: In response to a user's trigger operation on the trigger run control in the second display area, the task running progress and / or the current running stage of the task are displayed in the first display area and the second display area, and the cancel task control is displayed in the second display area, the cancel task control being used to cancel the currently executing model training task and / or model inference task.

6. The model verification method according to claim 4, characterized in that, The interactive guidance control also includes re-executing the control and reconfiguring the control; the method also includes: The re-execution control and the reconfiguration control are displayed in the second display area; wherein, the re-execution control is used to control the re-execution of the model training task and / or the model inference task based on the configuration information, and the reconfiguration control is used to control the display of multiple models and multiple configuration items in the first display area to re-acquire the new model to be verified selected by the user and / or the new configuration information.

7. The model verification method according to any one of claims 1 to 6, characterized in that, The step of invoking the model to be verified to perform model training and / or model inference tasks based on the configuration information includes: Generate a training and push task package based on the configuration information; In response to the user's trigger operation on the trigger run control, the training push task package is added to the task queue, and computing resources are allocated to the training push task package based on the task queue; According to the training and inference task package, the allocated computing resources are used to load the model to be verified and the running environment, and the model to be verified is invoked to execute the model training task and / or the model inference task.

8. The model verification method according to any one of claims 1 to 6, characterized in that, After invoking the model to be verified to perform model training and / or model inference tasks based on the configuration information, the method further includes: Record the running video of the target task; At least one key frame is extracted from the running video; wherein different key frames correspond to different running stages of the target task; Based on at least one of the keyframes, determine whether the target task has been achieved, and obtain the execution result of the target task.

9. The model verification method according to any one of claims 1 to 6, characterized in that, The generation of the verification report for the model to be verified includes: Obtain the operational metrics and task backtracking information of the model training task and / or the model inference task; wherein, the operational metrics include at least the operation time, resource consumption, and task success rate of the model training task and / or the model inference task, and the task backtracking information includes at least the task operation logs of the model training task and / or the model inference task; The verification report is generated based on the configuration information, the execution result of the target task, the running indicators, and the task backtracking information.

10. The model verification method according to claim 9, characterized in that, The method further includes: If the target task is not achieved, the failure record of the target task is determined based on the task execution log; Alternatively, the method may further include: Obtain historical failure records related to the target task and associate the historical failure records with the verification report; the historical failure records are the failure records generated when the target task was not achieved due to differences in the model to be verified, the operating environment and / or parameter configuration during the historical verification process.

11. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 10.

12. A computer program product, characterized in that, When the computer reads and executes the computer program product, the method as described in any one of claims 1 to 10 is performed.

Images