Method, device, equipment, storage medium and program product for model training

Abstract

Methods, apparatuses, devices, storage media, and program products for model training are provided. The method includes: in response to receiving a request for a training task, converting the request into a task description having a predetermined format, wherein the type of the training task includes at least one of: a model training task related to interaction with a physical environment, a model training task related to tool invocation, or a model training task related to an inference process; based on the task description, determining a training process corresponding to the training task and a computing resource for executing the training process; determining an environment instance for the training process, the environment instance being associated with the type of the training task; and based on the computing resource and the environment instance, executing a plurality of sub-tasks corresponding to the training process, the plurality of sub-tasks being executed in parallel in an asynchronous manner, and data being transmitted between the plurality of sub-tasks in a unified data format.

Description

Technical Field

[0001] The examples in this article generally relate to the field of computer science, and in particular to methods, apparatus, devices, storage media, and program products used for model training. Background Technology

[0002] With the development of artificial intelligence technology, machine learning models have been widely applied in various fields such as intelligent decision-making, automated control, and information processing. Reinforcement learning, as a machine learning method that optimizes the model's capabilities through interaction with the environment and based on feedback information, has gradually been applied to various types of model training scenarios. However, there is still room for improvement in providing unified support for different types of training tasks. Summary of the Invention

[0003] In a first aspect of this paper, a method for model training is provided. The method includes: in response to receiving a request for a training task, converting the request into a task description with a predetermined format, wherein the type of the training task includes at least one of the following: a model training task related to interaction with a physical environment, a model training task related to tool invocation, or a model training task related to an inference process; determining a training flow corresponding to the training task and computational resources for executing the training flow based on the task description; determining an environment instance for the training flow, the environment instance being associated with the type of training task; and executing multiple subtasks corresponding to the training flow based on the computational resources and the environment instance, the multiple subtasks being executed asynchronously in parallel, and data being transmitted between the multiple subtasks in a unified data format.

[0004] In a second aspect of this paper, an apparatus for model training is provided. The apparatus includes: a description determination module configured to, in response to receiving a request for a training task, convert the request into a task description having a predetermined format, wherein the type of the training task includes at least one of the following: a model training task related to interaction with a physical environment, a model training task related to tool invocation, or a model training task related to an inference process; a process determination module configured to, based on the task description, determine a training process corresponding to the training task and computational resources for executing the training process; an environment determination module configured to determine an environment instance for the training process, the environment instance being associated with the type of the training task; and an execution module configured to, based on the computational resources and the environment instance, execute multiple subtasks corresponding to the training process, the multiple subtasks being executed asynchronously in parallel, and data being transmitted between the multiple subtasks in a unified data format.

[0005] In a third aspect of this document, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this document, a computer-readable storage medium is provided. The computer-readable storage medium stores computer-executable instructions that can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of this document, a computer program product is provided, which is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0008] In this way, by unifying different types of training tasks into standardized task descriptions and combining them with a unified environment abstraction and asynchronous parallel execution architecture, multiple types of reinforcement learning training tasks can be supported in the same infrastructure, improving the versatility of the training infrastructure and the utilization efficiency of computing resources.

[0009] It should be understood that the content described in this section is not intended to limit the key or important features of the examples in this article, nor is it intended to restrict the scope of the solution. Other features will become readily apparent from the following description. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the various examples herein will become more apparent when taken in conjunction with the accompanying drawings and the following detailed description. In the accompanying drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A schematic diagram of the example environment is shown; Figure 2 Block diagrams of systems used for model training in some scenarios are shown; Figure 3 The diagrams show schematics of architectures used for model training in several scenarios; Figure 4 Flowcharts are shown for methods used in model training in several scenarios; Figure 5 Block diagrams of apparatuses for model training in several scenarios are shown; and Figure 6 A block diagram of an electronic device capable of implementing multiple illustrative scenarios is shown. Detailed Implementation

[0011] The examples in the text will now be described in more detail with reference to the accompanying drawings. While some examples are shown in the drawings, it should be understood that solutions can be implemented in various forms and should not be construed as limited to the examples presented herein. Rather, these examples are provided to provide a more thorough and complete understanding of the solutions. It should be understood that the drawings and examples in this document are for illustrative purposes only and are not intended to limit the scope of protection of the solutions.

[0012] It should be noted that the headings of any section / subsection provided herein are not restrictive. Various examples are described throughout this document, and examples of any type may be included under any section / subsection. Furthermore, examples described in any section / subsection may be combined in any way with any other examples described in the same section / subsection and / or different sections / subsections.

[0013] In the description of the examples in this document, the term "including" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "an example" or "the example" should be understood as "at least one example". The term "some examples" should be understood as "at least some examples". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0014] The examples in this document may involve user data, data acquisition, and / or use. All of these aspects comply with relevant laws, regulations, and rules. In the examples, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, when implementing each example, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained through appropriate means, in accordance with relevant laws and regulations. The specific methods of notification and / or authorization can vary depending on the actual situation and application scenario; the scope of the solution is not limited in this regard.

[0015] In this manual and the sample solutions, any processing of personal information will be conducted only under legal grounds (such as obtaining the consent of the data subject or being necessary for the performance of a contract) and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0016] The term "reinforcement learning" as used in this paper refers to a machine learning method that learns optimal policies through interaction with the environment. In this method, the agent continuously adjusts its policy by observing the environmental state, performing actions, and receiving reward signals to maximize cumulative rewards. The term "post-training" as used in this paper refers to the process of further optimizing a large language model after pre-training using techniques such as reinforcement learning or preference alignment. The term "training task" as used in this paper refers to a complete reinforcement learning training request, which may include configuration information such as task type, environment configuration, model parameters, training algorithm, and reward function.

[0017] With the development of artificial intelligence technology, intelligent systems based on machine learning models are increasingly being applied to more complex task processing scenarios. To improve the model's decision-making and execution capabilities in complex tasks, reinforcement learning can be used to train the model. During reinforcement learning training, the model can determine actions based on its current state and interact with the environment to obtain feedback information, thereby optimizing model parameters based on the data generated during the interaction.

[0018] Depending on the application scenario, reinforcement learning training tasks can take different forms. For example, in training tasks related to interaction with the physical environment, the model can learn to perform control and operation tasks by interacting with simulated or real environments; in training tasks related to tool invocation, the model can obtain feedback by invoking external tools or performing operations to improve its task execution capabilities; and in training tasks related to the inference process, the model can be optimized based on training data and evaluation feedback to improve its analysis and reasoning abilities.

[0019] However, in related technologies, due to significant differences in task objectives, interaction objects, and data formats among similar reinforcement learning training tasks, their corresponding training systems typically need to be designed separately for specific application scenarios. For example, for training tasks related to physical environment interaction, corresponding environment simulation, state acquisition, and action execution mechanisms need to be configured. For training tasks related to tool invocation, corresponding tool invocation processes and feedback mechanisms need to be configured; and for training tasks related to the inference process, corresponding data processing and result evaluation mechanisms need to be configured. Because different types of tasks differ in task description methods, environment interaction methods, and training execution processes, it is usually necessary to design separate training processes and infrastructure for different tasks.

[0020] Furthermore, reinforcement learning training typically involves multiple execution stages, including model training, environment interaction, and data processing. These stages require the exchange of data such as model output, environment feedback, and training samples. When different types of training tasks utilize independently built training systems, it becomes difficult to standardize data formats, environment access methods, and execution processes across these systems, thus limiting the reusability of training infrastructure.

[0021] In view of this, a scheme for model training is proposed. According to the scheme, in response to receiving a request for a training task, the request is converted into a task description with a predetermined format, wherein the type of training task includes at least one of the following: a model training task related to interaction with a physical environment, a model training task related to tool invocation, or a model training task related to the inference process; based on the task description, a training process corresponding to the training task and computing resources for executing the training process are determined; an environment instance for the training process is determined, the environment instance being associated with the type of training task; and based on the computing resources and the environment instance, multiple subtasks corresponding to the training process are executed, the multiple subtasks are executed asynchronously in parallel, and data is transmitted between the multiple subtasks in a unified data format.

[0022] The above approach enables the construction of a unified training infrastructure framework for various reinforcement learning tasks. This framework allows model training tasks related to physical environment interaction, tool invocation, and inference processes to be executed within the same training architecture. Specifically, by converting different types of training tasks into task descriptions in a unified format and determining the corresponding training flow, computing resources, and environment instances based on these descriptions, different types of model training tasks can be managed and executed through a unified training process. This improves the compatibility and scalability of the reinforcement learning training infrastructure for different task types.

[0023] Furthermore, by adopting a unified environment determination method and a unified data transmission format, the environmental interaction process and training data corresponding to different types of training tasks can be processed within the same infrastructure framework, reducing the adaptation cost between different task types and improving the data interaction efficiency between different execution stages during training.

[0024] Furthermore, by enabling multiple subtasks corresponding to the training process to be executed asynchronously in parallel, the waiting and coupling between different execution stages can be reduced, thereby improving the efficiency of computing resource utilization.

[0025] The following describes various examples of this scheme in further detail with reference to the accompanying drawings.

[0026] Figure 1 A schematic diagram of an example environment 100 according to some embodiments is shown. Figure 1 The diagram illustrates two distinct phases of the machine learning model 150: a training phase 130 and an application phase 140. A pre-training phase may precede the training phase 130, and a testing phase may follow the training phase (not shown in the diagram).

[0027] Example environment 100 involves model training system 110 and model application system 120. In training phase 130, training phase 130 is configured to perform training of machine learning model 150 using training dataset 115. At the start of training, the various components in machine learning model 150 may have initial parameter values. The training process involves updating the parameter values ​​of machine learning model 150 to desired values ​​based on data in training dataset 115. Training tasks are used to assist in updating the parameters of machine learning model 150.

[0028] In some embodiments, during the training phase 130, the model training system 110 can be configured to train the machine learning model 150 using training data 115 to update the parameters in the machine learning model 150. In some embodiments, the training phase 130 may include a reinforcement learning-based model training process. For example, the model training system 110 may enable the machine learning model 150 to interact with a corresponding environment, generate training samples based on the actions generated by the machine learning model 150, the feedback information returned by the environment, and the corresponding reward information, and update the parameters of the machine learning model 150 based on the training samples.

[0029] In some embodiments, training data 115 may include data generated during the interaction between the model and the environment, such as state information, observation information, action information, environmental feedback information, reward information, or other data used for model training. Depending on the type of training task, the environment in which the machine learning model 150 interacts may include different types of environments. For example, the environment may include a simulation environment for interacting with physical objects, a tool environment for executing tool calls, or an inference environment for verifying model inference results.

[0030] During the training phase 130, the machine learning model 150 can be optimized using training data 115 corresponding to multiple training tasks, enabling the machine learning model 150 to acquire the capabilities required to perform the target task. For example, for training tasks related to interaction with the physical environment, the machine learning model 150 can learn to generate control actions based on the environmental state; for training tasks related to tool invocation, the machine learning model 150 can learn to select and invoke the corresponding tool according to the task objective; and for training tasks related to the inference process, the machine learning model 150 can learn to generate inference results that meet the evaluation requirements.

[0031] In some embodiments, a testing phase may be included after the training phase 130, where the performance of the machine learning model 150 can be further tested using a test dataset. The dataset used in the testing phase is of the same type as that used in the fine-tuning phase. The machine learning model 150 may include, but is not limited to, neural network models, deep learning models, visual language action models (VLA), and other forms of models.

[0032] In the application phase 140, the obtained machine learning model 150 has trained parameter values ​​and can be provided to the model application system 120 for use. In the application phase 140, the machine learning model 150 can be used to process the corresponding model input 125 in the real-world scenario and provide the corresponding model output 126.

[0033] It should be understood that Figure 1 The components and arrangements in the illustrated example architecture 100 are merely examples, and a computing system suitable for implementing the exemplary implementations described in this disclosure may include one or more different components, other components, and / or different arrangements. For example, although shown as separate, the model training system 110 and the model application system 120 may be integrated in the same system or device. For example, at least the model training system 110 and the model application system 120 may be integrated in a model training system or device. Implementations of this disclosure are not limited in this respect. Figure 1 In this system, the model training system 110 and the model application system 120 can be deployed on any suitable electronic device. This electronic device can be any type of computing-capable device, including terminal devices or server devices.

[0034] The model training system 110 and the model application system 120 can be implemented on any type of electronic device. This electronic device can be any type of mobile terminal, fixed terminal, or portable terminal. In some embodiments, the model training system 110 and the model application system 120 can also be implemented on a standalone physical server, a server cluster consisting of multiple physical servers, or a distributed system. They can also be implemented on cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms.

[0035] It should be understood that the model training process described in this paper can be implemented using different computing architectures. For example, the model training process can be executed by a single computing device, which handles training data processing, environment interaction, and model parameter updates; or it can be executed collaboratively by multiple computing devices, where different computing devices can undertake different functions such as model training, model inference, environment execution, data processing, or resource management. The scope of the solution is not limited in this respect.

[0036] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of the scheme.

[0037] Various examples of this scheme are further described in detail below with reference to the accompanying drawings. In the following description, the examples will primarily be described with respect to the model training system 110. It should be understood that the actions described with respect to the model training system 110 can also be performed by the model training system 110 in conjunction with its server (e.g., a server).

[0038] Figure 2 Schematic diagrams of a system 200 for model training in several scenarios are shown. System 200 can be implemented in model training system 110. In some embodiments, system 200 can be used to support different types of model training tasks and provide a unified training infrastructure for different types of model training tasks.

[0039] In some embodiments, in response to receiving a training task request, the system 200 can represent the training task in a unified manner and determine the training process, computing resources, and environment instance corresponding to the training task based on the unified representation of the task information. In this way, the system 200 can support different types of model training tasks to be executed on the same infrastructure.

[0040] In some embodiments, to support the unified execution of training tasks for different types of models, system 200 can serve as a unified training infrastructure for supporting different types of model training tasks. System 200 can decouple functions such as task access, training process determination, environment adaptation, training execution, data management, and resource provisioning, and implement data transmission and communication between different functions through predefined interface specifications. Through functional decoupling, different functional modules can be deployed, extended, or replaced independently, thereby improving the flexibility and scalability of the training infrastructure.

[0041] like Figure 2 As shown, system 200 may include task access module 210, task scheduling module 220, environment adaptation module 230, execution module 240, data management module 250 and resource management module 260.

[0042] In some embodiments, the task access module 210 can be used to receive training task requests and generate a task description corresponding to the training task. The task scheduling module 220 can be used to determine the training process and the computing resources used to execute the training process based on the task description. The environment adaptation module 230 can be used to determine the environment instance corresponding to the training task and provide a unified access method for the environment instance. The execution module 240 can be used to execute model training based on the training process, computing resources, and environment instance. The data management module 250 can be used to manage the data generated during model training, and the resource management module 260 can be used to provide computing resources to the training process.

[0043] In some embodiments, the functional modules described above in system 200 can form a layered infrastructure architecture. For example, task access module 210 can correspond to a functional layer providing unified task access capabilities, task scheduling module 220 can correspond to a functional layer for performing task orchestration and resource scheduling, environment adaptation module 230 can correspond to a functional layer providing environment abstraction and environment adaptation capabilities, execution module 240 can correspond to a functional layer for performing model training-related computational tasks, data management module 250 can correspond to a functional layer providing training data and reward data management capabilities, and resource management module 260 can correspond to a functional layer providing underlying computing resource support. Data transmission and function calls can be performed between the functional layers through corresponding interfaces.

[0044] It should be understood that the above module division is only used to illustrate the exemplary functional modules included in system 200, and does not constitute a limitation on the specific implementation of system 200. In some embodiments, the above-mentioned multiple functional modules may be implemented by software, hardware, firmware or a combination thereof, and different functional modules may be deployed in the same computing device or in multiple interconnecting computing devices.

[0045] Figure 3 A schematic diagram of the detailed architecture 300 of the system 200 for model training in some scenarios is shown. See below for reference. Figure 3 This will further describe the internal structure and interaction relationships of each module in System 200.

[0046] like Figure 3 As shown, system 200 may include task access module 210. Task access module 210 can serve as a unified entry point for model training, shielding the differences in task description methods between different types of training tasks, and converting externally input training tasks into a unified representation form that can be recognized internally by system 200.

[0047] By setting up the task access module 210, the system 200 does not need to build separate access processes for different types of training tasks. Instead, it can provide standardized task information to subsequent functional modules based on a unified task description method, thereby supporting the execution of different types of model training tasks within the same infrastructure framework. For example, the task access module 210 can be used to receive training task requests from external sources and convert the training task requests into task descriptions with a predetermined format, so that the system 200 can subsequently perform operations such as determining the training process, allocating resources, and configuring environment instances based on the unified format task descriptions.

[0048] In some embodiments, a training task request may refer to request information submitted by a requester to perform a model training process using system 200. The training task request may be used to indicate the model training task to be performed and may include information related to the model training task, such as training task type, training objective, information about the model to be trained, training data-related information, environment-related information, training configuration parameters, and resource requirement information.

[0049] In some embodiments, the task access module 210 can receive training task requests through a unified interface gateway. For example... Figure 3 As shown, the task access module 210 may include an interface gateway and a task verification module.

[0050] The interface gateway can provide standardized external access interfaces to support different requesters in submitting training task requests to system 200 through a unified entry point. In some embodiments, the interface gateway can support multiple communication methods to adapt to the data interaction needs of different types of requesters. The interface gateway can perform operations such as receiving, parsing, and routing the received training task requests to transmit them to the corresponding processing modules. The interface gateway can also perform preliminary access control based on the requester information, such as determining whether the requester has permission to access system 200, or determining the corresponding request processing method based on the requester's configuration information.

[0051] In some embodiments, the task access module 210 may perform verification on the request through the task verification module to determine whether the training task request meets the conditions for entering the subsequent training process. In some examples, the verification may include at least one of the following: parameter integrity verification, resource quota verification, or permission verification. For example, the task verification module may determine whether the training task request contains the necessary parameters required to execute the training task, determine whether the requester has the corresponding resource usage permission, or determine whether the resources currently available to the requester meet the training task requirements.

[0052] In some embodiments, if the training task request passes verification, the task access module 210 can further determine the type of the training task and generate a corresponding task description based on the type of the training task.

[0053] In some embodiments, the training task type includes at least one of the following: model training task related to interaction with the physical environment, model training task related to tool invocation, or model training task related to the inference process.

[0054] Model training tasks related to interaction with the physical environment can refer to training tasks that require the model to interact with a pre-built physical or simulated environment and optimize the model's capabilities based on information returned by the environment. For example, training tasks can be used to train the model to perform control operations, decision-making operations, or other tasks that require interaction with changes in the environmental state.

[0055] Model training tasks related to tool invocation can refer to training tasks that require the model to invoke one or more external tools and optimize the model's capabilities based on the tool's execution results, returned information, or task completion status. For example, training tasks may involve the model selecting a target tool according to task requirements, generating tool invocation operations, and adjusting the model's behavior based on the tool's execution results.

[0056] Model training tasks related to the reasoning process can refer to training tasks that optimize the model's reasoning process. For example, training tasks can optimize the model's reasoning ability, task solving ability, or result generation ability based on preset data samples, reasoning process information, or evaluation feedback information.

[0057] In some embodiments, the task access module 210 can determine a target task description template from multiple task description templates based on the determined training task type, and generate a task description based on the target task description template. The multiple task description templates are used to indicate a predetermined format for the task description, enabling different types of training tasks to be converted into a unified format of information representation.

[0058] like Figure 3 As shown, the task access module 210 may include multiple task description templates, each corresponding to a different type of training task. For example, the multiple task description templates may include a first task description template, a second task description template, and a third task description template.

[0059] In some embodiments, the first task description template may correspond to a model training task related to interaction with the physical environment and be used to specify the task description format corresponding to this type of training task. For example, the first task description template may specify the representation of task information related to interaction with the physical environment, including format specifications for describing information such as environment configuration, state space, action space, and task objectives.

[0060] In some embodiments, the second task description template may correspond to a model training task related to tool invocation and be used to specify the task description format corresponding to this type of training task. For example, the second task description template may specify the representation of task information related to tool invocation, including format specifications for describing information such as callable tools, tool invocation rules, tool invocation process, and execution results.

[0061] In some embodiments, the third task description template may correspond to a model training task related to the inference process and be used to specify the task description format corresponding to this type of training task. For example, the third task description template may specify the representation of task information related to model inference optimization, including format specifications for describing training data, inference requirements, evaluation rules, and feedback information.

[0062] In this way, the task access module 210 can use the corresponding task description template for different types of model training tasks, and convert different types of training tasks into a unified format task description, so that subsequent training processes can be performed based on the unified format task description.

[0063] In some embodiments, a training task request may be associated with a requester identifier. The requester identifier may be used to identify the user, user group, application, organization, or other requesting entity that submitted the training task request. Based on the requester identifier, system 200 may determine the task execution environment corresponding to the requester and execute the training process corresponding to the training task within that task execution environment.

[0064] like Figure 3 As shown, the task access module 210 may further include an isolation management module. The isolation management module can be used to manage the isolation of training tasks corresponding to different requesters based on the requester identifier associated with the training task request. Specifically, the isolation management module can determine or create an execution environment corresponding to the requester identifier based on the requester identifier, and associate the training task and the subsequently generated training process with the execution environment.

[0065] Resource usage scope can be used to limit the type, quantity, or usage limit of computing resources that the requester can use during the execution of a training task. For example, resource usage scope can limit the processor resources, storage resources, number of environment instances, or network resources that the requester can use.

[0066] Data access permissions can be used to limit the range of data that a requesting party can access, read, write, or modify during the execution of a training task. For example, data access permissions can be used to restrict access to training data, model parameters, training trajectory data, and evaluation data between different requesting parties.

[0067] Namespaces can be used to provide distinct logical resource scopes for different requesters. System 200 can use different namespaces to differentiate the tasks, computing resources, environment instances, data, and runtime states corresponding to different requesters, thereby preventing interference between the training processes of different requesters. It should be understood that namespaces can also be replaced by other resource organization methods that can achieve logical isolation.

[0068] By setting up an isolation management module, when multiple requesters share the same training infrastructure, the scope of resource usage and data access for different requesters can be limited, reducing the possibility of resource contention, unauthorized data access, or mutual influence on the running status between training tasks of different requesters, and providing a model training process that is isolated from each other for different requesters.

[0069] Continue to refer to Figure 3 System 200 may include a task scheduling module 220. The task scheduling module 220 may be located between the task access module 210 and subsequent functional modules such as environment adaptation, training execution, and resource provisioning. It is used to convert the task descriptions provided by the task access module 210 into an executable training process and coordinate the various execution stages and corresponding computing resources involved in the training process. By setting up the task scheduling module 220, system 200 can uniformly convert the differences in specific training steps and resource requirements of different types of training tasks into a workflow consisting of multiple execution stages, enabling subsequent functional modules to execute different types of training tasks according to a unified process organization method.

[0070] In some embodiments, the task scheduling module 220 can determine multiple execution stages corresponding to the training task based on the task description, and determine the execution order of each execution stage according to the dependencies between the multiple execution stages to form a training process. Thus, the task access module 210 can focus on receiving and uniformly representing the training task, while the task scheduling module 220 can focus on further converting the uniformly represented task description into a processing procedure that can be executed by the system 200, thereby achieving the separation of the task access process and the task execution process.

[0071] In some embodiments, the task scheduling module 220 can determine a workflow including multiple execution stages based on the task description, obtain predetermined configuration information, and determine the execution order of multiple execution stages in the workflow based on the dependencies indicated by the predetermined configuration information, so as to determine the training process.

[0072] In this paper, a workflow can be an information structure used to describe multiple execution stages and their interrelationships. A workflow may include information such as stage identifiers, stage inputs, stage outputs, execution conditions, and dependencies between execution stages. A training process can be a procedure organized according to the execution order determined by the workflow, used to complete a training task.

[0073] like Figure 3 As shown, the task scheduling module 220 may include a task orchestration module. The task orchestration module can be used to organize the model training process into a configurable workflow consisting of multiple execution stages, and to orchestrate the execution order of each execution stage according to the dependencies between them.

[0074] Although the specific execution stages included in different types of training tasks may differ, the task orchestration module can use a unified workflow representation to describe multiple execution stages and their dependencies. Therefore, the task scheduling module 220 does not need to build independent flow control mechanisms for each type of training task, but can form training processes corresponding to different training tasks by configuring the execution stages and dependencies of the workflow.

[0075] In this way, the task scheduling module 220 can convert task descriptions with a unified format into configurable and executable training processes, and support different types of model training tasks with a unified workflow organization method, thereby reducing the coupling between task descriptions and specific execution modules, and improving the reusability and scalability of the training process.

[0076] Continue to refer to Figure 3 The task scheduling module 220 may include a resource scheduling module. The resource scheduling module can be used to determine the computing resources required to execute the training process and allocate one or more execution stages in the training process to the corresponding computing resources for execution.

[0077] In some embodiments, the resource scheduling module can determine the computing resources that match the training task based on the task characteristics, historical execution status of the training task, and hardware capabilities of candidate hardware nodes. Thus, the task scheduling module 220 can not only determine the execution order of each execution stage in the training process, but also determine the corresponding resource allocation for each execution stage based on the computing requirements of different execution stages, thereby coordinating the organization of the training process with the resource supply method.

[0078] In some embodiments, the resource scheduling module can determine the task information of the training task, the historical execution information corresponding to the training task, and the hardware capability information of the candidate hardware nodes. For example, the task information may include at least one of the following: the type of training task, the training stage, the scale of training data, or the frequency of environmental interaction; the historical execution information may include historical resource usage, execution efficiency, or resource utilization; the hardware capability information may include the computing power, storage capacity, communication capacity, and current resource status of the candidate hardware nodes.

[0079] In some examples, different types of training tasks can have different resource requirements. For instance, model training tasks related to interaction with the physical environment may have high requirements for environmental state computation or graphics processing; model training tasks related to tool calls may have high requirements for general computation, storage, or isolated execution; and model training tasks related to the inference process may have high requirements for model inference or model parameter updates.

[0080] In some examples, the training phase can be used to represent the current processing stage of a training task. For instance, a training phase may include an action sequence determination phase, an environment interaction phase, a reward data determination phase, a model parameter update phase, or a model evaluation phase. Since different training phases perform different operations, their requirements for computing power, storage capacity, and data transfer capabilities can also differ.

[0081] Furthermore, the resource scheduling module can determine the computing resources used to execute the training process based on at least two of the following: task information, historical execution information, and hardware capability information. For example, the resource scheduling module can determine the target hardware node that matches the training task from multiple candidate hardware nodes based on the type of training task and the hardware capability information of the candidate hardware nodes; or, it can determine the number of target hardware nodes and the resource allocation quota by combining the current training stage of the training task, historical resource usage, and the current resource status of the candidate hardware nodes.

[0082] In some embodiments, the resource scheduling module can convert the determined resource allocation result into a resource configuration instruction that the resource management module 260 can recognize, and provide the resource configuration instruction to the resource management module 260. The resource management module 260 can then reserve, allocate, start, or release computing resources according to the resource configuration instruction. Thus, the resource scheduling module can be responsible for generating resource scheduling decisions, while the resource management module 260 can be responsible for executing the specific resource configuration operations.

[0083] In this way, the resource scheduling module can determine computing resources by combining the characteristics of the training task itself, historical execution status and the capabilities of candidate hardware nodes. Compared with simply executing training tasks according to fixed resource configuration, it can improve the matching degree between training tasks and computing resources, and support the differentiated computing resource requirements of different types of training tasks and different training stages.

[0084] Continue to refer to Figure 3 The task scheduling module 220 may also include an execution control module, a task status management module, and a resource status management module. The execution control module can be used to control the execution of multiple execution stages or multiple sub-tasks according to the training process, and coordinate the execution relationship between multiple sub-tasks, providing process control for the asynchronous parallel execution of multiple subsequent sub-tasks.

[0085] The task status management module can be used to manage the running status of training tasks during execution, such as tracking the status changes of training tasks from different stages such as pending execution, in execution, to completion, so as to achieve unified management of training tasks. The resource status management module can be used to maintain the status information of computing resources in the system and provide the resource scheduling module with information on currently available resources to support the resource scheduling module in resource allocation and adjustment.

[0086] Through the collaboration between the task orchestration module, resource scheduling module, execution control module, task status management module, and resource status management module, the task scheduling module 220 can convert a unified format task description into an executable training process, and coordinate the execution order and computing resources of the training process in a unified manner, providing a consistent process organization and resource scheduling mechanism for training tasks of different types of models.

[0087] Continue to refer to Figure 3 System 200 may include an environment adaptation module 230. The environment adaptation module 230 can serve as an adaptation layer connecting the training process and the training environment in system 200, used to uniformly abstract the environment corresponding to different types of training tasks, and provide a unified environment interaction method to the execution module 240.

[0088] Different types of model training tasks can correspond to different types of training environments. For example, model training tasks related to interaction with the physical environment can correspond to a physical environment or a simulation environment; model training tasks related to tool invocation can correspond to a tool execution environment; and model training tasks related to the inference process can correspond to an inference verification environment. Since different environments may differ in terms of environment initialization methods, interaction methods, and environment feedback methods, the environment adaptation module 230 can uniformly convert different types of training environments into a consistent environment interaction method, thereby eliminating the need for the execution module 240 to implement training logic separately for different environments.

[0089] In some embodiments, the environment adaptation module 230 may include a unified environment interaction interface. Environment instances can be accessed through this unified environment interaction interface, which specifies a unified interaction method for each environment instance.

[0090] For example, a unified environment interaction interface can specify standard interaction operations such as environment initialization, environment interaction, and environment termination, as well as a unified representation specification for the observation space and action space of environment instances. The observation space can be used to represent the information that the model can obtain from environment instances; the action space can be used to represent the actions that the model can perform on environment instances. Thus, environment state and action information generated by different types of environments can be provided to the execution module 240 according to a unified representation, allowing the execution module 240 to complete environment interaction without needing to be aware of the specific environment type.

[0091] Continue to refer to Figure 3 The environment adaptation module 230 may also include multiple environment adapters. The unified environment interaction interface can communicate with the environment instance through at least one environment adapter to convert different types of environments into the interaction methods corresponding to the unified environment interaction interface.

[0092] In some embodiments, at least one environment adapter may include at least one of the following: a first adapter associated with interaction with the physical environment; a second adapter associated with the tool invocation environment; and a third adapter associated with the inference verification environment.

[0093] In some embodiments, the first adapter can be used to establish the correspondence between the unified environment interaction interface and the physical environment. For example, the first adapter can be responsible for operations such as environment initialization, state synchronization, action execution, and environment feedback acquisition. The second adapter can be used to establish the correspondence between the unified environment interaction interface and the tool invocation environment. For example, the second adapter can be responsible for operations such as sending tool invocation requests, acquiring tool execution results, and updating environment states. The third adapter can be used to establish the correspondence between the unified environment interaction interface and the inference verification environment. For example, the third adapter can be responsible for operations such as inference process verification, evaluation result acquisition, and feedback information generation.

[0094] In this way, different types of environments can access a unified environment interaction interface through their respective environment adapters, enabling the model training system to access different types of environment instances using a consistent interaction method, thus decoupling the environment adaptation process from the model training process.

[0095] Continue to refer to Figure 3The environment adaptation module 230 may further include an instance management module. The instance management module can manage the lifecycle of environment instances and determine the environment instance corresponding to the training task from the environment instance pool. In some embodiments, the environment instance pool can be used to store one or more environment instances that can be invoked by the training task.

[0096] The instance management module can dynamically manage the environment instance pool based on the environment interaction load corresponding to the training process. For example, the instance management module can perform at least one of the following operations: create environment instances; allocate environment instances; reclaim environment instances; release environment instances. For example, when the environment interaction load increases, the instance management module can increase the number of environment instances in the environment instance pool; when the environment interaction load decreases, it can release some environment instances to reduce the consumption of computing resources.

[0097] By unifying the environment interaction interface, environment adapter, and environment instance management mechanism, the environment adaptation module 230 can abstract the heterogeneous environments corresponding to different types of training tasks into a consistent environment interaction method, thereby separating the environment adaptation process from the model training process. This enables different types of model training tasks to share the same training execution framework and improves the reusability and scalability of the training environment.

[0098] In some embodiments, the model training system 110 can execute multiple subtasks corresponding to the training process based on computing resources and environment instances. The multiple subtasks are executed in parallel asynchronously, and data is transmitted between the multiple subtasks in a unified data format.

[0099] Continue to refer to Figure 3 System 200 may include execution module 240. Execution module 240 can serve as the core computing module in system 200, used to execute multiple computing tasks in the training process. Execution module 240 can decouple different types of computing tasks during model training, enabling different computing tasks to be executed separately and collaboratively completing model training through a unified data interaction method.

[0100] By setting up the execution module 240, the system 200 does not need to complete the action determination, environment interaction and model parameter update in a serial manner. Instead, it can assign multiple computing tasks to different execution units for processing, thereby supporting the asynchronous parallel execution of multiple computing tasks and improving the utilization of computing resources and training throughput during the training process.

[0101] In some embodiments, the multiple subtasks include: a first subtask for determining the action sequence of the model, a second subtask for performing interactive operations with an environment instance based on the action sequence to obtain environmental feedback data, and a third subtask for updating the parameters of the model based on training samples, wherein the training samples are determined based on the action sequence and the environmental feedback data.

[0102] In some examples, the first, second, and third subtasks can correspond to the three main computational processes in model training: action determination, environment interaction, and model update. These three subtasks logically form a complete training loop: the first subtask generates action sequences; the second subtask interacts with environment instances based on these action sequences and obtains environmental feedback data; and the third subtask forms training samples based on the action sequences and environmental feedback data and updates the model parameters. The updated model parameters can then be used to determine subsequent action sequences.

[0103] In some embodiments, the execution module 240 does not need to wait for the three subtasks to complete sequentially, but can assign the three subtasks to different execution units for execution, thereby achieving asynchronous parallel processing among multiple subtasks. For example, refer to Figure 3 The execution module 240 may include a first execution unit, a second execution unit, and a third execution unit. The first execution unit may be used to execute a first subtask to determine an action sequence based on the current model; the second execution unit may be used to execute a second subtask to perform interactive operations with the environment instance based on the action sequence and obtain environmental feedback data; the third execution unit may be used to execute a third subtask to update model parameters based on training samples.

[0104] In some embodiments, the first execution unit, the second execution unit, and the third execution unit can be implemented by independent computing engines. For example, the first execution unit can be implemented as an action sequence determination engine, used to perform model inference based on the current model to generate action sequences; the second execution unit can be implemented as an environment execution engine, used to call the unified environment interaction interface provided by the environment adaptation module 230, drive the environment instance to perform corresponding actions, and obtain environment feedback data; the third execution unit can be implemented as a model training engine, used to update model parameters based on training samples.

[0105] In some embodiments, the action sequence determination engine, the environment execution engine, and the model training engine can form independent execution pipelines. The three execution units exchange action sequences, environmental feedback data, and training samples through a unified data exchange mechanism, without having to wait for other execution units to complete all processing steps, thereby decoupling the three computationally intensive processing steps of model inference, environment interaction, and model training.

[0106] Continue to refer to Figure 3 In some embodiments, the first subtask, the second subtask, and the third subtask communicate with each other via an asynchronous message queue. The asynchronous message queue can serve as a data exchange channel between multiple execution units, used to transmit action sequences, environmental feedback data, training sample requests, and model parameter update notifications, enabling different execution units to collaborate through message passing without needing to establish direct synchronous call relationships.

[0107] Since multiple execution units exchange data through an asynchronous message queue, the first, second, and third execution units do not need to wait for other execution units to complete all processing before continuing to execute subsequent operations. This reduces the synchronization waiting between different execution units and improves the efficiency of parallel execution of multiple subtasks.

[0108] In some embodiments, environmental feedback data, parameter-related data, and reward data are stored in a unified cache area. This unified cache area serves as a shared data cache and exchange area for all execution units during training, temporarily storing data generated during the execution of multiple subtasks and providing a unified data access method to multiple execution units. By setting up a unified cache area, multiple execution units do not need to maintain independent copies of each other's data; instead, they can share the data generated during training, thereby improving the efficiency of training data flow and reducing the complexity of data exchange between multiple execution units.

[0109] Continue to refer to Figure 3 System 200 may include a data management module 250. The data management module 250 can serve as a data service module in system 200, used to uniformly manage the data generated during model training, and provide data services such as training data management, reward determination, and model parameter synchronization to the execution module 240.

[0110] By setting up the data management module 250, the system 200 can separate the training data management process, reward calculation process, and model parameter synchronization process from the model training process and environment interaction process, enabling multiple execution units to share a unified data management mechanism, thereby improving the reusability of training data and reducing the degree of data coupling between different execution units.

[0111] In some embodiments, the data management module 250 may include a parameter synchronization module. The parameter synchronization module may, in response to updating the model's parameters via a third subtask, provide a portion of the parameters to the execution unit performing the first subtask for further determining the action sequence, wherein the portion represents the changed parts of the parameters relative to historical parameter data already provided to the execution unit.

[0112] For example, the parameter synchronization module can determine the parameters that have changed during the current model parameter update relative to the historical model parameters already acquired by the first execution unit, and provide these changed parameters to the first execution unit. The first execution unit can then update its local model parameters based on the received parameter change information and continue executing the determined action sequence based on the updated model. By synchronizing only the changed parameters, the amount of data transmitted during model parameter synchronization can be reduced, thereby lowering the data transmission overhead and improving the synchronization efficiency of model parameter updates.

[0113] In some embodiments, the data management module 250 may include a training data management module, which can receive training trajectory data generated during the execution of multiple subtasks and manage the training trajectory data in a unified manner. For example, the training trajectory data includes at least one of the following: environmental feedback data, action sequences, or model parameter information. The training data management module can perform storage, organization, querying, and data transformation on the training trajectory data, enabling different execution units to access the training data based on a unified data organization method. Unified management of the training trajectory data avoids multiple execution units maintaining training data separately, improving the consistency and reusability of the training data.

[0114] Continue to refer to Figure 3 The data management module 250 may further include a reward determination module. The reward determination module can receive training trajectory data generated during the execution of multiple subtasks. The training trajectory data includes at least one of the following: environmental feedback data, action sequences, or model parameter information. Furthermore, the reward determination module can determine reward data corresponding to the training trajectory data based on the training trajectory data, and the reward data is used to generate training samples.

[0115] In some embodiments, the reward determination module can obtain a reward function configuration, which is associated with the type of training task. For example, different types of training tasks can correspond to different reward function configurations. After determining the training task type, the reward determination module can call one or more calculation modules to perform calculations on the training trajectory data according to the corresponding reward function configuration to determine the reward data. By making the reward determination process independent of the model training process, the reward calculation method can be flexibly configured according to different training tasks without modifying the training process in the execution module 240, thereby improving the configurability of reward calculation and the adaptability of the training framework to different training tasks.

[0116] By setting up a parameter synchronization module, a training data management module, and a reward determination module, the data management module 250 can decouple data-intensive processing processes such as model parameter synchronization, training data management, and reward calculation from the model training process and the environment interaction process, thereby achieving unified management and sharing of training data, providing consistent data service support for different types of model training tasks, and reducing the degree of data coupling between different execution units in the execution module 240.

[0117] Continue to refer to Figure 3 System 200 may include a resource management module 260. The resource management module 260 can serve as a resource supply module in system 200, used to provide computing resources to the training process and to manage different types of computing resources in a unified manner.

[0118] By setting up the resource management module 260, the system 200 can separate the training process from the underlying computing resources, so that the task scheduling module 220 and the execution module 240 do not need to implement training logic separately for different types of hardware, but can obtain the required computing resources through a unified resource access method, thereby improving the adaptability and scalability of the training infrastructure to different hardware platforms.

[0119] In some embodiments, the resource management module 260 may include a unified computing power interface and various types of computing resources. The unified computing power interface serves as a unified entry point for the training process to access underlying computing resources, shielding the differences between different types of hardware nodes and providing a unified method for accessing computing resources to the training process. For example, different types of hardware nodes can provide computing resources to the training process through the unified computing power interface, and the training process can call upon the corresponding computing resources to execute training tasks without adjusting its training logic based on hardware type.

[0120] In some embodiments, computing resources can originate from multiple types of hardware nodes. For example, these multiple types of hardware nodes may include: a first type of computing resource for performing model training-related computations; a second type of computing resource for performing environment interaction or environment simulation-related computations; a third type of computing resource for performing model inference-related computations; or a fourth type of computing resource for performing tool calls or other general computing tasks. It should be understood that the above-mentioned different types of computing resources are merely examples, and other types of computing resources can be configured according to the actual training task requirements.

[0121] In some embodiments, the resource management module 260 can determine target computing resources from various types of hardware nodes based on the resource configuration results provided by the task scheduling module 220, and provide the target computing resources to the training process through a unified computing power interface. For example, different execution stages or different execution units can obtain corresponding types of computing resources to meet the computing needs of different stages of the training process.

[0122] Continue to refer to Figure 3 The resource management module 260 may also include a resource communication network. The resource communication network can be used to support data transmission between different hardware nodes, providing communication capabilities for data exchange between different computing resources. For example, when the training process involves the collaborative execution of multiple computing resources, the resource communication network can be used to transmit model parameters, training samples, environmental feedback data, and other training data to support data interaction between multiple computing resources. In some cases, multiple functional modules in system 200 can transmit and communicate data through predefined interface specifications. These predefined interface specifications may include at least one of the following: task submission interface specification, resource allocation interface specification, environment provisioning interface specification, environment interaction interface specification, data exchange interface specification, or runtime interface specification.

[0123] In some embodiments, each functional module can exchange task information, resource information, environment information, and training data according to the corresponding interface specifications, without relying on the internal implementation of other functional modules. For example, the task access module 210 can provide a task description to the task scheduling module 220 through the task submission interface specification, the task scheduling module 220 can request computing resources from the resource management module 260 through the resource allocation interface specification, the environment adaptation module 230 can provide environment instances and environment interaction capabilities to the execution module 240 through the environment provision interface specification and the environment interaction interface specification, and the execution module 240 and the data management module 250 can transmit data such as action sequences, environment feedback data, training samples, and model parameters through the data exchange interface specification.

[0124] By adopting predefined interface specifications, different functional modules can interact with each other simply by following a unified interface agreement, without needing to be aware of the internal processing logic of other functional modules. Therefore, when a functional module is replaced, expanded, or upgraded, as long as the interface specification remains consistent, other functional modules do not need corresponding modifications, thereby reducing the coupling between functional modules and improving the modularity and scalability of the system. Furthermore, using unified data representation and interface specifications can reduce the computational overhead and data transmission latency caused by data format conversion between different functional modules, improving the data flow efficiency and overall execution efficiency of the training process.

[0125] Figure 4 A flowchart of an example process 400 for model training under certain conditions is shown. Process 400 can be implemented at model training system 110. See below for reference. Figure 4 To describe process 400.

[0126] like Figure 4As shown in box 410, in response to receiving a request for a training task, the model training system 110 converts the request into a task description with a predetermined format, wherein the type of training task includes at least one of the following: a model training task related to interaction with the physical environment, a model training task related to tool invocation, or a model training task related to the inference process.

[0127] In box 420, the model training system 110 determines the training process corresponding to the training task and the computing resources used to execute the training process based on the task description.

[0128] In box 430, the model training system 110 determines an environment instance for the training process, which is associated with the type of training task.

[0129] In box 440, the model training system 110 executes multiple subtasks corresponding to the training process based on computing resources and environment instances. The multiple subtasks are executed in parallel asynchronously, and data is transmitted between the multiple subtasks in a unified data format.

[0130] In some cases, the model training system 110 can determine the type of training task; determine the target task description template from multiple task description templates based on the type; and generate a task description based on the target task description template.

[0131] In some cases, multiple task description templates include: a first task description template corresponding to a model training task related to interaction with the physical environment; a second task description template corresponding to a model training task related to tool invocation; and a third task description template corresponding to a model training task related to the inference process.

[0132] In some scenarios, the model training system 110 can receive training task requests through a unified interface gateway; perform verification on the requests, including at least one of the following: parameter integrity verification, resource quota verification, or permission verification; and in response to the request passing verification, convert the request into a task description.

[0133] In some cases, a request for a training task is associated with a requester identifier, and the training process is executed in an execution environment corresponding to the requester identifier. The execution environment corresponds to the requester identifier and includes at least one of the following: resource usage scope, data access rights, and namespace.

[0134] In some cases, the model training system 110 can determine a workflow including multiple execution stages based on the task description; obtain predefined configuration information indicating the dependencies between multiple execution stages; and determine the execution order of multiple execution stages in the workflow based on the dependencies, so as to determine the training process.

[0135] In some cases, the model training system 110 can determine the task information of the training task; determine the historical execution information corresponding to the training task; determine the hardware capability information of the candidate hardware nodes; and determine the computing resources based on at least two of the task information, historical execution information, and hardware capability information.

[0136] In some cases, task information indicates at least one of the following: the type of training task; the training phase of the training task; the size of the training data for the training task; or the frequency of environmental interactions for the training task.

[0137] In some cases, environment instances are accessed through a predefined unified environment interaction interface, which defines the standard interactive operations for environment instances and the representation specifications of the observation space and action space of environment instances.

[0138] In some cases, the model training system 110 can determine the environment instance corresponding to the training task from the environment instance pool, wherein the capacity of the environment instance pool is adjusted based on the environment interaction load corresponding to the training process, and the environment instance pool is used for at least one of the following: creating an environment instance, allocating an environment instance, or releasing an environment instance.

[0139] In some cases, the unified environment interaction interface communicates with the environment instance through at least one adapter, which includes at least one of the following: a first adapter associated with the physical simulation environment; a second adapter associated with the tool invocation environment; or a third adapter associated with the inference verification environment.

[0140] In some cases, multiple subtasks include: a first subtask for determining the action sequence of the model; a second subtask for performing interactive operations with environmental instances based on the action sequence to obtain environmental feedback data; and a third subtask for updating the model's parameters based on training samples determined based on the action sequence and environmental feedback data.

[0141] In some cases, the first, second, and third subtasks are executed in their respective independent execution units, and they communicate with each other through asynchronous message queues. Environmental feedback data, parameter-related data, and reward data are stored in a unified cache area.

[0142] In some cases, the model training system 110 may, in response to updating the model's parameters through a third subtask, provide a portion of the parameters to the execution unit used to perform the first subtask for further determining the action sequence, wherein a portion of the parameters is the changed part relative to historical parameter data already provided to the execution unit.

[0143] In some cases, the model training system 110 can receive training trajectory data generated during the execution of multiple subtasks. The training trajectory data includes at least one of the following: environmental feedback data, action sequences, or model parameter information; and reward data corresponding to the training trajectory data is determined based on the training trajectory data. The reward data is used to generate training samples.

[0144] In some cases, the model training system 110 can obtain a reward function configuration, which is associated with the type of training task; and according to the reward function configuration, call the corresponding calculation module to perform calculations on the training trajectory data to determine the reward data.

[0145] In some cases, computing resources come from multiple types of hardware nodes, which provide computing resources to the training process through a unified computing power interface.

[0146] In some cases, the training process involves multiple functional modules, including: a task access module, which receives requests and converts them into task descriptions; a task scheduling module, which determines the training process based on the task descriptions; an environment adaptation module, which determines environment instances and provides a unified access interface to these instances; an execution module, which executes multiple subtasks; a data management module, which stores and manages the training data generated by the multiple subtasks; and a resource management module, which provides computing resources to the training process.

[0147] In some cases, multiple functional modules transmit and communicate data through predefined interface specifications. The predefined interface specifications include at least one of the following: task submission interface specification; resource allocation interface specification; environment provisioning interface specification; environment interaction interface specification; data exchange interface specification; or runtime interface specification.

[0148] A corresponding apparatus for implementing the above methods or processes is also provided.

[0149] Figure 5 Block diagrams of an apparatus 500 for model training in several scenarios are shown. Apparatus 500 can be implemented as or included in the model training system 110. The various modules / components in apparatus 500 can be implemented by hardware, software, firmware, or any combination thereof.

[0150] like Figure 5As shown, the apparatus 500 includes a description determination module 510, configured to convert a request for a training task into a task description with a predetermined format in response to receiving such a request; a process determination module 520, configured to determine a training process and computing resources based on the task description; an environment determination module 530, configured to determine an environment instance for the training process; and an execution module 540, configured to execute multiple sub-tasks corresponding to the training process based on the computing resources and environment instance.

[0151] In some cases, the description determination module 510 can also be configured to determine the type of the training task; determine a target task description template from multiple task description templates based on the type; and generate a task description based on the target task description template. In some cases, multiple task description templates include: a first task description template corresponding to a model training task related to interaction with the physical environment; a second task description template corresponding to a model training task related to tool invocation; and a third task description template corresponding to a model training task related to the inference process. In some cases, the description determination module 510 may also be configured to receive requests for training tasks through a unified interface gateway; perform verification on the requests, including at least one of the following: parameter integrity verification, or resource quota verification, or permission verification; and, in response to the requests passing verification, convert the requests into task descriptions.

[0152] In some cases, a request for a training task is associated with a requester identifier, and the training process is executed in an execution environment corresponding to the requester identifier. The execution environment corresponds to the requester identifier and includes at least one of the following: resource usage scope, data access rights, and namespace.

[0153] In some cases, the process determination module 520 can also be configured to determine a workflow including multiple execution stages based on the task description; obtain predefined configuration information indicating the dependencies between the multiple execution stages; and determine the execution order of the multiple execution stages in the workflow based on the dependencies, so as to determine the training process.

[0154] In some cases, the process determination module 520 can also be configured to determine the task information of the training task; determine the historical execution information corresponding to the training task; determine the hardware capability information of the candidate hardware nodes; and determine the computing resources based on at least two of the task information, historical execution information, and hardware capability information.

[0155] In some cases, task information indicates at least one of the following: the type of training task; the training phase of the training task; the size of the training data for the training task; or the frequency of environmental interactions for the training task.

[0156] In some cases, environment instances are accessed through a predefined unified environment interaction interface, which defines the standard interactive operations for environment instances and the representation specifications of the observation space and action space of environment instances.

[0157] In some cases, the environment determination module 530 can also be configured to determine the environment instance corresponding to the training task from the environment instance pool, wherein the capacity of the environment instance pool is adjusted based on the environment interaction load corresponding to the training process, and the environment instance pool is used for at least one of the following: creating an environment instance, allocating an environment instance, or releasing an environment instance.

[0158] In some cases, the unified environment interaction interface communicates with the environment instance through at least one adapter, which includes at least one of the following: a first adapter associated with the physical simulation environment; a second adapter associated with the tool invocation environment; or a third adapter associated with the inference verification environment.

[0159] In some cases, multiple subtasks include: a first subtask for determining the action sequence of the model; a second subtask for performing interactive operations with environmental instances based on the action sequence to obtain environmental feedback data; and a third subtask for updating the model's parameters based on training samples determined based on the action sequence and environmental feedback data.

[0160] In some cases, the first, second, and third subtasks are executed in their respective independent execution units, and they communicate with each other through asynchronous message queues. Environmental feedback data, parameter-related data, and reward data are stored in a unified cache area.

[0161] In some cases, the execution module 540 may also be configured to provide a portion of the parameters to the execution unit for executing the first subtask in response to updating the parameters of the model via the third subtask, for further determining the sequence of actions, wherein a portion of the parameters are the changed parts relative to historical parameter data provided to the execution unit.

[0162] In some cases, the execution module 540 may also be configured to receive training trajectory data generated during the execution of multiple subtasks, the training trajectory data including at least one of the following: environmental feedback data, action sequences, or model parameter information; and reward data corresponding to the training trajectory data determined based on the training trajectory data, the reward data being used to generate training samples.

[0163] In some cases, the execution module 540 can also be configured to obtain a reward function configuration, which is associated with the type of training task; and, based on the reward function configuration, call the corresponding calculation module to perform calculations on the training trajectory data to determine the reward data.

[0164] In some cases, computing resources come from multiple types of hardware nodes, which provide computing resources to the training process through a unified computing power interface.

[0165] In some cases, the training process involves multiple functional modules, including: a task access module, which receives requests and converts them into task descriptions; a task scheduling module, which determines the training process based on the task descriptions; an environment adaptation module, which determines environment instances and provides a unified access interface to these instances; an execution module, which executes multiple subtasks; a data management module, which stores and manages the training data generated by the multiple subtasks; and a resource management module, which provides computing resources to the training process.

[0166] In some cases, multiple functional modules transmit and communicate data through predefined interface specifications. The predefined interface specifications include at least one of the following: task submission interface specification; resource allocation interface specification; environment provisioning interface specification; environment interaction interface specification; data exchange interface specification; or runtime interface specification.

[0167] The modules included in device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some cases, one or more modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 500 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), and so on.

[0168] Figure 6 A block diagram of an electronic device 600 in which one or more examples may be implemented is shown. It should be understood that... Figure 6 The electronic device 600 shown is merely exemplary and should not be construed as limiting the functionality and scope of the examples described herein. Figure 6 The electronic device 600 shown can be used to implement the model training system 110 discussed above.

[0169] like Figure 6 As shown, electronic device 600 is in the form of a general-purpose electronic device. Components of electronic device 600 may include, but are not limited to, one or more processing units or processors 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processor 610 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 600.

[0170] Electronic device 600 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 620 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof). Storage device 630 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 600.

[0171] Electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 6 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 620 may include computer program product 625 having one or more program modules configured to perform various methods or actions of various examples.

[0172] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 600 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 600 can operate in a networked environment using logical connections to one or more other servers, networked personal computers, or another network node.

[0173] Input device 650 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 660 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 600 can also communicate with one or more external devices (not shown) via communication unit 640 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 600, or with any device that enables electronic device 600 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via an input / output (I / O) interface (not shown).

[0174] A computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. A computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0175] The flowcharts and / or block diagrams of the methods, apparatus, devices, and computer program products referred to herein describe various aspects. It should be understood that each block of the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0176] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0177] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0178] The flowcharts and block diagrams in the accompanying figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products under various scenarios. In this respect, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0179] Various examples have been described above. The foregoing descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for model training, comprising: In response to receiving a request for a training task, the request is converted into a task description with a predetermined format, wherein the type of the training task includes at least one of the following: a model training task related to interaction with the physical environment, a model training task related to tool invocation, or a model training task related to the inference process. Based on the task description, determine the training process corresponding to the training task and the computing resources used to execute the training process; Determine an environment instance for the training process, the environment instance being associated with the type of the training task; as well as Based on the computing resources and the environment instance, multiple subtasks corresponding to the training process are executed. The multiple subtasks are executed in parallel asynchronously, and data is transmitted between the multiple subtasks in a unified data format.

2. The method of claim 1, wherein converting the request into the task description comprises: Determine the type of the training task; Based on the type, a target task description template is determined from a plurality of task description templates, the plurality of task description templates being used to indicate the predetermined format; as well as The task description is generated based on the target task description template.

3. The method according to claim 2, wherein, The multiple task description templates include: The first task description template corresponding to the model training task related to the interaction with the physical environment; The second task description template corresponding to the model training task related to the tool invocation; and The third task description template corresponds to the model training task related to the reasoning process.

4. The method according to claim 1, further comprising: The training task request is received through a unified interface gateway; The request is validated, and the validation includes at least one of the following: parameter integrity validation, resource quota validation, or permission validation; and In response to the verification of the request, the request is converted into the task description.

5. The method of claim 1, wherein the request for the training task is associated with a requester identifier of the request, and the training process is executed in an execution environment corresponding to the requester identifier, the execution environment corresponding to the requester identifier and including at least one of the following: resource usage scope, data access rights, and namespace.

6. The method of claim 1, wherein determining the training process includes: Based on the task description, a workflow comprising multiple execution phases is determined; Obtain predetermined configuration information, which indicates the dependencies between the plurality of execution stages; as well as The execution order of the multiple execution stages in the workflow is determined based on the dependencies, thereby determining the training process.

7. The method of claim 1, wherein determining the computing resources comprises: Determine the task information for the training task; Determine the historical execution information corresponding to the training task; Determine the hardware capability information of candidate hardware nodes; The computing resources are determined based on at least two of the task information, the historical execution information, and the hardware capability information.

8. The method of claim 7, wherein the task information indicates at least one of the following: The type of training task; The training phase of the training task; The training data size of the training task; or The frequency of environmental interactions for the training task.

9. The method according to claim 1, wherein the environment instance is accessed through a preset unified environment interaction interface, the unified environment interaction interface being used to define standard interactive operations for the environment instance and the representation specifications of the observation space and action space of the environment instance.

10. The method of claim 9, wherein determining the environment instance comprises: An environment instance corresponding to the training task is determined from the environment instance pool, wherein the capacity of the environment instance pool is adjusted based on the environment interaction load corresponding to the training process, and the environment instance pool is used for at least one of the following: creating the environment instance, allocating the environment instance, or releasing the environment instance.

11. The method of claim 9, wherein the unified environment interaction interface communicates with the environment instance through at least one adapter, the at least one adapter comprising at least one of the following: The first adapter related to the physical simulation environment; A second adapter related to the tool's calling environment; or A third adapter related to the inference verification environment.

12. The method of claim 1, wherein the plurality of subtasks comprises: The first subtask is used to determine the action sequence of the model. The second subtask is used to perform interactive operations based on the action sequence and the environment instance to obtain environmental feedback data, and The third subtask is used to update the parameters of the model based on training samples, which are determined based on the action sequence and the environmental feedback data.

13. The method of claim 12, wherein the first subtask, the second subtask, and the third subtask are executed in their respective independent execution units, and the first subtask, the second subtask, and the third subtask communicate with each other via an asynchronous message queue, and the environmental feedback data, the data related to the parameters, and the reward data are stored in a unified cache area.

14. The method of claim 12, further comprising: In response to updating the parameters of the model through the third subtask, a portion of the parameters is provided to the execution unit for performing the first subtask to further determine the action sequence, wherein the portion is the part of the parameters that has changed relative to historical parameter data already provided to the execution unit.

15. The method of claim 12, further comprising: The system receives training trajectory data generated during the execution of the plurality of subtasks, wherein the training trajectory data includes at least one of the following: the environmental feedback data, the action sequence, or the parameter information of the model; and Based on the training trajectory data, reward data corresponding to the training trajectory data is determined, and the reward data is used to generate the training samples.

16. The method of claim 15, wherein, Determining the reward data includes: Obtain the reward function configuration, which is associated with the type of the training task; and According to the reward function configuration, the corresponding calculation modules are called to perform calculations on the training trajectory data to determine the reward data.

17. The method according to claim 1, wherein the computing resources originate from multiple types of hardware nodes, and the multiple types of hardware nodes provide the computing resources to the training process through a unified computing power interface.

18. The method according to claim 1, wherein the training process involves multiple functional modules, the multiple functional modules including: The task access module is used to receive the request and convert the request into the task description. The task scheduling module is used to determine the training process based on the task description. The environment adaptation module is used to determine the environment instance and provide a unified access interface for the environment instance. The execution module is used to execute the multiple sub-tasks. The data management module is used to store and manage the training data generated by the multiple sub-tasks, and The resource management module is used to provide the computing resources to the training process.

19. The method according to claim 18, wherein, The multiple functional modules transmit and communicate through a predefined interface specification, which includes at least one of the following: Task submission interface specifications; Resource allocation interface specification; Environmental supply interface specifications; Environment interaction interface specification; Data exchange interface specification; or Runtime interface specification.

20. An apparatus for model training, comprising: The description determination module is configured to, in response to receiving a request for a training task, convert the request into a task description with a predetermined format, wherein the type of the training task includes at least one of the following: a model training task related to interaction with the physical environment, a model training task related to tool invocation, or a model training task related to the inference process. The process determination module is configured to determine the training process corresponding to the training task and the computing resources used to execute the training process based on the task description. An environment determination module is configured to determine an environment instance for the training process, the environment instance being associated with the type of the training task. as well as The execution module is configured to execute multiple subtasks corresponding to the training process based on the computing resources and the environment instance. The multiple subtasks are executed in parallel asynchronously, and data is transmitted between the multiple subtasks in a unified data format.

21. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 19.

22. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 19.

23. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 19.

Images