Method, apparatus, device, storage medium and program product for data processing

Abstract

According to embodiments of this disclosure, a method, apparatus, device, storage medium, and program product for data processing are provided. The method includes: acquiring a trajectory data sequence corresponding to at least one type of reinforcement learning task of multiple types, the trajectory data sequence including step data corresponding to multiple steps in an interaction process; encapsulating the trajectory data sequence based on a trajectory data structure to obtain standardized trajectory data, the trajectory data structure including a unified step representation and multiple data regions associated with the unified step representation, the unified step representation including type-specific fields corresponding to multiple types, the multiple data regions being used to record portions of the step data generated by different processing stages; and providing the standardized trajectory data as training samples for model training.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to methods, apparatus, devices, computer-readable storage media, and computer program products for data processing. Background Technology

[0002] With the development of computer technology, reinforcement learning is being applied to an increasing number of task types. The data generated during the interaction process of different task types often differ in field composition, organization, and value format. In scenarios where interaction data from different task types needs to be aggregated for model training, it is usually necessary to define a separate data organization method for each task type and perform corresponding conversions and adaptations during aggregation and retrieval. Therefore, a unified approach is needed to organize, store, and process interaction data from multiple task types. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for data processing is provided. The method includes: acquiring a trajectory data sequence corresponding to at least one type of reinforcement learning task of multiple types, the trajectory data sequence including step data corresponding to multiple steps in an interaction process; encapsulating the trajectory data sequence based on a trajectory data structure to obtain standardized trajectory data, the trajectory data structure including a unified step representation and multiple data regions associated with the unified step representation, the unified step representation including type-specific fields corresponding to multiple types, the multiple data regions being used to record portions of the step data generated by different processing stages; and providing the standardized trajectory data as training samples for model training.

[0004] In a second aspect of this disclosure, an apparatus for data processing is provided. The apparatus includes: a trajectory data sequence acquisition module configured to acquire a trajectory data sequence corresponding to at least one of multiple types of reinforcement learning tasks, the trajectory data sequence including step data corresponding to multiple steps in an interaction process, the step data including observation data, action data, and reward data; a standardized trajectory data acquisition module configured to encapsulate the trajectory data sequence based on a trajectory data structure to obtain standardized trajectory data, the trajectory data structure including a unified step representation and multiple data regions associated with the unified step representation, the unified step representation including type-specific fields corresponding to multiple types, the multiple data regions being used to record portions of the step data generated by different processing stages; and a standardized trajectory data providing module configured to provide standardized trajectory data as training samples for model training.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores computer-executable instructions that can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0008] It should be understood that the description in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown; Figure 2 A schematic diagram of a trajectory data structure for data processing according to some embodiments of the present disclosure is shown; Figure 3 A schematic diagram of an example architecture for data processing according to some embodiments of the present disclosure is shown; Figure 4 A flowchart of a process for data processing according to some embodiments of the present disclosure is shown; Figure 5 A block diagram of an apparatus for data processing according to some embodiments of the present disclosure is shown; and Figure 6 A block diagram of an electronic device capable of implementing one or more embodiments of the present disclosure is shown. Detailed Implementation

[0010] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0011] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0012] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0013] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0014] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0016] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0017] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0018] As used in this paper, the term "model" refers to a system that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. In this paper, "model" may also be referred to as a "machine learning model," a "machine learning network," or simply a "network," and these terms are used interchangeably. A model can also include different types of processing units or networks.

[0019] As used in this paper, the term "agent" refers to a software entity that can autonomously or semi-autonomously perceive its environment, make decisions, and perform actions to complete a specific task. Agents can include, but are not limited to, conversational agents based on large language models, automated workflow agents, and multimodal data processing agents.

[0020] The term "reinforcement learning (RL)" as used in this paper refers to a class of machine learning methods that typically involve an agent interacting with its environment. Reinforcement learning can instruct agents to adjust their policies based on reward signals obtained during the interaction.

[0021] Figure 1 A schematic diagram of an example environment 100 capable of performing data processing under certain circumstances is shown. Figure 1 The diagram illustrates two distinct phases of the machine learning model 150: a training phase 130 and an application phase 140. A pre-training phase may precede the training phase 130, and a testing phase may follow the training phase (not shown in the diagram).

[0022] Example environment 100 involves model training system 110 and model application system 120. In training phase 130, training phase 130 is configured to perform training of machine learning model 150 using training dataset 115. At the start of training, the various components in machine learning model 150 may have initial parameter values. The training process involves updating the parameter values ​​of machine learning model 150 to desired values ​​based on data in training dataset 115. Training tasks are used to assist in updating the parameters of machine learning model 150.

[0023] In some embodiments, the data processing described herein can be implemented by a computing device, which may be implemented as or included in the model training system 110. For example, the model training system 110 may acquire trajectory data sequences and encapsulate the trajectory data sequences based on trajectory data structures to obtain standardized trajectory data, and then provide the standardized trajectory data to the training dataset 115 as training samples for training the machine learning model 150.

[0024] During the training phase 130, the machine learning model 150 can learn strong generalization capabilities using a training dataset 115 that includes a large amount of data. After training, the parameter values ​​of the machine learning model 150 have been updated to have pre-trained parameter values. In some embodiments, the machine learning model 150 may include a Vision-Language-Action (VLA) model, and the training dataset 115 used to train the machine learning model 150 may include multiple training samples, each training sample including sample language input, sample visual input, and sample action instructions. The trained machine learning model 150 can accurately generate corresponding action instructions based on the received language input and visual input.

[0025] In some embodiments, a testing phase may be included after the training phase 130, where the performance of the machine learning model 150 can be tested using a test dataset. The dataset used in the testing phase is the same type as that used in the fine-tuning phase.

[0026] In the application phase 140, the obtained machine learning model 150, with trained parameter values, can be provided to the model application system 120 for use. In the application phase 140, the machine learning model 150 can be used to process the corresponding model input 125 in the real-world scenario and provide the corresponding model output 126. As an example only, the model input 125 may include language input and visual input, and the model output 126 may include corresponding action instructions.

[0027] It should be understood that Figure 1The components and arrangements in the illustrated example architecture 100 are merely examples, and a computing system suitable for implementing the exemplary implementations described in this disclosure may include one or more different components, other components, and / or different arrangements. For example, although shown as separate, the model training system 110 and the model application system 120 may be integrated in the same system or device. For example, at least the model training system 110 and the model application system 120 may be integrated in a model training system or device. Implementations of this disclosure are not limited in this respect. Figure 1 In this system, the model training system 110 and the model application system 120 can be deployed on any suitable electronic device. This electronic device can be any type of computing-capable device, including terminal devices or server devices.

[0028] Machine learning model 150 may include, but is not limited to, neural network models, deep learning models, visual language action models (VLA), and other forms of models that can be trained on time-series data. In some embodiments, machine learning model 150 may be a model designed to make predictions based on action sequences, which may include actions at multiple consecutive time steps, for example.

[0029] The model training system 110 and the model application system 120 can be implemented on any type of electronic device. This electronic device can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, virtual reality / augmented reality (VR / AR) devices, robot control terminals, in-vehicle terminals, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the model training system 110 and the model application system 120 may also be implemented on independent physical servers, server clusters consisting of multiple physical servers, or distributed systems. They may also be implemented in cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms.

[0030] When the model training system 110 and the model application system 120 are implemented on different electronic devices, a communication connection can be established between them. This communication connection can be established via wired or wireless means. The communication connection can include, but is not limited to, Bluetooth, mobile network, Universal Serial Bus (USB), and Wireless Fidelity (WiFi) connections. In some embodiments, the model training system 110 and the model application system 120 can interact with each other through their communication connection, for example, transmitting the trained parameter values ​​determined in the training phase 130 to the machine learning model 150 used in the application phase 140.

[0031] It should be understood that the data processing described herein can be implemented using various architectures. As an example, the data processing described above can be executed locally by a single computing device, which locally acquires, encapsulates, and provides standardized trajectory data without relying on other devices. As another example, the data processing can also be executed by a server; for instance, a computing device collects trajectory data sequences and sends them to a server via a communication connection, where the server encapsulates the data and returns or persistently stores the standardized trajectory data. As yet another example, the data processing can also be executed collaboratively by multiple devices. For example, multiple devices, each carrying out different processing stages such as environment execution, policy inference, reward calculation, and training, can collaboratively generate different parts of the step data, and one of these devices or a dedicated device can encapsulate the trajectory data sequences. The processing corresponding to each of these examples can be deployed in a distributed manner across one or more computing devices or servers, and the scope of the solution is not limited in this respect.

[0032] As mentioned above, in scenarios where reinforcement learning is applied to multiple types of tasks, the data generated during the interaction process of different types of tasks often differ in field composition, organization, and value format. For example, trajectories interacting with the simulation environment (e.g., trajectories output by an embodied agent), trajectories interacting with the tool execution environment (e.g., trajectories output by an agent), and trajectories interacting with the inference verification environment (e.g., trajectories output by an inference model) may have significant differences in field composition and organization. Correspondingly, data from the same step may be generated separately by different processing stages such as the environment execution engine, policy inference engine, reward calculation, and training. In scenarios where different types of interaction data are organized in independent ways, additional conversion and adaptation are often required during collection, storage, retrieval, and training. Consequently, it is difficult to expand the data without affecting existing data when adding new task types or fields.

[0033] In view of this, a data processing scheme is proposed. According to this scheme, a trajectory data sequence is obtained, corresponding to at least one type of reinforcement learning task. This trajectory data sequence may include step data corresponding to multiple steps in the interaction process. Accordingly, the trajectory data sequence is encapsulated based on a trajectory data structure to obtain standardized trajectory data. The trajectory data structure includes a unified step representation and multiple data regions associated with the unified step representation. The unified step representation includes type-specific fields corresponding to multiple types, and the multiple data regions are used to record the portions of the step data generated by different processing stages. Then, the standardized trajectory data is provided as training samples for model training.

[0034] In this approach, by employing a unified step representation to carry multiple types of step data, and using multiple data regions divided according to processing stages to record the portions generated by different processing stages, multiple types of interactive data can be compatible with a single data structure. This reduces the conversion and adaptation required during aggregation and retrieval, thereby lowering the overhead of cross-type data processing. Furthermore, it can improve the reusability of training samples and data processing efficiency, and enhance the scalability of the data structure to new task types and fields.

[0035] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of the scheme.

[0036] Various examples of this scheme are further described in detail below with reference to the accompanying drawings. In the following description, the examples will primarily be described with respect to the model training system 110. It should be understood that the actions described with respect to the model training system 110 can also be performed by the model training system 110 in conjunction with its server (e.g., a server).

[0037] In embodiments of this disclosure, the model training system 110 acquires a trajectory data sequence, which corresponds to at least one type of reinforcement learning task among multiple types. The trajectory data sequence includes step data corresponding to multiple steps in the interaction process. In some embodiments, the reinforcement learning task can refer to a task that can be processed using reinforcement learning, and may include, but is not limited to, tasks that interact with a simulation environment, tasks that interact with a tool execution environment, and tasks that interact with an inference verification environment.

[0038] In some embodiments, multiple types of reinforcement learning tasks may include a first type of reinforcement learning task. The first type corresponds to trajectories interacting with a simulation environment; for example, the model training system 110 may acquire a sequence of trajectory data output by a machine learning model. The machine learning model may include one or more models. If the machine learning model includes multiple models, these multiple models may include multiple types of models. The machine learning model may include a Vision-Language-Action (VLA) model. A VLA model is a multimodal model that integrates vision, language, and action. The machine learning model may, for example, include at least a Visual Language Model (VLM) and an action model. The Visual Language Model has the ability to process images and natural language. The Action Model can control robotic devices to perform various actions. The machine learning model may also include, for example, an image processing model, a text processing model, and so on.

[0039] In some embodiments, the machine learning model may include a content-generative model capable of generating corresponding outputs based on model inputs. In some implementations, the machine learning model may receive text-modal model inputs (e.g., natural language and / or machine language) and / or non-text-modal model inputs (e.g., images, speech, video, etc.), and may obtain corresponding model outputs based on the model inputs and prompts, thereby completing the task execution.

[0040] In some embodiments, the multiple types of reinforcement learning tasks may include a second type of reinforcement learning task. The second type corresponds to trajectories interacting with a tool execution environment; for example, the model training system 110 may acquire trajectory data sequences output by an agent. The multiple types of reinforcement learning tasks may also include a third type of reinforcement learning task. The third type corresponds to trajectories interacting with an inference verification environment; for example, the model training system 110 may acquire trajectory data sequences output by an inference model.

[0041] In some embodiments, the trajectory data sequence may include step data corresponding to multiple steps in the interaction process. The trajectory data sequence can refer to a sequence composed of multiple steps in the interaction process. For example, the trajectory data sequence may include, but is not limited to, a sequence of step data corresponding to a round or a portion of a round. The interaction process can refer to a series of interactions between the agent and the environment, such as, but not limited to, multiple steps, each step of which may include observation, action, and reward.

[0042] In some embodiments, step data can be data associated with a step in the interaction process, such as, but not limited to, at least one of observation data, action data, and reward data. In some examples, observation data can be data used to describe the state of the environment, and may include, but is not limited to, state description information collected for different types of reinforcement learning tasks corresponding to that type. Action data can refer to data corresponding to the action taken by the agent at a certain step in the interaction process. For example, action data can include representations of actions corresponding to different types of tasks. Reward data can refer to feedback signals used to evaluate actions or states. In some embodiments, reward data can come from multiple reward sources, such as, but not limited to, environmental reward sources, rule-based reward sources, model reward sources, and preference reward sources.

[0043] In embodiments of this disclosure, the model training system 110 encapsulates trajectory data sequences based on a trajectory data structure to obtain standardized trajectory data. The trajectory data structure includes at least a unified step representation and multiple data regions associated with the unified step representation. Further, the model training system 110 provides standardized trajectory data as training samples for model training. The unified step representation includes type-specific fields corresponding to multiple types, and the multiple data regions are used to record portions of the step data generated by different processing stages.

[0044] In some embodiments, after receiving trajectory data sequences corresponding to different types of reinforcement learning tasks, the model training system 110 can encapsulate these trajectory data sequences using the trajectory data structure of a trajectory protocol to form standardized trajectory data. This facilitates the subsequent reuse of these trajectory data sequences. In some embodiments, to facilitate the organization, storage, and processing of interactive data from multiple task types, this solution proposes a trajectory protocol, which can be called the Unified Rollout Trajectory Protocol (URTP). References will follow below. Figure 2 and Figure 3 This section describes in detail how to encapsulate these trajectory data sequences based on trajectory data structures to form standardized trajectory data.

[0045] In some embodiments, a trajectory data structure can refer to a data structure used to organize different trajectory data sequences. The trajectory data structure may include a unified step representation, multiple data regions associated with the unified step representation, a header region, a trajectory-level aggregation region, and an extension region. Specific trajectory data structures will be referenced below. Figure 2To describe this. In some examples, a uniform step representation can refer to a data representation that uses a uniform form to represent steps for different types of reinforcement learning tasks. A uniform step representation may include, but is not limited to, type-specific fields corresponding to multiple types, thereby enabling step data from different types of tasks to be carried in a uniform form.

[0046] This approach allows for the use of a single data structure to accommodate multiple types of interactive data. This reduces the need for conversion and adaptation during aggregation and retrieval, thereby lowering the overhead of cross-type data processing.

[0047] The following will be referenced Figure 2 This describes the trajectory data structure used for data processing in this disclosure. Figure 2 A schematic diagram of a trajectory data structure 200 for data processing according to some embodiments of the present disclosure is shown.

[0048] Trajectory data structure 200, for example, can be derived from... Figure 1 The model training system 110 shown is used to encapsulate trajectory data sequences. For example... Figure 2 As shown, the trajectory data structure 200 may include a multi-layered data structure, such as a four-layer data structure. The first layer may include a header region 210, the second layer may include a step sequence 220, the third layer may include a trajectory-level aggregation region 230, and the fourth layer may include an extension region 240. In some examples, the model training system 110 may divide a single trajectory data sequence into four layers: a fixed header, a step sequence, trajectory-level aggregation data, and extension metadata. Accordingly, the trajectory data structure 200 may be processed via a streaming serialization pipeline 250. It should be understood that the number and division of the above layers are merely exemplary, and the scope of the scheme is not limited in this respect.

[0049] In some embodiments, the trajectory data structure 200 may include a header region 210. For example... Figure 2 As shown, the fixed header at header region 210 can have a predetermined length (e.g., a fixed length). Header region 210 can be used to record identification information for the trajectory data sequence. In some examples, the identification information may include a trajectory data sequence identifier 211; for example, the fixed header may include an identifier field indicating the trajectory data sequence. This identifier field may be in a 128-bit UUID encoding format. The identification information may include a task type identifier 212; for example, the fixed header may include a type field indicating the trajectory data sequence. This type field may be in an enumerated data format, for example, the enumerated values ​​may be limited to machine learning model reinforcement learning task (EMBODIED_RL), agent reinforcement learning task (AGENTIC_RL), inference reinforcement learning task (REASONING_RL), or any other suitable value.

[0050] The identification information may include a start timestamp 213 indicating the creation time of the trajectory data sequence. The identification information may include a policy version identifier 214; for example, the fixed header may include a generation policy version field indicating the generation strategy version corresponding to the trajectory data sequence, used to record the policy model version used to generate the current interactive trajectory. In some examples, the fixed header may include a training job identifier field, serving as a unique identifier for the distributed training job to which the trajectory data sequence belongs, thereby achieving the aggregation of all trajectory data sequences under the same batch of training jobs. The fixed header may include an environment configuration identifier field, used as an index to associate with the environment configuration database and bind the environment configuration parameters used to generate the trajectory data sequence. The fixed header may include a protocol version field, used to record the URTP protocol version number used to encapsulate this trajectory. The fixed header may also include a baseline reference policy version field, used to store the baseline reference policy version number used for KL divergence calculation. This provides a version tracing basis for policy-constrained reinforcement learning algorithms. The fixed header may also include a first-step timestamp and a last-step timestamp, used to record the microsecond-level execution timestamps of the first and last interaction steps within the trajectory data sequence, respectively. This defines the complete time interval of a single trajectory data sequence.

[0051] Optionally / additionally, the fixed header may also include a total environment time field, used to calculate the cumulative time spent on environment engine interactions throughout the entire trajectory data sequence, in microseconds. The fixed header may also include a total inference time field, used to calculate the cumulative time spent on inference engine policy operations throughout the entire trajectory data sequence, in microseconds, which can be used for inference performance statistical analysis. The fixed header may also include an environment instance identifier field, used to indicate the globally unique number of the environment instance under the parallel distributed sampling architecture, which can be used to locate the running environment instance corresponding to the trajectory. The fixed header may also include an environment type field, which can be used to mark the underlying type of the simulation environment. The fixed header may also include an environment adapter version field, used to record the version of the environment adapter program used to interface with the simulation environment. The fixed header may also include process worker node identifiers, storing the asynchronous sampling worker process number and the environment execution worker process number that generate the trajectory data sequence, used for distributed cluster node fault tracing, computing power operation and maintenance troubleshooting, etc.

[0052] In some embodiments, the header region 210 can be used to index and access standardized trajectory data. Thus, by setting a header region 210 of a predetermined length to record identification information, standardized trajectory data can be quickly indexed and accessed based on the header region 210, thereby reducing data retrieval latency.

[0053] In some embodiments, the trajectory data structure 200 may include a sequence of steps 220. The sequence of steps 220 may employ a uniform step representation, and each step may include multiple data regions. The uniform step representation may include type-specific fields corresponding to multiple types, thereby enabling the uniform representation of step data from different types of tasks. In some embodiments, the multiple data regions may include a first data region 221, a second data region 222, a third data region 223, a fourth data region 224, and a fifth data region 225. The multiple data regions may be used to record portions of the step data generated by different processing stages. The following illustrations show multiple data regions, their corresponding processing stages, and examples of the content recorded in each data region, intended to illustrate the data regions and not to limit the scope of the scheme:

[0054] By dividing the step data into multiple data areas and recording them separately according to different processing stages, data from different processing stages can be decoupled. This facilitates independent reading and writing and parallel processing by each processing stage, thereby reducing the processing overhead caused by data coupling and improving the clarity and maintainability of data organization.

[0055] In some embodiments, the portion recorded in the first data area 221 (also referred to as the output area of ​​the environment execution engine) may include observation data corresponding to multiple types. For example, in the scenario of trajectory data sequences output by a machine learning model, the observation data may include, but is not limited to, multi-view image frame references, point cloud data, joint states, force / force matrix sensor data, or language commands. For example, in the scenario of trajectory data sequences output by an agent, the observation data may include, but is not limited to, tool return results, DOM tree states, code execution output, and error information. As another example, in the scenario of trajectory data sequences output by an inference model, the observation data may include, but is not limited to, the original question text, historical inference step text, expected output format, or question metadata.

[0056] The first data area 221 may record a portion that includes the original reward value, which can be a floating-point baseline reward value directly calculated and output by the environment execution engine based on the result of a single interaction. This value can be the original input data before the Distributed Reward Computation Framework (DRCAF) performs multi-source reward weighted aggregation operations, such as a value without the superposition and weight adjustment of rule rewards, model rewards, and preference rewards. The first data area 221 may also record end-state data, which can be used to record the round termination status information of the trajectory data sequence in which a single step interaction occurs. In some examples, end-state data may include data recording forced end-state states, such as termination caused by manual truncation, exceeding the step limit, or timeout. In some examples, end-state data may also include natural end-state data. In some examples, end-state data may also include a termination reason enumeration field, the value of which can indicate various termination scenarios such as task completion, task failure, step limit exceeding, execution timeout, environmental anomaly, constraint violation, and user interruption. End-state data may also include data used to record the maximum allowed number of interaction steps configured for the current trajectory data sequence. The termination status data may also include the number of interaction steps actually completed in the trajectory data sequence, thereby fully quantifying the scale of trajectory execution and the termination triggers.

[0057] The first data area 221 may contain environmental state summary data, which can be used for environmental state recovery and consistency comparison. For example, the environmental state summary data can be a byte array type of environmental state hash digest data, generated by the environmental execution engine based on the real-time full environmental state. This field can be used to lightweightly represent the overall environmental state at the moment of interaction, enabling fast consistency comparison of environmental states across different samples and at different times. Correspondingly, this field can serve as an index identifier for environmental snapshots, supporting accurate retrieval of subsequent environmental states and restoration and reproduction of interaction scenarios.

[0058] In some embodiments, the portion recorded in the first data area 221 may further include environmental metadata. Environmental metadata may include custom key-value pairs, thereby supporting flexible and extended storage of various environmental business parameters. In some examples, for scenarios involving trajectory data sequences output by a machine learning model, environmental metadata may include physical simulation time, number of physical substeps, and a list of collision events. For scenarios involving trajectory data sequences output by an agent, environmental metadata may include tool execution time, number of tool retries, and a sandbox state summary. For scenarios involving trajectory data sequences output by an inference model, environmental metadata may include the validator identifier used and inference verification time. This enables unified structured retention of multiple types of environmental data.

[0059] The portion recorded in the first data area 221 may also include environmental step delay, used to record the total latency of a single environmental interaction with microsecond precision. For example, the total latency from the inference engine issuing an action command to the environment engine completing the simulation execution and returning the observation data.

[0060] By recording the above content in the first data area 221, it is possible to support the recovery and consistency comparison of environmental state, thereby improving the reproducibility of trajectory data and reducing the overhead of repeated sampling caused by missing state information.

[0061] In some embodiments, the portion recorded in the second data area 222 may include multiple types of action data. In some embodiments, the action data may be encapsulated using a predetermined method (e.g., oneof). In some embodiments, for scenarios involving trajectory data sequences output by a machine learning model, the action data may include embodied control actions such as end-effector pose, joint torque, joint position, Cartesian velocity, and diffusion model action parameters. For scenarios involving trajectory data sequences output by an agent, the action data may include agent interaction actions such as tool call data (e.g., tool name, tool version, parameters, etc.), code execution data (e.g., code content, language, context, timeout settings, etc.), browser operation data, API call data, and database query data. For scenarios involving trajectory data sequences output by an inference model, the action data may include inference output information such as generated token ID sequences, decoded text, inference tag markers, answer tag markers, and thought chain segment markers. Thus, unified encapsulation and differentiated storage of three types of task actions can be achieved.

[0062] The portion recorded in the second data area 222 may include multi-granularity policy probabilities, which can be multi-granularity policy log probability structures used for hierarchical retention of policy inference probability information, adaptable to both continuous action spaces and discrete token action spaces. Multi-granularity policy log probabilities may include action-level log probabilities, suitable for single-action log probabilities in continuous action spaces. Multi-granularity policy log probabilities may also include token-level log probability lists, suitable for token-by-token probability lists in discrete token spaces, such as recording token numbers, corresponding logarithmic probabilities, information entropy, and sequence position information for each token. Multi-granularity policy log probabilities may also include sequence-level total log probabilities, which can be used to characterize the joint output probability of a complete action sequence.

[0063] The second data region 222 may contain value estimation data, which can be the value estimation structure output during the inference phase. This value estimation data is used to retain the inference side's predicted values ​​of the current state and action's merits, providing initial value basis for subsequent training and optimization. Value estimation data may include, for example, state value, discounted reward, generalized advantage estimation (GAE), discount factor, and GAE smoothing parameters.

[0064] In some embodiments, the second data area 222 can also record key-value (KV) cache reference identifiers and speculative decoding metadata. The second data area 222 can also record inference performance metadata, which can be used to record the runtime overhead and hardware resource usage of a single inference iteration. Inference performance metadata may include microsecond-level total inference latency, preprocessing stage time, decoding stage time, batch processing slot number, speculative decoding acceptance rate, number of KV cache pages occupied, and GPU memory allocation. This provides quantitative data support for inference performance optimization, computational scheduling, and evaluation of acceleration mechanism effectiveness. The second data area 222 can also record the pre-calculated KL divergence penalty term value. The KL divergence penalty term value can be used to penalize policy offsets during reward aggregation, ensuring stable convergence during training.

[0065] By recording action data, multi-granularity policy probability, and value estimation data in the second data region 222, richer usable information can be provided for the training phase. Furthermore, this supports the reuse of the same trajectory data by multiple training algorithms, improving the utilization rate of training samples.

[0066] In some embodiments, the multiple data regions may include a third data region 223, which may be used to record the portion of the step data generated by reward calculation. The third data region 223 may be used to store the original reward values ​​of each of the multiple reward sources. The multiple reward sources may include at least one of environmental reward sources, rule reward sources, and model reward sources. In some examples, the rule reward source may be, for example, the calculation result from a rule engine. The model reward source may be, for example, the inference result from an Outcome Reward Model (ORM) or a Process Reward Model (PRM). The preference reward source may be, for example, the calculation result from a preference rating model. In some embodiments, the third data region 223 may also be used to store the aggregated reward value obtained by weighted aggregation based on the original reward values ​​of each of the multiple reward sources, and the weight configuration used for weighted aggregation. As an example, the aggregated reward value may be obtained by weighted summation based on the original reward values ​​of each of the multiple reward sources and their corresponding weights.

[0067] Specifically, aggregate reward value It can be and The sum of, among which Indicates the number of reward sources. Indicates the first The weight corresponding to each reward source in the weight configuration. Indicates the first The original reward value for each reward source, with KL representing the KL divergence penalty term. Weight configuration can be flexibly configured as needed, for example, for each weight... Appropriate non-negative values ​​can be taken. In some embodiments, the sum of the weights can be equal to the total weight. The corresponding predetermined value. It should be understood that the above weighted summation aggregation method is merely exemplary, and other aggregation methods may be used in some implementations.

[0068] In some embodiments, the third data area 223 can also record a dynamically registerable list of extended reward sources, supporting hot access to newly added custom reward sources without modifying the main protocol structure. The third data area 223 can also record reward aggregation metadata, used to record the calculation formula, normalization factor, and aggregation completion timestamp used in this aggregation. The third data area 223 can also record real-time weight snapshots of each reward source stored in key-value pairs, thus accurately preserving the weight configuration for each reward calculation. The third data area 223 can also record a version number used to mark the current reward weight configuration, supporting weight iteration tracing and version management. In some examples, the model training system 110 can divide reward sources into multiple independent reward sources, each using a standardized SourceReward structure for storage. For example, each reward source may include a reward source name, original calculated value, weighted value, enabled status flag, plugin version number, and microsecond-level calculation time.

[0069] By storing the original reward values ​​from multiple reward sources separately, along with the aggregated reward value and weight configuration, the above method enables reward decomposition analysis and flexible weight adjustment without losing any original reward information. This reduces information loss caused by retaining only the aggregation result and facilitates the reuse of trajectory data without re-interactive sampling.

[0070] In some embodiments, the model training system 110 can also adjust the weight configuration of multiple reward sources based on their respective original reward values ​​stored in the third data region 223 to obtain updated standardized trajectory data. In this way, the aggregation method of rewards can be adjusted without re-interactive sampling, thereby reducing the overhead of resampling caused by changes in reward configuration and improving the reusability of trajectory data.

[0071] In some embodiments, the portion recorded in the fourth data region 224 may include at least one of the following: an advantage function value, a reward value, or an importance sampling ratio. The advantage function value may be a single-step action advantage function value calculated based on a generalized advantage estimation algorithm, used to characterize the superiority or inferiority of the current action relative to the average level. The reward value may be the discounted cumulative reward from the current step to the trajectory endpoint, serving as a supervision objective for value network training. The importance sampling ratio may be the ratio of the importance sampling of the new and old policies, which can be used to correct distribution biases in offline sampled data and ensure training effectiveness. In some examples, the portion recorded in the fourth data region 224 may also include a pruning threshold parameter to constrain the magnitude of policy updates. The portion recorded in the fourth data region 224 may also include a Boolean flag to indicate whether the current policy update triggers a pruning constraint.

[0072] By recording the advantage function value, reward value, and importance sampling ratio in the fourth data region 224, the intermediate quantities generated during training can be saved along with the trajectory, thereby avoiding the repeated calculation of these intermediate quantities in each training session and reducing computational costs.

[0073] In some embodiments, the fifth data area 225 can be used to record metadata for each step in a series of steps. As an example, the fifth data area 225 may include at least one of step priority, environment instance identifier, asynchronous step version number, and end-to-end time. In some examples, the fifth data area 225 may be used to record microsecond-level start and end timestamps for a single step. The fifth data area 225 may be used to record the total latency of a single-step complete interaction link. The fifth data area 225 may include a sample validity flag to filter out abnormal, incomplete, or invalid training samples. The fifth data area 225 may include a sample quality alarm list to record various data anomalies such as missing observations and reward anomalies. The fifth data area 225 may include sample priority scores required for priority experience replay, used for priority sampling and scheduling of high-value samples in the sample repository system.

[0074] In the alternative / additional area, the fifth data area 225 can also record the number of times the current step is consumed by training sampling. The fifth data area 225 can also be used to record the complete flow path of a sample, from sampling and reward calculation to training consumption, enabling full-link data traceability. Thus, by recording the metadata of the steps in the fifth data area 225, a basis can be provided for sampling, scheduling, and performance analysis, thereby facilitating the filtering and sorting of steps and improving the flexibility of data processing.

[0075] In some embodiments, the trajectory data structure 200 may further include a trajectory-level aggregation region 230, which can be used to record aggregation information determined based on multiple steps. In some embodiments, the trajectory-level aggregation region 230 can be used to record aggregation information based on multiple steps of the trajectory data sequence. As an example, the aggregation information may include at least one of total round reward 231, total round length 232, success rate 233, and aggregation metric 234. In some examples, total round reward 231 may indicate the sum of reward values ​​corresponding to all steps in the trajectory data sequence. Total round length 232 may indicate the total length of all steps in the trajectory data sequence, reflecting the complexity and efficiency of the task. Success rate 233 may indicate the success rate of the trajectory data sequence, reflecting whether the task was successfully executed. Aggregation metric 234 may indicate statistical information of each source reward, such as the average, standard deviation, maximum value, etc., of each source reward. Aggregation metric 234 can be used for statistical analysis of batch data.

[0076] In some examples, the trajectory-level aggregation region 230 can be used to record the execution time corresponding to the trajectory data sequence. The trajectory-level aggregation region 230 can also be used to record performance statistics corresponding to the trajectory data sequence, such as average environment latency, average inference latency, average step latency, total GPU time, total environment time, and so on. Thus, by setting the trajectory-level aggregation region 230 to record aggregation information determined based on multiple steps, trajectory-level statistics can be obtained without traversing all steps, thereby reducing the computational overhead of aggregation statistics.

[0077] In some embodiments, the trajectory data structure 200 may further include an extension region 240, which may employ extended metadata constructed using a key-value pair structure. This allows for rapid expansion to new environment and task types without modifying the protocol major version number. The model training system 110 may use a reverse domain name style (e.g., com.xxx.xxx.v1) to determine the names of extended fields to avoid conflicts. In some embodiments, the extension region 240 may be used to record newly added fields. As an example, the extension region 240 may include a task identifier and environment startup parameters. The extension region 240 may also include a custom key-value container, which can be used to store arbitrary key-value pairs. In some embodiments, the extension region 240 may support version compatibility, for example, allowing new fields to be added dynamically while retaining existing fields. By setting up an extension region 240 with a key-value pair structure to record newly added fields, new fields can be added dynamically while retaining existing fields, thereby improving the scalability of the data structure to new requirements and maintaining version compatibility.

[0078] In some embodiments, the trajectory data structure 200 may be processed via a streaming serialization pipeline 250. As an example, the streaming serialization pipeline 250 may include an inference engine 251, a trajectory protocol encoder 252, a reward calculation module 253, a sample trajectory management and storage module 254, and a training engine 255. (See also...) Figure 2 The model training system 110 can generate interactive actions and observation sequences via the inference engine 251. The model training system 110 inputs the interactive actions and observation sequences into the trajectory protocol encoder 252. The trajectory protocol encoder 252 can encapsulate, compress, and verify the integrity of multiple data regions.

[0079] Furthermore, the model training system 110 inputs the standardized encoded data into the reward calculation module 253. The reward calculation module 253 can perform multi-source reward calculation and aggregation based on this data. Correspondingly, the model training system 110 can backfill the calculated and aggregated reward information into the trajectory data structure and store it in the Sample & Trajectory Management System (STMS) 254. The STMS 254 can persistently store and index the standardized trajectory data. Then, the model training system 110 can perform on-demand reading, batch serialization, and sampling operations on the standardized trajectory data from the STMS 254 via the training engine 255. In some examples, the streaming serialization pipeline 250 can be compatible with multiple serialization storage formats, such as protocol encoding, columnar format, JSON format, and offline analysis format. This makes it suitable for various data usage scenarios, including sampling transmission, online training, and offline data analysis.

[0080] In some embodiments, the model training system 110 can serialize standardized trajectory data based on a columnar storage format to store the values ​​of the same field continuously across multiple steps. Accordingly, the model training system 110 can store the serialized standardized trajectory data in a predetermined buffer as training samples for model training. The predetermined buffer may include, for example, a Shared Experience Buffer Pool (SEBP), a Unified Experience Block (UEB), or any other received buffer. By storing the values ​​of the same field continuously across multiple steps in a predetermined buffer based on a columnar storage format, access locality during batch reading and sampling can be improved, thereby reducing data reading latency and storage access overhead, and improving the efficiency of training sample provisioning.

[0081] Figure 3 A schematic block diagram of an example architecture 300 for data processing according to some embodiments of the present disclosure is shown. Example architecture 300 schematically illustrates the correspondence between multiple processing stages and multiple data regions in a trajectory data structure. Figure 3 As shown, the example architecture 300 may include a sample trajectory management system (STMS) 311, a reward calculation module 312 (e.g., a distributed reward calculation framework), a training engine 313 (e.g., a distributed training engine), an environment execution engine 314, and an inference engine 315 (e.g., an asynchronous adoption inference engine). The example architecture 300 may include, for example... Figure 2 The described head region 210, step sequence 220 (including first data region 221 to fifth data region 225), trajectory-level aggregation region 230, and extension region 240 are described. In this paper, reward calculation module 312, training engine 313, environment execution engine 314, and inference engine 315 can be deployed in the machine learning model, inference model, and agent, respectively.

[0082] In some embodiments, multiple data regions can be used to record portions of the step data generated by different processing stages. For example, the environment execution engine 314 can generate portions for recording in the first data region 221. The inference engine 315 can generate portions for recording in the second data region 222. The reward calculation module 312 can generate portions for recording in the third data region 223. The training engine 313 can generate portions for recording in the fourth data region 224. The sample trajectory management system 311 can manage standardized trajectory data, including but not limited to writing to and reading from the trajectory-level aggregation region 230 and the extended region 240. It should be understood that the above processing stages can be deployed on a single computing device or in a distributed manner across multiple computing devices or servers; the scope of the solution is not limited in this respect.

[0083] In some embodiments, refer to Figure 3 During the interactive execution process, the model training system 110 sequentially fills various data regions within multiple data areas using the environment execution engine 314, inference engine 315, reward calculation module 312, and training engine 313. As an example, after a single interactive step, the environment execution engine 314, based on the environmental feedback data, uses the corresponding machine learning model's environment adapter, agent environment adapter, or inference environment adapter to fill in the data. After the policy model completes forward inference, the inference engine 315 can fill in inference actions, probabilities, performance metadata, etc. After completing the multi-source reward weighted aggregation operation, the reward calculation module 312 can backfill the aggregated reward and reward configuration snapshot. After the training engine 313 completes offline numerical solutions based on the GAE generalized advantage estimation algorithm, it can write in training derived parameters such as advantage value, discounted reward, and importance sampling ratio.

[0084] In scenarios where a single trajectory data sequence triggers natural termination or forced truncation conditions, the trajectory protocol encoder can assemble the trajectory header region, trajectory-level aggregation region, and extended region. As an example, the trajectory protocol encoder generates and encapsulates a fixed-format header while simultaneously generating trajectory-level aggregation data. Correspondingly, the trajectory protocol encoder can fill the trajectory extended region with auxiliary parameters such as task identifiers, version information, and environment configurations, thus completing the full assembly of the URTP four-layer protocol structure. Subsequently, the model training system 110 can perform columnar serialization and storage of the assembled trajectory data. This storage mechanism can continuously and centrally arrange and store fields with the same attribute throughout the entire trajectory (e.g., the original environmental rewards corresponding to all steps). This continuous arrangement of fields optimizes the cache hit rate, thereby enabling efficient batch field reading.

[0085] Furthermore, the model training system 110 inputs the assembled trajectory data (i.e., standardized trajectory data) into a predetermined buffer pool based on a communication protocol. In some examples, the communication protocol may be an open-source protocol for Remote Direct Memory Access (RDMA). Subsequently, the model training system 110 can adaptively select a compression algorithm to perform data compression based on the data content type.

[0086] In some embodiments, type-specific fields can be recorded using a predetermined structure to enable access to cross-type fields across multiple data regions via a unified interface during training based on standardized trajectory data. By recording type-specific fields with a predetermined structure, cross-type fields across multiple data regions can be accessed via a unified interface during training, thereby reducing the development and processing overhead of adapting to different types separately and improving the efficiency of cross-type data processing.

[0087] Figure 4 A flowchart of an example process 400 for data processing according to some scenarios is shown. Process 400 can be implemented at the model training system 110. See below for reference. Figure 4 To describe process 400.

[0088] like Figure 4 As shown in box 410, the model training system 110 acquires a trajectory data sequence, which corresponds to at least one of multiple types of reinforcement learning tasks. The trajectory data sequence includes step data corresponding to multiple steps in the interaction process.

[0089] In box 420, the model training system 110 encapsulates the trajectory data sequence based on the trajectory data structure to obtain standardized trajectory data. The trajectory data structure includes a unified step representation and multiple data regions associated with the unified step representation. The unified step representation includes type-specific fields corresponding to multiple types, and the multiple data regions are used to record the parts of the step data generated by different processing stages.

[0090] In box 430, the model training system 110 provides standardized trajectory data as training samples for model training.

[0091] In this way, by using a unified step representation to carry multiple types of step data, and by using multiple data areas divided according to processing stages to record the parts generated by different processing stages, multiple types of interactive data can be compatible with the same data structure.

[0092] In some embodiments, the multiple data regions may include at least one of the following: a first data region for recording the portion of the step data generated by the environment execution engine; a second data region for recording the portion of the step data generated by the policy inference engine; a third data region for recording the portion of the step data generated by reward calculation, wherein the third data region stores the original reward values ​​of each of the multiple reward sources separately; a fourth data region for recording the portion of the step data generated during training; and a fifth data region for recording metadata of the steps in the multiple steps. This allows data from different processing stages to be decoupled, facilitating independent reading, writing, and parallel processing of each processing stage, thereby reducing the processing overhead caused by data coupling.

[0093] In some embodiments, the portion recorded in the first data region may include at least one of the following: observation data corresponding to multiple types, original reward values, end-state data, or environmental state summary data, wherein the environmental state summary data is used for environmental state recovery and consistency comparison. This enables environmental state recovery and consistency comparison, thereby improving the reproducibility of trajectory data and reducing the overhead of resampling due to missing state information.

[0094] In some embodiments, the portion recorded in the second data region may include at least one of multiple types of action data, multi-granularity policy probabilities, and value estimation data. This approach provides richer available information for the training phase, thereby supporting the reuse of various training algorithms and improving the utilization rate of training samples.

[0095] In some embodiments, the multiple reward sources may include at least one of environmental reward sources, rule-based reward sources, model reward sources, and preference reward sources. The third data area may also be used to store the aggregated reward value obtained by weighted aggregation based on the original reward values ​​of the multiple reward sources, and the weight configuration used for weighted aggregation. In this way, reward decomposition analysis and flexible adjustment of weights can be supported without losing the original reward information, thereby reducing the information loss caused by only retaining the aggregation results and facilitating the reuse of trajectory data without resampling.

[0096] In some embodiments, the portion recorded in the fourth data region may include at least one of the advantage function value, reward value, or importance sampling ratio. In this way, intermediate values ​​generated during training can be saved along with the trajectory, thereby avoiding the recalculation of these intermediate values ​​during each training iteration and reducing computational overhead.

[0097] In some embodiments, the trajectory data structure may further include a header region with a predetermined length. The header region records identification information for the trajectory data sequence, including a task type identifier, a policy version identifier, and a timestamp. The header region is used for indexing and accessing standardized trajectory data. This approach enables rapid indexing and access to standardized trajectory data based on the header region, thereby reducing data retrieval latency.

[0098] In some embodiments, the trajectory data structure may include an extended region having a key-value pair structure, used to record newly added fields. This approach allows for the dynamic addition of new fields while retaining existing ones, thereby improving the scalability of the data structure to new requirements and maintaining version compatibility.

[0099] In some embodiments, the trajectory data structure may further include a trajectory-level aggregation region for recording aggregation information determined based on multiple steps. This approach enables the acquisition of trajectory-level statistical information without traversing all steps, thereby reducing the computational overhead of aggregation statistics.

[0100] In some embodiments, type-specific fields can be recorded through a predetermined structure to enable access to cross-type fields across multiple data regions via a unified interface when training based on standardized trajectory data. This reduces the development and processing overhead of adapting to different types separately, and improves the efficiency of cross-type data processing.

[0101] In some embodiments, the model training system 110 can also serialize the standardized trajectory data based on a columnar storage format to store the values ​​of the same field continuously across multiple steps; and store the serialized standardized trajectory data in a predetermined buffer as training samples for model training. This approach improves access locality during batch reading and sampling, thereby reducing data reading latency and storage access overhead, and improving the efficiency of training sample provision.

[0102] In some embodiments, step data may include at least one of the following: observation data, action data, or reward data. This approach allows for the coverage of key information during the interaction process with fewer data items, thereby reducing storage overhead while ensuring information integrity.

[0103] In some embodiments, the multiple types may include a first type, a second type, and a third type, where the first type corresponds to a trajectory interacting with a simulation environment, the second type corresponds to a trajectory interacting with a tool execution environment, and the third type corresponds to a trajectory interacting with an inference verification environment. In this way, a single data structure can be used to accommodate multiple significantly different interaction environments, thereby improving the versatility of the data structure and reducing the overhead of designing separate data formats for different environments.

[0104] A corresponding apparatus for implementing the above methods or processes is also provided.

[0105] Figure 5 Block diagrams of a data processing apparatus 500 are shown for several scenarios. Apparatus 500 can be implemented as or included in the model training system 110. The various modules / components in apparatus 500 can be implemented by hardware, software, firmware, or any combination thereof.

[0106] like Figure 5 As shown, the device 500 includes a trajectory data sequence acquisition module 510, configured to acquire trajectory data sequences corresponding to at least one type of reinforcement learning task among multiple types. The trajectory data sequence includes step data corresponding to multiple steps in the interaction process, and the step data includes observation data, action data, and reward data. A standardized trajectory data acquisition module 520 is configured to encapsulate the trajectory data sequence based on a trajectory data structure to obtain standardized trajectory data. The trajectory data structure includes a unified step representation and multiple data regions associated with the unified step representation. The unified step representation includes type-specific fields corresponding to multiple types, and the multiple data regions are used to record the parts of the step data generated by different processing stages. A standardized trajectory data providing module 530 is configured to provide standardized trajectory data as training samples for model training.

[0107] In some embodiments, the standardized trajectory data acquisition module 520 may further be configured to include at least one of the following in multiple data regions: a first data region for recording the portion of step data generated by the environment execution engine; a second data region for recording the portion of step data generated by the policy inference engine; a third data region for recording the portion of step data generated by reward calculation, wherein the third data region stores the original reward values ​​of each of the multiple reward sources respectively; a fourth data region for recording the portion of step data generated during training; and a fifth data region for recording metadata of the steps in the multiple steps.

[0108] In some embodiments, the standardized trajectory data acquisition module 520 may also be configured to record a portion of the first data region that includes at least one of the following: observation data corresponding to multiple types respectively, original reward values, end state data, or environmental state summary data, wherein the environmental state summary data is used for environmental state recovery and consistency comparison.

[0109] In some embodiments, the standardized trajectory data acquisition module 520 may also be configured to: record a portion of the second data region including at least one of multiple types of action data, multi-granularity policy probability, and value estimation data.

[0110] In some embodiments, the standardized trajectory data acquisition module 520 may also be configured to: include at least one of environmental reward sources, rule reward sources, model reward sources, and preference reward sources among multiple reward sources, and to further store the aggregated reward value obtained by weighted aggregation based on the original reward values ​​of the multiple reward sources and the weight configuration used for weighted aggregation.

[0111] In some embodiments, the standardized trajectory data acquisition module 520 may also be configured to include at least one of the following in the fourth data region: a dominant function value, a reward value, or an importance sampling ratio.

[0112] In some embodiments, the standardized trajectory data acquisition module 520 may also be configured to: make the trajectory data structure further include a header region, the header region having a predetermined length, the header region being used to record identification information of the trajectory data sequence, the identification information including task type identifier, strategy version identifier and timestamp, the header region being used to index and access the standardized trajectory data.

[0113] In some embodiments, the standardized trajectory data acquisition module 520 can also be configured to include an extended region in the trajectory data structure, the extended region having a key-value pair structure, and the extended region being used to record newly added fields.

[0114] In some embodiments, the standardized trajectory data acquisition module 520 may also be configured to include a trajectory-level aggregation region in the trajectory data structure, the trajectory-level aggregation region being used to record aggregation information determined based on multiple steps.

[0115] In some embodiments, the standardized trajectory data acquisition module 520 may also be configured to record type-specific fields through a predetermined structure, so as to enable access to cross-type fields in multiple data areas through a unified interface when training based on standardized trajectory data.

[0116] In some embodiments, the standardized trajectory data providing module 530 may also be configured to: serialize the standardized trajectory data based on a columnar storage format to store the values ​​of the same field continuously at multiple steps; and store the serialized standardized trajectory data in a predetermined buffer as training samples for model training.

[0117] In some embodiments, the trajectory data sequence acquisition module 510 may also be configured to include at least one of the following in the step data: observation data, action data, or reward data.

[0118] In some embodiments, the trajectory data sequence acquisition module 510 may also be configured to include multiple types, including a first type, a second type, and a third type, wherein the first type corresponds to a trajectory interacting with a simulation environment, the second type corresponds to a trajectory interacting with a tool execution environment, and the third type corresponds to a trajectory interacting with an inference verification environment.

[0119] The modules included in device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 500 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0120] Figure 6 A block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 6 The electronic device 600 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 6 The electronic device 600 shown can be used to achieve Figure 1 The model training system 110.

[0121] like Figure 6As shown, electronic device 600 is in the form of a general-purpose computing device. Components of electronic device 600 may include, but are not limited to, one or more processing units or processors 610, memory 620, storage devices 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processor 610 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 600.

[0122] Electronic device 600 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 620 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 630 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 600.

[0123] Electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 6 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 620 may include computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0124] The communication unit 640 enables communication with other computing devices via a communication medium. Additionally, the functionality of the components of the electronic device 600 can be implemented as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 600 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or another network node.

[0125] Input device 650 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 660 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 600 can also communicate with one or more external devices (not shown) via communication unit 640 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable data processing between the user and electronic device 600, or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interfaces (not shown).

[0126] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0127] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0128] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0129] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0130] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0131] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A data processing method, comprising: Obtain a trajectory data sequence, the trajectory data sequence corresponding to at least one of multiple types of reinforcement learning tasks, the trajectory data sequence including step data corresponding to multiple steps in the interaction process respectively; Based on the trajectory data structure, the trajectory data sequence is encapsulated to obtain standardized trajectory data. The trajectory data structure includes a unified step representation and multiple data regions associated with the unified step representation. The unified step representation includes type-specific fields corresponding to the multiple types respectively. The multiple data regions are used to record the parts of the step data generated by different processing stages. as well as The standardized trajectory data is provided as training samples for model training.

2. The method of claim 1, wherein the plurality of data areas comprises at least one of the following: A first data area is used to record the portion of the step data generated by the environment execution engine. The second data area is used to record the portion of the step data generated by the strategy inference engine. The third data area is used to record the portion of the step data generated by the reward calculation, and the third data area is used to store the original reward values ​​of each of the multiple reward sources respectively. The fourth data region is used to record the portion of the step data generated during the training process; as well as The fifth data area is used to record metadata of the steps in the plurality of steps.

3. The method of claim 2, wherein the portion recorded in the first data area includes at least one of the following: The observation data corresponding to each of the multiple types. Original reward value, End status data, or Alternatively, environmental state summary data, which is used for environmental state recovery and consistency comparison.

4. The method according to claim 2, wherein the portion recorded in the second data region includes at least one of the action data, multi-granularity policy probability, and value estimation data corresponding to the plurality of types respectively.

5. The method of claim 2, wherein the plurality of reward sources includes at least one of environmental reward sources, rule reward sources, model reward sources, and preference reward sources. The third data area is further used to store the aggregated reward value obtained by weighted aggregation of the original reward values ​​of the multiple reward sources and the weight configuration used for the weighted aggregation.

6. The method of claim 2, wherein the portion of the data recorded in the fourth data region includes at least one of the advantage function value, the return value, or the importance sampling ratio.

7. The method according to claim 1, wherein the trajectory data structure further includes a header region, the header region having a predetermined length, the header region being used to record identification information of the trajectory data sequence. in, The identification information includes a task type identifier, a strategy version identifier, and a timestamp. The header area is used to index and access the standardized trajectory data.

8. The method according to claim 1, wherein the trajectory data structure includes an extended region having a key-value pair structure, wherein the extended region is used to record newly added fields.

9. The method according to claim 1, wherein the trajectory data structure further includes a trajectory-level aggregation region for recording aggregation information determined based on the plurality of steps.

10. The method of claim 1, wherein the type-specific fields are recorded through a predetermined structure to enable access to cross-type fields in the plurality of data regions through a unified interface when training based on the standardized trajectory data.

11. The method of claim 1, further comprising: Based on the columnar storage format, the standardized trajectory data is serialized so that the values ​​of the same field in the multiple steps are stored continuously. as well as The serialized standardized trajectory data is stored in a predetermined buffer to serve as training samples for model training.

12. The method of claim 1, wherein the step data includes at least one of the following: observation data, action data, or reward data.

13. The method of claim 1, wherein the plurality of types includes a first type, a second type and a third type, the first type corresponding to a trajectory interacting with a simulation environment, the second type corresponding to a trajectory interacting with a tool execution environment, and the third type corresponding to a trajectory interacting with an inference verification environment.

14. An apparatus for request processing, comprising: The trajectory data sequence acquisition module is configured to acquire trajectory data sequences, which correspond to at least one type of reinforcement learning task. The trajectory data sequence includes step data corresponding to multiple steps in the interaction process, and the step data includes observation data, action data, and reward data. The standardized trajectory data acquisition module is configured to encapsulate the trajectory data sequence based on the trajectory data structure to obtain standardized trajectory data. The trajectory data structure includes a unified step representation and multiple data regions associated with the unified step representation. The unified step representation includes type-specific fields corresponding to the multiple types respectively. The multiple data regions are used to record the parts of the step data generated by different processing stages. as well as A standardized trajectory data providing module is configured to provide the standardized trajectory data as training samples for model training.

15. An electronic device comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 13 when executed by the at least one processor.

16. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 13.

17. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 13.

Images