Reinforcement training method, device and equipment of desktop operation engine, medium and product

Abstract

The present disclosure relates to a reinforcement training method, device, equipment, medium and product of a desktop operation engine, which is used in a reinforcement training system of the desktop operation engine. The system comprises a control terminal and a plurality of device nodes for creating virtual machine instances. At least one environment service endpoint is run in each device node, and each environment service endpoint is used to operate one virtual machine instance on the corresponding device node. The method comprises: determining, by the control terminal, a target environment service endpoint from the plurality of environment service endpoints in response to a training session request; collecting, by the control terminal, observation data about a target virtual machine instance performing a specified system operation under a specified initial configuration via the target environment service endpoint, wherein the operation content of the specified system operation is determined based on operation information output by the desktop operation engine; and feeding back, by the control terminal, the observation data to the desktop operation engine for reinforcement learning training of the desktop operation engine.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and more particularly to a method, apparatus, device, medium, and product for reinforcement training of a desktop operating engine. Background Technology

[0002] Reinforcement learning training for GUI (Graphical User Interface) agents based on multimodal models typically employs a cyclical approach of observation-decision-execution-re-observation: the model first observes information such as the screen display, interface structure tree, and terminal output, then makes a decision and generates operation instructions. After the operation is executed, changes occur in window state, file content, network connection, process execution, etc., and the model continues to interact based on the new state until the task is completed or fails.

[0003] Compared to traditional game or simulation environments, desktop operating system (such as Windows and Linux) environments have the following characteristics for GUI agents: extremely large state space, diverse forms of observation data (including images, text, and structured data), strong side effects of operations, complex pre-task preparation, and high environment reset costs. In reinforcement learning training of desktop GUI agents, a large number of interaction samples are needed to optimize model policies, thus placing higher demands on the environment's processing power, stability, and reproducibility.

[0004] In current practice, the common approach is to use a single virtual machine or a small number of remote desktop instances to execute tasks, driving mouse and keyboard operations through scripts. However, as the training scale increases, this approach encounters significant bottlenecks in areas such as instance creation and recycling, session scheduling, health checks, and anomaly isolation, making it difficult to form a controllable and scalable training infrastructure. Summary of the Invention

[0005] This disclosure provides a method, apparatus, device, medium, and product for enhancing the training of a desktop operating engine, at least addressing the problem of how related technologies can provide a controllable and scalable training infrastructure.

[0006] According to a first aspect of the present disclosure, a method for reinforcement training of a desktop operating engine is provided, and a system for reinforcement training of a desktop operating engine is provided. The system includes a control terminal and multiple device nodes for creating virtual machine instances. Each device node runs at least one environment service endpoint, and each environment service endpoint is used to operate a virtual machine instance on the corresponding device node. The method includes: determining a target environment service endpoint from the multiple environment service endpoints through the control terminal in response to a training session request; collecting observation data about a target virtual machine instance performing a specified system operation under a specified initial configuration via the target environment service endpoint through the control terminal, wherein the target virtual machine instance is a virtual machine instance operated by the target environment service endpoint, the operation content of the specified system operation is determined based on operation information output by the desktop operating engine, and the observation data is used to describe the changes that occur in the target virtual machine instance due to the execution of the specified system operation; and feeding back the observation data to the desktop operating engine through the control terminal for reinforcement learning training of the desktop operating engine.

[0007] Optionally, determining the target environment service endpoint from multiple environment service endpoints in response to a training session request via the control terminal includes: determining the target environment service endpoint from the multiple environment service endpoints based on endpoint information of the multiple environment service endpoints in response to the training session request via the control terminal, wherein the endpoint information includes endpoint status, and the endpoint status includes at least one of the following: allocated, available, and unavailable, wherein the available and unavailable statuses are obtained by detecting the running status of the corresponding environment service endpoint and its corresponding virtual machine instance.

[0008] Optionally, the step of collecting observation data about the target virtual machine instance performing a specified system operation under a specified initial configuration via the control terminal and the target environment server endpoint includes: calling the initial configuration interface, system operation interface, and observation data collection interface of the target environment server endpoint via the control terminal, wherein the initial configuration interface, the system operation interface, and the observation data collection interface adopt a mutual exclusion mechanism; and performing the specified initial configuration and the specified system operation on the target virtual machine instance based on the initial configuration interface, the system operation interface, and the observation data collection interface, and collecting the observation data.

[0009] Optionally, the configuration content of the specified initial configuration includes environment reset and task pre-configuration; and / or, the configuration content of the specified initial configuration includes cache service address injection.

[0010] Optionally, the training session request includes a batch training session request, wherein determining the target environment service endpoint from multiple environment service endpoints in response to the training session request includes: determining the number of sessions of the batch training session request in response to the batch training session request, and determining multiple target environment service endpoints corresponding to the number of sessions from the multiple environment service endpoints.

[0011] Optionally, the operation content of the specified system operation includes an executable operation sequence, wherein the step of feeding back the observation data to the desktop operation engine through the control terminal includes: recording each executable operation in the executable operation sequence and its corresponding observation data in sequence through the control terminal, and generating an operation trajectory file; and feeding back the operation trajectory file to the desktop operation engine.

[0012] Optionally, the step of sequentially recording each executable operation in the executable operation sequence and its corresponding observation data, and generating an operation trajectory file, includes: creating an initial operation trajectory file; continuously writing the currently executed executable operation and the currently collected observation data into the created operation trajectory file during the execution of the executable operation sequence by the target virtual machine instance; and continuing to write termination information into the created operation trajectory file after the execution of the executable operation sequence is completed, thereby obtaining the operation trajectory file, wherein the termination information includes at least one of the following: end time, task success or failure.

[0013] According to a second aspect of the present disclosure, a reinforcement training apparatus for a desktop operating engine is provided, for a reinforcement training system of a desktop operating engine, the system including a control terminal and a plurality of device nodes for creating virtual machine instances, each device node running at least one environment service endpoint, each environment service endpoint for operating a virtual machine instance on a corresponding device node, wherein the apparatus includes: a determining unit configured to determine a target environment service endpoint from the plurality of environment service endpoints in response to a training session request via the control terminal; an execution unit configured to collect observation data via the control terminal, through the target environment service endpoint, regarding the target virtual machine instance performing a specified system operation under a specified initial configuration, wherein the target virtual machine instance is a virtual machine instance operated by the target environment service endpoint, the operation content of the specified system operation is determined based on operation information output by the desktop operating engine, and the observation data is used to describe the changes that occur in the target virtual machine instance due to the execution of the specified system operation; and a feedback unit configured to feed back the observation data to the desktop operating engine via the control terminal for reinforcement learning training of the desktop operating engine.

[0014] Optionally, the determining unit is further configured to, in response to the training session request, determine a target environment service endpoint from the plurality of environment service endpoints based on the endpoint information of the plurality of environment service endpoints via the control terminal, wherein the endpoint information includes endpoint status, and the endpoint status includes at least one of the following: allocated, available, unavailable, and the available and unavailable statuses are obtained by detecting the running status of the corresponding environment service endpoint and its corresponding virtual machine instance.

[0015] Optionally, the execution unit is further configured to call the initial configuration interface, system operation interface, and observation data collection interface of the target environment service endpoint through the control terminal, wherein the initial configuration interface, the system operation interface, and the observation data collection interface adopt a mutual exclusion mechanism; based on the initial configuration interface, the system operation interface, and the observation data collection interface, the specified initial configuration and the specified system operation are executed on the target virtual machine instance, and the observation data is collected.

[0016] Optionally, the configuration content of the specified initial configuration includes environment reset and task pre-configuration; and / or, the configuration content of the specified initial configuration includes cache service address injection.

[0017] Optionally, the training session request includes a batch training session request, and the determining unit is further configured to, in response to the batch training session request, determine the number of sessions in the batch training session request through the control terminal, and determine a plurality of target environment service endpoints corresponding to the number of sessions from the plurality of environment service endpoints.

[0018] Optionally, the operation content of the specified system operation includes an executable operation sequence, and the feedback unit is further configured to: record each executable operation in the executable operation sequence and its corresponding observation data in sequence through the control terminal, and generate an operation trajectory file; and feed back the operation trajectory file to the desktop operation engine.

[0019] Optionally, the feedback unit is further configured to: create an initial operation trajectory file through the control terminal; continuously write the currently executed executable operation and the currently collected observation data into the created operation trajectory file during the execution of the executable operation sequence by the target virtual machine instance; and continue to write termination information into the created operation trajectory file after the execution of the executable operation sequence is completed, thereby obtaining the operation trajectory file, wherein the termination information includes at least one of the following: end time, task success or failure.

[0020] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform an enhanced training method for a desktop operating engine according to exemplary embodiments of the present disclosure.

[0021] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein instructions in the computer-readable storage medium, when executed by at least one processor, cause at least one processor to perform a method for enhancing the training of a desktop operating engine according to an exemplary embodiment of the present disclosure.

[0022] According to a fifth aspect of the present disclosure, a computer program product is provided, including computer instructions that, when executed by at least one processor, cause at least one processor to perform an enhanced training method for a desktop operating engine according to an exemplary embodiment of the present disclosure.

[0023] The technical solutions provided by the embodiments of this disclosure offer at least the following beneficial effects: According to the desktop operating engine reinforcement training method, apparatus, device, medium, and product of this disclosure, by creating multiple virtual machine instances in a distributed cluster composed of multiple device nodes, and creating corresponding environment service endpoints for each, unified management and operation of each virtual machine instance can be achieved using the environment service endpoints, including initial configuration, system operation, and observation data collection, thus enabling convenient and stable acquisition of interactive samples. Furthermore, when it is necessary to add virtual machine instances, only more environment service endpoints need to be created accordingly. Therefore, this disclosure can form a controllable and scalable training infrastructure, meeting the needs of large-scale parallel sampling and reinforcement learning training of desktop operating engines.

[0024] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0026] Figure 1 This is a schematic diagram of the overall architecture of a desktop operating engine reinforcement training system according to an exemplary embodiment of the present disclosure.

[0027] Figure 2 This is a flowchart of a method for enhancing the training of a desktop operating engine according to an exemplary embodiment of the present disclosure.

[0028] Figure 3This is a schematic diagram of the multi-node program distribution, capacity detection, and instance allocation process according to exemplary embodiments of the present disclosure.

[0029] Figure 4 This is a schematic diagram of the endpoint state of an environmental service endpoint according to an exemplary embodiment of the present disclosure.

[0030] Figure 5 This is a schematic diagram of a cache download process according to an exemplary embodiment of the present disclosure.

[0031] Figure 6 This is a schematic diagram of the session interaction execution and trajectory acquisition process according to an exemplary embodiment of the present disclosure.

[0032] Figure 7 This is a block diagram of an enhanced training apparatus for a desktop operating engine according to an exemplary embodiment of the present disclosure.

[0033] Figure 8 This is a block diagram of an electronic device according to exemplary embodiments of the present disclosure. Detailed Implementation

[0034] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0035] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0036] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. Another example is "performing at least one of step one and step two", which means the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0037] Existing reinforcement learning training techniques for GUI agents mainly fall into two categories in terms of desktop environment execution and training platforms.

[0038] One approach uses remote desktop control technologies, such as VNC (Virtual Network Computing) and RDP (Remote Desktop Protocol), to acquire the desktop screen and perform operations by issuing mouse and keyboard events through robotic process automation (BFA) or automated scripts. This method is relatively fast to implement, but it often can only execute specific scripts and is typically designed for single-machine execution or low concurrency, making it difficult to directly support the large-scale parallel sampling required for reinforcement learning training.

[0039] Another type encapsulates the desktop environment as a reinforcement learning environment (providing interfaces for initialization, action execution, etc.), but it is also usually run on a single machine or a fixed small number of machines. Scaling up and reproducing the environment requires manual alignment of the environment scripts, virtual machine images, and dependency versions.

[0040] To address the shortcomings of related technologies, this disclosure provides a method, apparatus, device, medium, and product for reinforcement training of a desktop operating engine. By creating multiple virtual machine instances in a distributed cluster composed of multiple device nodes and establishing corresponding environment service endpoints for each instance, unified management and operation of each virtual machine instance can be achieved using these environment service endpoints. This includes initial configuration, system operation, and observation data collection, enabling convenient and stable acquisition of interactive samples. Furthermore, when additional virtual machine instances are needed, only more environment service endpoints need to be created accordingly. Therefore, this disclosure provides a controllable and scalable training infrastructure, meeting the needs of large-scale parallel sampling and reinforcement learning training for desktop operating engines.

[0041] The following will describe in detail, with reference to the accompanying drawings, a method, apparatus, device, medium, and product for enhancing the training of a desktop operating engine according to exemplary embodiments of the present disclosure. Desktop operating engines include, but are not limited to, GUI agents.

[0042] The desktop operating engine reinforcement training method (hereinafter referred to as the training method) according to the exemplary embodiments of the present disclosure is used in a desktop operating engine reinforcement training system (hereinafter referred to as the training system). Figure 1 This is a schematic diagram of the overall architecture of the training system. As a training infrastructure, this system is used to actually execute the decisions made by the desktop operating engine and collect post-execution observation data for the desktop operating engine to make further decisions.

[0043] like Figure 1 As shown, the training system includes a control terminal and multiple device nodes for creating virtual machine instances, i.e. Figure 1The system consists of device node 1, device node 2, ..., device node M. Each device node has a desktop operating system installed, serving as the environment for users to manage their personal computers. It manages system hardware and software resources and supports basic functions such as task scheduling, printing, I / O, peripheral device control, and memory allocation. Common desktop operating systems include Windows, macOS, and various Linux distributions. The desktop operating system contains a desktop environment, which is a layer of the desktop operating system, running on top of the operating system and providing a graphical interface for human-computer interaction. Each device node runs at least one environment server endpoint, for example... Figure 1 The system consists of environment server endpoints 1 and 2 running on device node 1, environment server endpoints 3 and 4 running on device node 2, and environment server endpoint N running on device node M. Each environment server endpoint can run a virtual machine instance (used to run a desktop operating system, which can be rolled back to the initial state via snapshots). Each environment server endpoint is used to operate a virtual machine instance on the corresponding device node and can provide interfaces for environment reset, action execution, and status observation. The training system forms a device cluster.

[0044] The control terminal implements its functions through a control terminal service, which runs on the cluster entry node (which can be one of multiple device nodes used to create virtual machine instances, or...). Figure 1 The image shows a service process for a separate device node (in addition to the multiple device nodes). This process can remotely call each device node to create virtual machine instances. This can be achieved, for example, through, but not limited to, REST APIs (Representational State Transfer Application Programming Interface, a web interface based on the HTTP protocol). REST APIs include, but are not limited to, / init (used to create training sessions), / session / (used for session management), / shutdown (used to close a session), / cache / (Used for managing session cache).

[0045] The control terminal may include, but is not limited to, such as Figure 1The following modules are shown: Configuration Management, Node Management and Orchestration, Resource Pool & Session Management, Cache Download, Action Parsing & Reinforcement Learning Framework Adaptation, and Observation Data Collection. These modules are responsible for managing the configuration operations of training tasks, managing device nodes, managing resource pools and training sessions, downloading files via cache service addresses, processing actions to be executed as determined by the GUI agent, and collecting observation data during the execution of actions determined by the GUI agent. The resource pool is a set of environment service endpoints maintained by the control terminal. Other functions will be described in detail later and will not be elaborated on here.

[0046] The training system adopts a layered architecture of control terminal—node management—environment services—desktop environment encapsulation—action parsing and adaptation—observation data collection. It forms a scalable multi-node training infrastructure through remote invocation, thus providing a distributed virtual machine pooled reinforcement learning training platform. The control terminal connects to the upper-layer training system, which directly trains the desktop operation engine. The upper-layer training system sends the operation information output by the desktop operation engine (i.e., the desktop operation engine's decisions) to the control terminal via a session. The session is the interaction context between the upper-layer training system and a specific endpoint, distinguished by a session identifier, and includes information such as task configuration, trajectory recording, and time allocation. Based on the received operation information, the control terminal centrally manages the operation of virtual machine instances on each device node and feeds back the observation data generated by each virtual machine instance to the upper-layer training system.

[0047] As an example, the training system includes the following core components.

[0048] 1. Control Server Service: Responsible for reading the configuration information of training tasks and providing interfaces for resource initialization, session allocation, batch session allocation, interactive execution, session recycling, and cluster shutdown. Internally, it maintains the queue of tasks to be allocated and the session record table, and drives the update of the resource pool status based on health check results.

[0049] 2. Node Management and Orchestration Module: Responsible for program distribution, capacity detection, and environment service startup across multiple nodes. It parses environment service endpoint information from deployment results and writes the endpoints to the resource pool file. It supports batch shutdown and destruction of instances in specified states.

[0050] 3. Resource Pool and Session Management Module: This module maintains service endpoints and status using structured data files, recording fields such as node address information, port information, virtual machine identifier, network address, environment service endpoint, status identifier, allocation time, and session identifier, to achieve session allocation and reclamation similar to a leasing mechanism.

[0051] 4. Environment Service: Each virtual machine instance corresponds to an environment service process (i.e., an environment service endpoint), providing interfaces for health checks, information queries, environment resets, action execution, status observation, and service shutdown. The service shutdown interface can be used to initiate shutdown requests for endpoints in a specified state and, when necessary, destroy the virtual machine instance through the virtual machine manager, simultaneously removing the corresponding endpoint from the resource pool to prevent zombie resource occupation. Internally, the environment service uses a mutual exclusion mechanism to ensure the serialization of operations on the same instance, ensuring concurrency safety.

[0052] 5. Desktop Environment Encapsulation Module: This module encapsulates desktop environment operations within the environment service. It is used to start or connect to virtual machines, roll back initialization snapshots, execute desktop actions, and obtain observation data such as screenshots, interface structure tree, and terminal output.

[0053] As an example, logically, this training system can be divided into a control terminal and several device node modules. The control terminal provides a unified remote call interface, while the node side provides an environment service interface; the control terminal and the node side interact via network remote calls. As an example, token authentication can be enabled between the control terminal and the environment service endpoints. When authentication is enabled, all interface requests must carry authentication information, which can reduce the risk of abuse in distributed deployments.

[0054] The node side may include, but is not limited to, the following components: virtual machine manager (for creating and managing virtual machines), desktop environment encapsulation module, environment service, and desktop operation and screenshot service inside the virtual machine (for providing screenshots and auxiliary information).

[0055] The interfaces provided by the control terminal may include, but are not limited to: resource initialization interface, single session start interface, batch session start interface, interactive execution interface, session end interface, and cluster shutdown interface.

[0056] The interfaces provided by the environment service may include, but are not limited to: health check interface, information query interface, environment reset interface, action execution interface, status observation interface, and service shutdown interface.

[0057] Figure 2 This is a flowchart of a method for enhancing the training of a desktop operating engine according to an exemplary embodiment of the present disclosure.

[0058] Reference Figure 2 In step 201, the target environment service endpoint is determined from multiple environment service endpoints in response to the training session request via the control terminal.

[0059] It should be understood that the target environment service endpoint is the environment service endpoint used to execute the task corresponding to the currently received training session request, and can specifically operate the corresponding target virtual machine instance to execute the task.

[0060] As an example, when the training session request is a single training session request (used to apply for a single training session), the number of target environment service endpoints is one; when the training session request is a batch training session request (used to apply for multiple training sessions in batches), the number of target environment service endpoints is multiple corresponding to the number of training sessions applied for in the batch training session request. For the latter, step 201 may include: through the control terminal, in response to the batch training session request, determining the number of sessions in the batch training session request, and determining multiple target environment service endpoints corresponding to the number of sessions from multiple environment service endpoints, that is, one environment service endpoint corresponds to one training session. The latter can support batch session allocation, that is, applying for multiple training sessions at once in the batch startup interface, which can realize batch startup of environment services, thereby running multiple reinforcement learning interaction samples of the desktop operation engine in parallel, improving reinforcement learning training efficiency. By providing desktop instances in parallel through a multi-node resource pool, the upper-layer training system can obtain sessions on demand and generate trajectory data in parallel, significantly shortening the training cycle and improving throughput and scalability.

[0061] For batch training session requests, as an example, this step can be completed by the node management orchestration module on the control end through automated deployment tools to complete code distribution, capacity detection and instance allocation, thereby realizing automated deployment and capacity allocation of multiple nodes.

[0062] Specifically, such as Figure 3 As shown, taking the case where no virtual machine instances or environment service endpoints have been created yet, this step first loads the configuration information of the training task, then distributes the environment service code to each device node, and performs capacity probing on these device nodes to determine the number of virtual machines that can be installed on each device node. Subsequently, based on the probing results, at least one device node is selected to execute the training task, and a reasonable number of virtual machine instances is allocated to each selected device node. Once the number is allocated, virtual machine instances can be created and started in batches on the selected device nodes according to the corresponding number; these virtual machine instances are the target virtual machine instances. After the virtual machine instances have started, an environment service process can be started for each virtual machine instance, i.e., an environment service endpoint is created as the target environment service endpoint, and the endpoint information of the environment service endpoint corresponding to each started virtual machine instance is collected and written to the resource pool to achieve resource pooling scheduling and session management.

[0063] Furthermore, during the capacity probing phase, the computing resources of each device node can be calculated based on virtual machine template resource parameters (such as the number of processor cores, memory size, and disk size) to obtain the maximum number of instances that each device node can support. The capacity probing results can be obtained by parsing from the automated deployment artifacts.

[0064] The instance allocation phase supports two modes, for example: 1. Specified Node Quota Mode: In this mode, the configuration directly specifies the number of requests allocated to each node. A quota verification mode is also configured, where the control panel or node management orchestration module verifies whether the number of requests is within the capacity range of the allocated nodes. If so, allocation continues; otherwise, an error is reported or the allocation is truncated. 2. Total Quantity Proportional Allocation Mode: In this mode, the configuration only specifies the target total number of training sessions. The node management orchestration module allocates quotas proportionally based on the capacity of each node, and pads any remaining quotas with their decimal parts to ensure the total allocation meets the target.

[0065] As an example, the control panel can also be configured with a timeout recycling mechanism. If the number of sessions requested by the training session cannot be met within the timeout period, the allocated endpoints will be released and a failure will be returned to avoid resource leakage caused by partial success.

[0066] The previous example used the case where no virtual machine instance or environment service endpoint has been created. It should be understood that if a virtual machine instance and corresponding environment service endpoint have been created, the existing environment service endpoint can be used as the target environment service endpoint, and the virtual machine instance can be used as the target virtual machine instance and started.

[0067] Accordingly, step 201 includes: via the control terminal, in response to a training session request, determining a target environment service endpoint from multiple environment service endpoints based on endpoint information of multiple environment service endpoints. In other words, each time a target environment service endpoint is determined, a subset of endpoints can be selected as the target environment service endpoint based on the endpoint information of existing environment service endpoints in the resource pool, and corresponding virtual machine instance resources can be used. This eliminates the need to use all existing environment service endpoints and their corresponding virtual machine instance resources, achieving automatic scaling down in conjunction with training load. Furthermore, if the existing environment service endpoints and virtual machine instances are insufficient to meet session requirements, new virtual machine instances can be automatically created and new environment service endpoints can be started as the target environment service endpoints, using the computing resources of each device node to execute the training operations corresponding to the training session request, achieving automatic scaling up in conjunction with training load. This embodiment achieves automated deployment without relying on manual operation and helps improve the utilization of computing resources.

[0068] As an example, the resource pool is maintained using structured data files that record endpoint information for each environment service. Endpoint information includes, but is not limited to, node address, port, virtual machine name (or virtual machine identifier), environment identifier, network address, service endpoint, remote desktop port, allocation time, allocator, session identifier, etc.

[0069] In addition to the endpoint information listed above, the endpoint information may optionally include endpoint status, which includes at least one of the following: allocated, available, or unavailable. "Assigned" indicates that the corresponding environment service endpoint has been allocated and is executing a training task. The available and unavailable statuses are obtained by detecting the running status of the corresponding environment service endpoint and its corresponding virtual machine instance. By performing health checks on each environment service endpoint and marking the available status of the endpoints, the use of unavailable endpoints, such as those experiencing desktop instance crashes, network interruptions, or service incompleteness, can be avoided. This achieves health detection and failure isolation, reduces the risk of faulty instances continuously occupying resources and polluting training data, and ensures the smooth progress of training. As an example, when action execution or status observation fails, the control terminal can migrate the current endpoint to unavailable and terminate the session and trajectory recording to prevent the continuous propagation of errors. In batch allocation scenarios, if the allocation is not satisfied, the allocated endpoints can be rolled back and released.

[0070] Figure 4 The diagram illustrates the transition relationships between different endpoint states. For an endpoint in an available state, if the endpoint is selected when the control end allocates it from the resource pool, its state can be updated from available to allocated, and the allocation time and session identifier can be written. If the endpoint's health probe fails, the endpoint can be migrated to unavailable, and the reason and migration time can be recorded for isolating faulty instances. For an endpoint in an allocated state, when the session ends or times out, the endpoint's state is reset to available, and the session identifier is cleared. When the endpoint experiences an execution exception, it is migrated to unavailable, and the reason and migration time can be recorded for isolating faulty instances. As an example, for unavailable endpoints, a separate unavailable pool file can be maintained, or they can be distinguished by a status field within the same resource pool file, to facilitate the investigation, batch shutdown, or destruction of unavailable endpoints. As an example, if an unavailable endpoint recovers after investigation, it can be updated to an available endpoint.

[0071] It should be understood that when determining the target environment service endpoint from multiple existing environment service endpoints based on endpoint information, it can specifically select from available endpoints. As an example, after receiving a training session request, the control terminal adds the request to the allocation queue. The allocation thread then loops through the resource pool to select candidate endpoints and initiates health probes on the candidate endpoints. If the health probe returns a normal status and the internal service of the instance is available, the control terminal generates a session identifier for the endpoint, sets the endpoint status to allocated, and returns endpoint information to the upper-layer training system. If the probe fails or the status is abnormal, the control terminal migrates the endpoint to unavailable and continues to attempt to allocate other endpoints. Furthermore, to avoid long-term resource occupation, the control terminal can periodically scan the allocation time of allocated endpoints. If the allocation time exceeds a threshold and the endpoint is not released, a forced reclamation is triggered, setting the endpoint status back to available, calling the environment service's reset interface to clean up the environment, clearing session records, and ending the observation data collection.

[0072] Return to reference Figure 2 In step 202, through the control terminal and via the target environment server endpoint, observation data is collected regarding the target virtual machine instance performing the specified system operation under the specified initial configuration.

[0073] The target virtual machine instance is the virtual machine instance that is operated on by the target environment server endpoint.

[0074] The operation content of a specified system operation is determined based on the operation information output by the desktop operation engine. As an example, the operation content of a specified system operation can be a sequence of executable operations parsed by the control terminal according to specified parsing rules, that is, including at least one executable operation arranged in execution order. Executable operations can include structured actions (such as click, drag, scroll, input, etc., where, to improve action robustness, special characters in input actions can be escaped or "clipboard copy + shortcut key paste" can be used instead of character-by-character input; for example, control actions such as wait, complete, and fail), and can also include code actions (such as automated scripts). This disclosure does not impose any limitations on this. Since the desktop operation engine is usually a multimodal model, its output operation information is usually natural language or semi-structured actions. By performing stable operation parsing and execution according to specified parsing rules, the frequency of training interruptions can be reduced, data quality improved, and training reliability enhanced.

[0075] Observational data describes the changes that occur to a target virtual machine instance due to the execution of specified system operations. As an example, observational data is environment state information returned by the environment service, which may include screenshots and can be expanded to include auxiliary information such as mouse position, interface structure tree, terminal output, and task description. The adapter on the control side can uniformly package the observational data, such as adding completion and failure status fields, to adapt to the conventions of different reinforcement learning frameworks for environment interfaces; it can also send necessary historical information back to the desktop operating engine's parser for memory updates.

[0076] Optionally, step 202 includes: calling the initial configuration interface, system operation interface, and observation data collection interface of the target environment server endpoint through the control terminal, wherein the initial configuration interface, system operation interface, and observation data collection interface adopt a mutual exclusion mechanism; and performing specified initial configuration and specified system operations on the target virtual machine instance based on the initial configuration interface, system operation interface, and observation data collection interface, and collecting observation data. By providing clear standardized environment interfaces for implementing operations such as environment initialization, action execution, and state observation, it is possible to align with reinforcement learning frameworks, thereby enabling unified sampling, reward calculation, and trajectory recording during the training phase. By adopting a mutual exclusion mechanism among these interfaces, it is possible to ensure that operations on the same instance are serialized, avoiding uncertain states caused by concurrent execution of the same instance.

[0077] Optionally, the initial configuration may include environment reset and task pre-configuration. After successful session allocation, the control terminal can call the environment service's reset interface and distribute task configurations. Task configurations may include task description, task name, source file address, network proxy configuration, login information, and initialization commands. During the reset process, the environment service first rolls back the virtual machine instance to its initial snapshot (e.g., including but not limited to calling the POST / reset interface) to obtain a consistent base state. Then, it executes the pre-configuration according to the task configuration and finally returns the initial observation data. By providing environment reset and task pre-configuration verification based on snapshot rollback, reset costs can be reduced, thereby reducing the risk of environment state drift and helping to ensure the consistency of task execution across different instances, enabling the reproduction of the same task and reducing training noise.

[0078] Optionally, the initial configuration may include cache service address injection. Cache service address injection refers to injecting the cache service address into the environment service during the reset request. The environment service then writes this address into the environment variables, causing the desktop environment to prioritize obtaining files through the cache interface provided by the control panel when downloading files. This reduces bandwidth waste and inconsistency risks caused by concurrent downloads from multiple instances.

[0079] For embodiments where the specified initial configuration includes all three items mentioned above, regarding pre-task configuration, desktop tasks typically involve preliminary preparations such as downloading data files, uploading materials, configuring network proxies, logging into accounts, installing software dependencies, and launching applications. When multiple instances run concurrently, providing a unified configuration system and caching mechanism can avoid duplicate downloads, reduce initialization costs, and minimize interference introduced by concurrent writes and environmental differences, thus helping to ensure training data quality and model convergence performance.

[0080] As an example, Figure 5 The diagram illustrates the cache download process. The control terminal's cache interface, in response to a received cache download request, generates a deterministic cache filename for the same network address, and then checks if a file with that cache filename exists among the cached files. Figure 5The diagram illustrates filenames in cached files, such as the data archive file abc123…data.zip, and the model parameter file def456…model.pth. If a cache hit occurs, the cached file can be read directly, and its contents returned. If a cache miss occurs, it indicates the file was not cached beforehand and can be downloaded. Specifically, it's downloaded in a streaming manner to a temporary file, and after downloading, it's atomically renamed to replace the original cached file. The reason for using a temporary file + atomic renaming approach for cache misses is because different processes (e.g., ...) Figure 5 Processes A and B, being different processes with different process identifiers (PIDs), might simultaneously request to download the same file. If both are included in the cached file, there is a risk of concurrency. However, by first downloading to a temporary file and then performing mutually exclusive atomic renaming, only the first file that successfully completes the download and renaming, along with its filename, is retained (e.g., the first file downloaded and renamed). Figure 5 The final filename retained is xyz789.zip, and subsequent downloaded files will not be retained, thus achieving concurrency safety and cache consistency.

[0081] Return to reference Figure 2 In step 203, the observation data is fed back to the desktop operating engine through the control terminal so that the desktop operating engine can perform reinforcement learning training.

[0082] Optionally, the specified system operation content includes the aforementioned sequence of executable operations. Step 203 includes: recording each executable operation and its corresponding observation data in the sequence of executable operations sequentially through the control terminal, and generating an operation trajectory file; and feeding back the operation trajectory file to the desktop operation engine. By generating a closed-loop operation trajectory file according to the operation execution order, the upper-level training system can replay and perform quality screening of the trajectory, and further use it for supervising data construction, failure analysis, and retraining, thereby improving the success rate and generalization ability of the desktop operation engine in desktop tasks.

[0083] Optionally, each executable operation in the executable operation sequence and its corresponding observation data are recorded sequentially, and an operation trajectory file is generated, including: creating an initial operation trajectory file; continuously writing the currently executed executable operation and the currently collected observation data to the created operation trajectory file during the execution of the executable operation sequence in the target virtual machine instance; and continuing to write termination information to the created operation trajectory file after the executable operation sequence is completed, thereby obtaining the operation trajectory file. The termination information includes at least one of the following: end time and task success or failure. By writing the first executable operation executed in the order of occurrence during the execution of the operation, then writing the various observation data collected for the first operation (which can be written item by item according to the collection order, or the various observation data for the same executable operation can be merged and written together as a single data item, which is not limited in this disclosure), then writing the second executable operation executed, and then writing the various observation data collected for the second operation, and so on, until all executable operations in the executable operation sequence are executed, and finally including the end time and / or the termination information of task success or failure, the entire operation trajectory can be completely recorded according to the timeline and the end status.

[0084] Accordingly, such as Figure 1 The observation data collection module shown can specifically be a trajectory management module. As an example, Figure 6 This diagram illustrates the execution of session interactions and trajectory acquisition, showcasing the complete session lifecycle and the trajectories recorded during the interaction. Specifically, at the start of the session, the control terminal can first reset the target virtual machine instance's environment (i.e., snapshot rollback) and perform pre-configuration of tasks, creating an initial operation trajectory file and metadata file. The metadata can include at least the agent type, virtual machine name, node identifier, task name, and start time. Then, during the interaction loop based on the executable operation sequence, the reinforcement learning adapter can determine the currently used reinforcement learning training framework and, accordingly, parse one action from the operation information output by the GUI agent into an executable operation. This executable operation is written to the created operation trajectory file and sent to the target virtual machine instance's environment manager (i.e., the target environment server endpoint) to execute the operation in the desktop environment and collect observation data, which is also written to the operation trajectory file. The process continues to the next action parsing step, repeating until the session ends. When the session ends, the environment is cleaned up and the target environment server endpoint is reclaimed. Termination information, including the end time and task success / failure, is written back to the operation trajectory file, and the target environment server endpoint's status is updated from allocated to available. This process enables trajectory acquisition, archiving, and data closure, facilitating subsequent playback, analysis, and retraining.

[0085] In general, the core technologies of the exemplary embodiments of this disclosure are reflected in the following aspects.

[0086] 1. Multi-node virtual machine pooling technology: Through automated deployment and capacity detection, desktop virtual machine instances are created in batches according to proportion or node quotas and environment services are started to form a dynamically scalable resource pool.

[0087] 2. Resource Pool State Machine and Session-based Allocation Technology: The system maintains endpoint metadata and availability (allocated, unavailable) status using structured data files. Combined with health detection (which simultaneously detects the running status of virtual machine instances and the availability of services within the virtual machine, and determines the endpoints as normal, degraded, or faulty based on the detection results, and isolates or reclaims them) and timeout reclamation, it achieves stable session allocation and reclamation. Furthermore, it can avoid duplicate allocation by leveraging session records and resource pool status fields during session allocation.

[0088] 3. Cache injection and concurrency consistency control technology: The control end provides a cache download interface and injects the cache service address during the reset phase, so that the download takes priority through the cache; temporary files are used with atomic renaming to ensure concurrency safety and cache consistency, thereby reducing network bandwidth consumption and initialization overhead.

[0089] 4. Unified observation and trajectory closure technology: The environmental service uniformly returns screenshots and auxiliary information, the control end is packaged to adapt to different reinforcement learning frameworks, and the trajectory and screenshots are recorded step by step for playback and retraining.

[0090] 5. Security and Operations Technology: For virtual machine instances, it supports identity authentication, batch shutdown and destruction, unavailability isolation and exception rollback, reducing the risks of distributed deployment.

[0091] Figure 7 This is a block diagram of a desktop operating engine reinforcement training apparatus according to an exemplary embodiment of the present disclosure. The apparatus is for a desktop operating engine reinforcement training system, the system including a control terminal and multiple device nodes for creating virtual machine instances. Each device node runs at least one environment service endpoint, and each environment service endpoint is used to operate a virtual machine instance on a corresponding device node. (Refer to...) Figure 7 The device includes a determination unit 701, an execution unit 702, and a feedback unit 703.

[0092] The determination unit 701 is configured to determine the target environment service endpoint from multiple environment service endpoints in response to a training session request via a control terminal.

[0093] The execution unit 702 is configured to collect observation data about a target virtual machine instance performing a specified system operation under a specified initial configuration via a control terminal and a target environment server endpoint. The target virtual machine instance is the virtual machine instance operated by the target environment server endpoint. The operation content of the specified system operation is determined based on the operation information output by the desktop operation engine. The observation data is used to describe the changes that occur in the target virtual machine instance due to the execution of the specified system operation.

[0094] Feedback unit 703 is configured to feed back observation data to the desktop operating engine via the control terminal for reinforcement learning training.

[0095] The reinforcement training apparatus for a desktop operating engine according to an exemplary embodiment of this disclosure creates multiple virtual machine instances in a distributed cluster composed of multiple device nodes, and creates corresponding environment service endpoints for each. These environment service endpoints enable unified management and operation of each virtual machine instance, including initial configuration, system operation, and observation data collection, achieving convenient and stable acquisition of interactive samples. Furthermore, when more virtual machine instances are needed, only more environment service endpoints need to be created accordingly. Therefore, the exemplary embodiment of this disclosure can form a controllable and scalable training infrastructure, meeting the needs of large-scale parallel sampling and reinforcement learning training for desktop operating engines.

[0096] Optionally, the determining unit 701 is further configured to, in response to a training session request, determine a target environment service endpoint from multiple environment service endpoints based on endpoint information from multiple environment service endpoints via a control terminal. The endpoint information includes endpoint status, which includes at least one of the following: allocated, available, or unavailable. The available and unavailable statuses are obtained by detecting the running status of the corresponding environment service endpoint and its corresponding virtual machine instance. By performing health checks on each environment service endpoint and marking the available status of the endpoints, the use of unavailable endpoints, such as those experiencing desktop instance crashes, network interruptions, or service incompleteness, can be avoided. This achieves health detection and failure isolation, reducing the risk of faulty instances continuously occupying resources and polluting training data, thus ensuring the smooth progress of training. As an example, when action execution or status observation fails, the control terminal can migrate the current endpoint to unavailable and terminate the session and trajectory recording to prevent the continuous propagation of errors. In batch allocation scenarios, if the allocation is not satisfied, the allocated endpoints can be rolled back and released.

[0097] Optionally, the execution unit 702 is further configured to call the initial configuration interface, system operation interface, and observation data collection interface of the target environment server endpoint through the control terminal, wherein the initial configuration interface, system operation interface, and observation data collection interface adopt a mutual exclusion mechanism; based on the initial configuration interface, system operation interface, and observation data collection interface, it performs specified initial configuration and specified system operations on the target virtual machine instance and collects observation data. By providing clear standardized environment interfaces for implementing operations such as environment initialization, action execution, and state observation, it can be aligned with reinforcement learning frameworks, thereby enabling unified sampling, reward calculation, and trajectory recording during the training phase. By adopting a mutual exclusion mechanism among these interfaces, it can ensure that operations on the same instance are serialized, avoiding state uncertainty caused by concurrent execution of the same instance.

[0098] Optionally, the initial configuration may include environment reset and task pre-configuration. After successful session allocation, the control terminal can call the environment service's reset interface and distribute the task configuration. During the reset process, the environment service first rolls back the virtual machine instance to its initial snapshot to obtain a consistent base state, then executes the pre-configuration according to the task configuration, and finally returns the initial observation data. By providing environment reset and task pre-configuration verification based on snapshot rollback, reset costs can be reduced, thereby reducing the risk of environment state drift and helping to ensure the consistency of task execution across different instances, enabling the reproduction of the same task and reducing training noise.

[0099] Optionally, the initial configuration may include cache service address injection. Cache service address injection refers to injecting the cache service address into the environment service during the reset request. The environment service then writes this address into the environment variables, causing the desktop environment to prioritize obtaining files through the cache interface provided by the control panel during file downloads. This reduces bandwidth waste and inconsistency risks caused by concurrent downloads from multiple instances.

[0100] For embodiments where the specified initial configuration includes all three items mentioned above, regarding pre-task configuration, desktop tasks typically involve preliminary preparations such as downloading data files, uploading materials, configuring network proxies, logging into accounts, installing software dependencies, and launching applications. When multiple instances run concurrently, providing a unified configuration system and caching mechanism can avoid duplicate downloads, reduce initialization costs, and minimize interference introduced by concurrent writes and environmental differences, thus helping to ensure training data quality and model convergence performance.

[0101] Optionally, the training session request includes a batch training session request. The determining unit 701 is further configured to, in response to the batch training session request, determine the number of sessions in the batch training session request via the control terminal, and determine multiple target environment service endpoints corresponding to the number of sessions from multiple environment service endpoints. This embodiment supports batch session allocation, that is, applying for multiple training sessions at once in the batch startup interface, which can realize the batch startup of environment services, thereby running multiple reinforcement learning interaction samples of the desktop operation engine in parallel and improving reinforcement learning training efficiency. By providing desktop instances in parallel through a multi-node resource pool, the upper-layer training system can obtain sessions on demand and generate trajectory data in parallel, significantly shortening the training cycle and improving throughput and scalability.

[0102] Optionally, the specified system operation includes an executable operation sequence. The feedback unit 703 is further configured to: record each executable operation in the executable operation sequence and its corresponding observation data sequentially via the control terminal, and generate an operation trajectory file; and feed the operation trajectory file back to the desktop operation engine. By generating a closed-loop operation trajectory file according to the operation execution order, the upper-level training system can replay and perform quality screening of the trajectory, and further use it for supervised data construction, failure analysis, and retraining, thereby improving the success rate and generalization ability of the desktop operation engine in desktop tasks.

[0103] Optionally, the feedback unit 703 is further configured to: create an initial operation trajectory file via the control terminal; continuously write the currently executed executable operation and the currently collected observation data to the created operation trajectory file during the execution of the executable operation sequence of the target virtual machine instance; and continue writing termination information to the created operation trajectory file after the executable operation sequence is completed, thereby obtaining the operation trajectory file. The termination information includes at least one of the following: end time, task success or failure. By writing the first executable operation, the observation data collected for the first operation, the second executable operation, and so on, in chronological order during operation execution, until all executable operations in the sequence are completed, and finally including the end time and / or task success or failure information, the entire operation trajectory can be completely recorded according to the timeline and the completion status.

[0104] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0105] According to embodiments of this disclosure, an electronic device may be provided. Figure 8This is a block diagram of an electronic device 800 according to an embodiment of the present disclosure. The electronic device includes at least one memory 801 and at least one processor 802. The at least one memory stores a set of computer-executable instructions 8011 and an operating system 8012. When the set of computer-executable instructions 8011 is executed by the at least one processor 802, a method for enhancing the training of a desktop operating engine according to an embodiment of the present disclosure is executed.

[0106] As an example, electronic device 800 may be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned set of instructions. Here, electronic device 800 is not necessarily a single electronic device, but may be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 800 may also be part of an integrated control system or system manager, or may be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0107] In electronic device 800, processor 802 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor 802 may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0108] The processor 802 can execute instructions or code stored in memory, wherein memory 801 can also store data. Instructions and data can also be sent and received over a network via a network interface device, wherein the network interface device can employ any known transmission protocol.

[0109] The memory 801 may be integrated with the processor 802, for example, by placing RAM or flash memory within an integrated circuit microprocessor. Alternatively, the memory 801 may include a separate device, such as an external disk drive, a storage array, or other storage device usable by any database system. The memory 801 and the processor 802 may be operatively coupled, or may communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor 802 to read files stored in the memory 801.

[0110] In addition, the electronic device 800 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device can be interconnected via a bus and / or network.

[0111] According to embodiments of this disclosure, a computer-readable storage medium may also be provided, wherein when instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor causes the processor to perform the enhanced training method for the desktop operating engine of the present disclosure. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0112] According to an embodiment of this disclosure, a computer program product is provided, including computer instructions, which, when executed by a processor, implement an enhanced training method for a desktop operating engine according to an embodiment of this disclosure.

[0113] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

[0114] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for reinforcement training of a desktop operating engine, characterized in that, A reinforcement training system for a desktop operating engine, the system comprising a control terminal and multiple device nodes for creating virtual machine instances, each device node running at least one environment service endpoint, each environment service endpoint for operating a virtual machine instance on the corresponding device node, wherein the method includes: Through the control terminal, in response to a training session request, the target environment service endpoint is determined from multiple environment service endpoints; Through the control terminal and via the target environment server endpoint, observation data is collected regarding the target virtual machine instance performing a specified system operation under a specified initial configuration. The target virtual machine instance is the virtual machine instance operated by the target environment server endpoint. The operation content of the specified system operation is determined based on the operation information output by the desktop operation engine. The observation data is used to describe the changes that occur in the target virtual machine instance due to the execution of the specified system operation. The control terminal feeds back the observation data to the desktop operating engine for reinforcement learning training.

2. The reinforcement training method for the desktop operating engine as described in claim 1, characterized in that, The step of determining the target environment service endpoint from multiple environment service endpoints through the control terminal in response to a training session request includes: In response to the training session request, the control terminal determines the target environment service endpoint from the multiple environment service endpoints based on the endpoint information of the multiple environment service endpoints. The endpoint information includes the endpoint status, which includes at least one of the following: allocated, available, and unavailable. The available and unavailable statuses are obtained by detecting the running status of the corresponding environment service endpoint and its corresponding virtual machine instance.

3. The reinforcement training method for the desktop operating engine as described in claim 1, characterized in that, The process of collecting observation data about the target virtual machine instance performing specified system operations under a specified initial configuration via the control terminal and the target environment server endpoint includes: The control terminal calls the initial configuration interface, system operation interface, and observation data collection interface of the target environment server endpoint, respectively, wherein the initial configuration interface, the system operation interface, and the observation data collection interface adopt a mutual exclusion mechanism. Based on the initial configuration interface, the system operation interface, and the observation data collection interface, the specified initial configuration and the specified system operation are executed on the target virtual machine instance, and the observation data is collected.

4. The method for enhancing the training of a desktop operating engine as described in claim 1, characterized in that, The specified initial configuration includes environment reset and task pre-configuration; and / or The configuration content of the specified initial configuration includes cache service address injection.

5. The method for enhancing the training of a desktop operating engine as described in claim 1, characterized in that, The training session request includes a batch training session request, wherein determining the target environment service endpoint from multiple environment service endpoints in response to the training session request includes: In response to the batch training session request, the number of sessions for the batch training session request is determined, and multiple target environment service endpoints corresponding to the number of sessions are determined from the multiple environment service endpoints.

6. The method for enhancing the training of a desktop operating engine as described in claim 1, characterized in that, The specified system operation includes an executable sequence of operations, wherein the step of feeding back the observed data to the desktop operation engine through the control terminal includes: The control terminal records each executable operation in the executable operation sequence and its corresponding observation data in sequence, and generates an operation trajectory file. The operation trajectory file is fed back to the desktop operation engine.

7. The method for enhancing the training of a desktop operating engine as described in claim 6, characterized in that, The step of sequentially recording each executable operation in the executable operation sequence and its corresponding observation data, and generating an operation trajectory file, includes: Create an initial operation trajectory file; During the execution of the executable operation sequence by the target virtual machine instance, the currently executed executable operation and the currently collected observation data are continuously written to the created operation trajectory file; After the executable operation sequence is completed, termination information is written to the created operation trajectory file to obtain the operation trajectory file. The termination information includes at least one of the following: end time, task success or failure.

8. A desktop operating engine enhancement training device, characterized in that, A reinforcement training system for a desktop operating engine, the system comprising a control terminal and multiple device nodes for creating virtual machine instances, each device node running at least one environment server endpoint, each environment server endpoint for operating a virtual machine instance on the corresponding device node, wherein the apparatus includes: The determining unit is configured to determine the target environment service endpoint from multiple environment service endpoints in response to a training session request via the control terminal. An execution unit is configured to collect observation data about a target virtual machine instance performing a specified system operation under a specified initial configuration via the control terminal and the target environment server endpoint. The target virtual machine instance is a virtual machine instance operated by the target environment server endpoint. The operation content of the specified system operation is determined based on the operation information output by the desktop operation engine. The observation data is used to describe the changes that occur to the target virtual machine instance due to the execution of the specified system operation. The feedback unit is configured to feed back the observation data to the desktop operating engine through the control terminal, so that the desktop operating engine can perform reinforcement learning training.

9. An electronic device, characterized in that, include: At least one processor; At least one memory that stores computer-executable instructions. Wherein, when the computer-executable instructions are executed by the at least one processor, the at least one processor causes the at least one processor to execute the enhanced training method for the desktop operating engine as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by at least one processor, they cause the at least one processor to perform the reinforcement training method for the desktop operating engine as described in any one of claims 1 to 7.

11. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by at least one processor, they cause the at least one processor to perform the reinforcement training method for the desktop operating engine as described in any one of claims 1 to 7.

Images