Large-scale parallel sampling environment construction method and device for reinforcement learning of intelligent computing cloud platform through computing power

Abstract

The invention provides a large-scale parallel sampling environment construction method and device for reinforcement learning of an intelligent computing cloud platform through computing power, and the method comprises the steps: obtaining a plurality of physical disks of a plurality of computing power nodes of the intelligent computing cloud platform, and constructing the plurality of physical disks into a bottom storage pool through an RAID technology, storing the basic mirror image of the operating system in a qcow2 format, and further generating a plurality of virtual machine instances and running the virtual machine instances; packaging the QEMU process and the running environment of the QEMU process into a container, and performing distributed scheduling on the container according to the disk topological relation among the plurality of bottom storage pools on the plurality of computing power nodes and the real-time I / O load condition. Therefore, the construction of a large-scale parallel sampling environment of reinforcement learning can be realized by utilizing the computing power of an intelligent computing cloud platform and integrating storage optimization, hardware virtualization and container arrangement technologies.

Description

Technical Field

[0001] This invention relates to the field of intelligent computing cloud platform and computing infrastructure technology, specifically to a method and apparatus for constructing a large-scale parallel sampling environment for reinforcement learning through computing power in an intelligent computing cloud platform. Background Technology

[0002] With the rapid development of artificial intelligence technology, the "intelligent computing cloud platform" has emerged.

[0003] "Intelligent computing cloud platform" refers to a cloud computing platform that provides intelligent computing services based on the hardware and software resources of an intelligent computing center.

[0004] "Intelligent computing" refers to computing driven by intelligent computing power (GPU, FPGA, ASIC, etc.), and is a general term for computing theories, methods, architectures, and technical capabilities characterized by data intensity and model-based nature. Its goal is to obtain better results by utilizing specific computing resources and appropriate computing methods according to the needs of the task.

[0005] A "Smart Computing Center" refers to a facility that utilizes large-scale heterogeneous computing resources, including general-purpose computing power (CPU) and intelligent computing power (GPU, FPGA, ASIC, etc.), to provide the necessary computing power, data, and algorithms for artificial intelligence applications (such as the development, training, and inference of deep learning models). A Smart Computing Center encompasses facilities, hardware, and software, and can provide full-stack capabilities from underlying computing power to top-level application enablement.

[0006] "Computing power" is the core of "intelligent computing cloud platform". It is the ability of computer equipment or computing / data center to process information. It is the ability of computer hardware and software to work together to perform a certain computing requirement. It is the computing power to achieve the target output by processing information data. It is a new type of productivity that integrates information computing power, network carrying capacity and data storage capacity. It mainly provides services to society through computing power infrastructure.

[0007] In large-scale reinforcement learning, especially in the training of general-purpose agents, a long-standing core need is how to efficiently and massively construct tens of thousands of isolated, reproducible, and high-performance parallel sampling environments. These sampling environments typically include complete operating systems and graphical user interfaces, such as the Ubuntu desktop and Windows applications.

[0008] However, existing technologies face the following technical challenges in fulfilling this requirement: First, traditional container technologies (such as Docker) cannot provide a complete operating system environment, lacking necessary kernel support, graphical interfaces, and device emulation capabilities, thus limiting their applicability in complex application scenarios. Second, traditional virtual machines have slow startup speeds, and when using qcow2 (QEMU Copy On Write version 2) images, disk concurrent read performance is poor, easily causing I / O (Input / Output) to become a bottleneck for system performance. Furthermore, the limitations of single-machine resources make horizontal scaling difficult, failing to effectively support the needs of large-scale parallel environments. Finally, although lightweight operating systems (such as Ubuntu) have advantages in resource consumption, in low-configuration environments (such as 2C5G), the smoothness of their graphical interfaces is often affected, severely impacting the realism of agent-environment interaction and training effectiveness. Therefore, existing technologies have consistently lacked a method for constructing large-scale parallel sampling environments for reinforcement learning.

[0009] In summary, since the emergence of intelligent computing centers, how to construct a large-scale parallel sampling environment for reinforcement learning has become an urgent technical problem to be solved. Summary of the Invention

[0010] This invention provides a method and apparatus for constructing a large-scale parallel sampling environment for reinforcement learning on an intelligent computing cloud platform, in order to solve the technical problem of how to construct a large-scale parallel sampling environment for reinforcement learning since the emergence of intelligent computing centers.

[0011] To address the aforementioned technical problems, the present invention provides the following aspects: In a first aspect, the present invention provides a method for constructing a large-scale parallel sampling environment for reinforcement learning on an intelligent computing cloud platform, the method comprising: Step S1: Obtain multiple physical disks from multiple computing nodes of the intelligent computing cloud platform, construct the multiple physical disks into an underlying storage pool using RAID technology, and store the base image of the operating system in the underlying storage pool in qcow2 format; Step S2: Based on the base image of the operating system, generate multiple state-isolated virtual machine instances through read-only mounting and copy-on-write mechanisms; Step S3: Create a virtualization environment based on the KVM kernel module integrated in the computing node and the hardware virtualization extension capability of the CPU of the computing node, and use the QEMU process to simulate the hardware device of the computing node in the virtualization environment to run the virtual machine instance. Step S4: Based on container technology, the QEMU process that hosts the virtual machine instance and the runtime environment of the QEMU process are encapsulated into a container. According to the disk topology relationship between multiple underlying storage pools on multiple computing power nodes of the intelligent computing cloud platform and the real-time I / O load, the container is distributed and scheduled using Kubernetes.

[0012] Optionally, step S1 includes: step S11: obtaining multiple physical disks, wherein the number of physical disks is not less than ten; step S12: constructing the underlying storage pool based on the physical disks using RAID technology; step S13: storing the base image of the operating system including Ubuntu and / or Windows in the underlying storage pool in qcow2 format.

[0013] Optionally, step S2 includes: step S21: creating an independent, initially empty write layer file for each virtual machine instance to be generated; step S22: mounting the operating system's base image in read-only mode and combining it with the write layer file to form a system disk view; step S23: starting and generating the corresponding virtual machine instance based on the system disk view; wherein, when a data write request occurs during the operation of the virtual machine instance, the data to be modified is copied from the read-only base image to the corresponding write layer file for modification through the copy-on-write mechanism, so as to achieve state isolation between multiple virtual machine instances.

[0014] Optionally, step S3 includes: step S31, loading and running the KVM kernel module in the operating system kernel of the computing node to enable the invocation of the hardware virtualization extension capability of the CPU of the computing node; step S32, using the KVM kernel module to put the CPU of the computing node into a mode capable of executing the instructions of the virtual machine instance to create the virtualization environment; step S33, starting the QEMU process in the virtualization environment and using the QEMU process to simulate the hardware device of the computing node; step S34, loading and running the virtual machine instance based on the QEMU process.

[0015] Optionally, after step S3, the method further includes: step S5: performing reproducible state management on the virtual machine instance.

[0016] Optionally, step S5 includes: step S51: creating a state snapshot of the virtual machine instance at a specific point in time based on the system disk view corresponding to the virtual machine instance, wherein the state snapshot records the complete state of the base image and the write layer file at the specific point in time; step S52: storing the state snapshot in the underlying storage pool; step S53: reproducing the sampling environment corresponding to the state snapshot by rolling back the current state of the virtual machine instance to the state snapshot.

[0017] Optionally, after step S4, the method further includes: Step S6: Perform full lifecycle management on the virtual machine instance, which includes: power-on management, operation management, and release management; The boot management includes: allocating computing power to the virtual machine instance according to the type of operating system corresponding to the virtual machine instance, and starting the virtual machine instance; wherein, for a virtual machine instance configured to run the Ubuntu desktop operating system, 2 CPUs and 5GB of memory are allocated; for a virtual machine instance configured to run the Windows operating system, 4 CPUs and 8GB of memory are allocated. The operation management includes: performing simulated mouse operations and / or simulated keyboard operations on the virtual machine instance during its operation, and capturing the screen image corresponding to the virtual machine instance. The release management includes: shutting down the virtual machine instance and reclaiming the computing power occupied by the virtual machine instance.

[0018] Secondly, the present invention provides a device for constructing a large-scale parallel sampling environment for reinforcement learning using computing power on an intelligent computing cloud platform, the device comprising: The acquisition module is used to execute step S1: acquire multiple physical disks of multiple computing nodes of the intelligent computing cloud platform, construct the multiple physical disks into an underlying storage pool through RAID technology, and store the base image of the operating system in the underlying storage pool in qcow2 format; The execution module is used to execute step S2: based on the base image of the operating system, multiple state-isolated virtual machine instances are generated through read-only mounting and copy-on-write mechanism; Step S3: Create a virtualization environment based on the KVM kernel module integrated in the computing node and the hardware virtualization extension capability of the CPU of the computing node, and use the QEMU process to simulate the hardware device of the computing node in the virtualization environment to run the virtual machine instance. Step S4: Based on container technology, the QEMU process that hosts the virtual machine instance and the runtime environment of the QEMU process are encapsulated into a container. According to the disk topology relationship between multiple underlying storage pools on multiple computing power nodes of the intelligent computing cloud platform and the real-time I / O load, the container is distributed and scheduled using Kubernetes.

[0019] Thirdly, the present invention provides a server comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the method for constructing a large-scale parallel sampling environment for reinforcement learning by a smart computing cloud platform as described in the first aspect above.

[0020] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a method for constructing a large-scale parallel sampling environment for reinforcement learning using computing power on an intelligent computing cloud platform as described in the first aspect above.

[0021] Fifthly, the present invention provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of a method for constructing a large-scale parallel sampling environment for reinforcement learning via computing power in an intelligent computing cloud platform as described in the first aspect above.

[0022] In this invention, step S1 fully utilizes the computing power of the intelligent computing cloud platform and employs RAID technology to construct the underlying storage pool, significantly improving the concurrent read I / O performance of the base image and effectively solving the disk bottleneck problem of the traditional qcow2 image in high-concurrency scenarios. Step S2 combines read-only mounting and copy-on-write mechanisms to achieve second-level fast cloning and state isolation of virtual machine instances, ensuring the independence and reproducibility of the environment. Step S3, based on KVM and QEMU's complete hardware virtualization technology, provides a real OS environment with an independent kernel and graphical interface, overcoming the inherent defects of container technology in this regard. Finally, step S4 containerizes the QEMU process and its runtime environment and utilizes Kubernetes to intelligently schedule the computing power of the intelligent computing cloud platform, achieving elastic scaling and load balancing of resources across computing power nodes, breaking through the single-machine resource limitation, and supporting a parallel sampling environment of tens of thousands.

[0023] In summary, through the synergistic effect of the above technologies and based on the computing power of the intelligent computing cloud platform, a high-performance, high-fidelity, and elastically scalable large-scale parallel sampling environment for reinforcement learning has been successfully constructed. Attached Figure Description

[0024] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating a method for constructing a large-scale parallel sampling environment for reinforcement learning using computing power in an intelligent computing cloud platform, as provided by this invention. Figure 2 A structural block diagram of a device for constructing a large-scale parallel sampling environment for reinforcement learning using computing power in an intelligent computing cloud platform, provided by the present invention. Figure 3 This is a schematic diagram of the electronic device of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] The technical terms involved in this invention will be briefly explained below.

[0027] The “computing power” mentioned in this invention refers to: the ability of computer equipment or computing / data center to process information; the ability of computer hardware and software to work together to perform a certain computing requirement; the computing power to achieve the target result output by processing information data; and a new type of productivity that integrates information computing power, network carrying capacity, and data storage capacity, mainly providing services to society through computing power infrastructure.

[0028] The "computational power" (CP) described in this invention refers to the ability of a data center server to process data and output results. It is a comprehensive indicator of a data center's computing power, encompassing general computing power, supercomputing power, and intelligent computing power. The commonly used unit of measurement is floating-point operations per second (FLOPS, 1 EFLOPS = 10^18 FLOPS), with higher values ​​indicating stronger overall computing power. It is estimated that 1 EFLOPS is approximately equivalent to the computing power output of 5 Tianhe-2A supercomputers, 500,000 mainstream server CPUs, or 2 million mainstream laptops. The calculation formula is: CP = CP 通用 +CP 智能 +CP 超级 .

[0029] The "Network Power" (NP) mentioned in this invention refers to the performance of data transmission capability of computing facilities, which includes comprehensive capabilities such as network architecture, network bandwidth, transmission latency, intelligent management and scheduling, and involves network transmission within and between data centers. It is a comprehensive indicator for measuring network transmission scheduling capability.

[0030] The "Storage Power" (SP) described in this invention refers to the comprehensive capabilities of a data center in four aspects: data storage capacity, performance, security and reliability, and green and low-carbon operation. It is a comprehensive indicator for measuring the data storage capacity of a data center, including external storage devices such as storage arrays and internal storage devices within servers. The commonly used unit of measurement for storage capacity is exabytes (EB, 1EB = 2^60 bytes), the commonly used unit of measurement for performance is the number of read / write operations per second (IOPS / TB), and the disaster recovery ratio is an important indicator of security and reliability.

[0031] The "computing infrastructure" mentioned in this invention refers to a new type of information infrastructure that integrates information computing power, network carrying capacity, and data storage capacity, enabling centralized computing, storage, transmission, and application of information.

[0032] The "new information infrastructure" mentioned in this invention refers to network infrastructure such as 5G networks, fiber optic broadband networks, backbone networks, international communication networks, and satellite internet; computing infrastructure such as data centers, general computing centers, intelligent computing centers, and supercomputing centers; and new technology facilities such as artificial intelligence, blockchain, and quantum computing.

[0033] The “computing power” mentioned in this invention includes: “general computing power”, “intelligent computing power” and “supercomputing power”.

[0034] The "general computing power" mentioned in this invention refers to the computing power provided by servers based on CPU (Central Processing Unit) chips, which is used to support basic general computing such as cloud computing and edge computing.

[0035] The "intelligent computing power" mentioned in this invention refers to: a computing platform deployed on a large scale based on dedicated chips such as GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), and ASIC (Application Specific Integrated Circuit) for various artificial intelligence innovative applications, such as natural language processing and machine vision.

[0036] The “supercomputing power” mentioned in this invention refers to the computing power provided by high-performance computing clusters such as supercomputers. It utilizes the centralized computing resources of multiple computer systems working in parallel and uses a dedicated operating system to handle extremely complex or data-intensive problems. It is mainly used for computing in cutting-edge scientific fields, such as planetary simulation, drug molecule design, and gene analysis.

[0037] The "intelligent computing" mentioned in this invention refers to computing driven by intelligent computing power (GPU, FPGA, ASIC, etc.), which is a general term for computing theories, methods, architectures and technical capabilities with characteristics such as data intensity and model-based computing. Its goal is to obtain better results by utilizing specific computing resources and appropriate computing methods according to the needs of the task.

[0038] The "intelligent computing cloud platform" mentioned in this invention refers to a cloud computing platform that provides intelligent computing services based on the hardware and software resources of an intelligent computing center.

[0039] The "intelligent computing center" described in this invention refers to a facility that utilizes large-scale heterogeneous computing resources, including general-purpose computing power (CPU) and intelligent computing power (GPU, FPGA, ASIC, etc.), to primarily provide the necessary computing power, data, and algorithms for artificial intelligence applications (such as deep learning model development, model fine-tuning, and model inference). The intelligent computing center encompasses facilities, hardware, and software, and can provide full-stack capabilities from underlying computing power to top-level application enablement.

[0040] The "intelligent computing center" mentioned in this invention includes, but is not limited to, "smart computing center".

[0041] The "intelligent computing center" mentioned in this invention, also known as an artificial intelligence computing center, is a type of computing infrastructure that provides computing power services, data services, and algorithm services required for artificial intelligence applications, based on artificial intelligence theory and adopting an artificial intelligence computing architecture.

[0042] The "computing center" mentioned in this invention refers to a facility that is mainly composed of infrastructure such as wind, thermal, hydro, and electricity, and IT hardware and software equipment, and has computing power, carrying capacity, and storage capacity, including general data centers, intelligent computing centers, supercomputing centers, etc.

[0043] The "supercomputing center" mentioned in this invention refers to a supercomputing data center, which is a data center based on supercomputers or large-scale computing clusters. It can provide large-scale computing, storage and network services and is widely used in aerospace, defense, oil exploration, climate modeling and genome sequencing and other application scenarios.

[0044] The “computing resources” mentioned in this invention refer to the technologies and facilities required for the development of the digital society that have the ability to compute, transmit, store and apply information, including but not limited to computing resources such as CPUs and GPUs, network resources such as switches and routers, storage resources such as storage arrays and distributed storage, security resources such as firewalls and intrusion detection systems, and supporting and guaranteeing resources such as wind, fire, water and electricity.

[0045] The "computing node" mentioned in this invention refers to the computing resources of a server / container capable of processing computing tasks.

[0046] The "agent" described in this invention refers to an agent capable of perceiving its environment and taking actions to achieve specific goals. It can be software, hardware, or a system, possessing autonomy, adaptability, and interactivity. The agent perceives changes in the environment (e.g., through sensors or data input), makes judgments and decisions based on its learned knowledge and algorithms, and then executes actions to influence the environment or achieve predetermined goals. Agents are widely used in the field of artificial intelligence, commonly found in automated systems, robots, virtual assistants, and game characters. Their core lies in their ability to learn autonomously and continuously evolve to better complete tasks and adapt to complex environments.

[0047] The "RAID (Redundant Array of Independent Disks) technology" described in this invention refers to a data storage scheme that combines multiple physical disks into a single logical disk. By striping data across all disks, concurrent read and write operations are achieved, thereby significantly improving storage performance, but without data redundancy.

[0048] The "qcow2 format" mentioned in this invention refers to a dynamically allocated, copy-on-write disk image file format commonly used in virtual machines. It only occupies physical storage space when data is actually written and supports advanced features such as snapshots and compression.

[0049] The "virtual machine instance" mentioned in this invention refers to a logical computer system with complete hardware system functions simulated through virtualization technology. It runs in an isolated environment and has an independent operating system and applications.

[0050] The "KVM (Kernel-based Virtual Machine) kernel module" mentioned in this invention refers to an open-source virtualization module integrated into the Linux kernel that can transform the Linux kernel into a bare-metal management program, thereby allowing the host machine to directly run multiple unmodified guest operating systems.

[0051] The "QEMU (Quick Emulator) process" mentioned in this invention refers to an open-source machine simulator and virtualization software that simulates CPU, memory, storage devices and peripheral hardware in a purely software manner. It often works in conjunction with KVM to provide efficient hardware virtualization capabilities.

[0052] The "container technology" described in this invention refers to a lightweight, portable virtualization solution. Its core lies in packaging an application and its complete runtime dependencies (including code, libraries, configuration files, etc.) into an independent, standardized unit called a "container" through operating system-level resource isolation and process encapsulation. Containers share the host machine's operating system kernel but run in isolation within user space, thereby ensuring rapid and reliable deployment and scaling of applications across different computing environments.

[0053] The "Kubernetes (K8S)" mentioned in this invention refers to an open-source container orchestration and management system used for the automatic deployment, scaling, and management of containerized applications. It is widely used for container orchestration, helping developers and operations personnel efficiently manage large numbers of containers.

[0054] In this invention, "Pod" refers to the smallest and simplest atomic unit that can be scheduled to host and run containerized applications in the Kubernetes container orchestration system.

[0055] First, it is important to emphasize that building a large-scale parallel sampling environment for reinforcement learning faces multi-dimensional technical challenges. The efficiency and quality of reinforcement learning training directly depend on the ability to quickly and stably generate massive amounts of high-fidelity environmental interaction data. This not only requires individual sampling environments to have complete operating system functions and realistic interactive interfaces to support complex agent tasks, but also demands the ability to simultaneously coordinate and manage tens of thousands of such environment instances at a cluster scale, achieving efficient resource utilization, rapid instance start-up and shutdown, reliable state isolation, and smooth I / O performance.

[0056] However, existing technologies suffer from irreconcilable contradictions in several key dimensions, including environmental integrity, resource elasticity, I / O efficiency, and scheduling flexibility. This has made constructing a large-scale parallel sampling environment for reinforcement learning that simultaneously meets all requirements a long-standing architectural challenge. The method described below for constructing a large-scale parallel sampling environment for reinforcement learning using a smart computing cloud platform is precisely designed to address this challenge.

[0057] Figure 1 This invention illustrates a method for constructing a large-scale parallel sampling environment for reinforcement learning using computing power on an intelligent computing cloud platform, according to an embodiment of this application. Figure 1 As shown, the method includes: Step S1: Obtain multiple physical disks from multiple computing nodes of the intelligent computing cloud platform, construct the multiple physical disks into an underlying storage pool using RAID technology, and store the base image of the operating system in the underlying storage pool in qcow2 format; Step S2: Based on the operating system base image, generate multiple state-isolated virtual machine instances using read-only mounting and copy-on-write mechanisms; Step S3: Create a virtualization environment based on the KVM kernel module integrated in the computing node and the hardware virtualization extension capability of the computing node's CPU. In the virtualization environment, use the QEMU process to simulate the hardware device of the computing node to run virtual machine instances. Step S4: Based on container technology, the QEMU process that hosts the virtual machine instance and the runtime environment of the QEMU process are encapsulated into a container. According to the disk topology relationship between multiple underlying storage pools on multiple computing power nodes of the intelligent computing cloud platform and the real-time I / O load, Kubernetes is used to perform distributed scheduling of the containers.

[0058] It should be noted that this invention describes a method for constructing a large-scale parallel sampling environment for reinforcement learning training on an intelligent computing cloud platform, utilizing its computing power. The core of the method lies in the efficient and flexible transformation of physical hardware resources into a large number of usable virtual machine instances through the synergistic integration of a series of technologies.

[0059] The specific implementation process is as follows: Multiple physical disks from multiple computing nodes in the intelligent computing cloud platform are acquired, and RAID technology is used to construct a unified, high-performance underlying storage pool. The base image of the operating system (such as Ubuntu or Windows) is then stored in this underlying storage pool in qcow2 format. This leverages the parallel data read / write capabilities of RAID to significantly improve I / O performance during subsequent high-concurrency access. Based on this, utilizing the characteristics of the qcow2 format, the operating system's base image is accessed via read-only mounting. Combined with a copy-on-write mechanism, an independent write layer is created for each virtual machine instance to be generated, thereby quickly deriving multiple completely isolated virtual machine instances in terms of disk state. This ensures the reproducibility and isolation of the final sampling environment.

[0060] The operation of a virtualized environment depends on the capabilities of the computing node itself. A high-efficiency virtualization environment can be created by loading and running the KVM kernel module within the computing node's operating system kernel, leveraging the CPU's hardware virtualization capabilities (Intel VT-x / AMD-V). Within this virtualized environment, the required hardware devices (such as CPU, memory, and network devices) can be simulated by starting the QEMU process, thereby loading and running the generated virtual machine instances. This provides a real operating system environment with a complete independent kernel and graphical interface, which is impossible to achieve with container technology.

[0061] Ultimately, to achieve flexible management and cross-node scheduling of massive virtual machine instances, the QEMU process carrying the virtual machine instances and its complete runtime environment can be further encapsulated into standard containers using container technology. Then, leveraging the powerful container orchestration system Kubernetes, these containers can be intelligently distributed and allocated resources based on the disk topology between multiple underlying storage pools on multiple computing power nodes within the intelligent computing cloud platform and the real-time I / O load. This avoids scheduling multiple high-I / O Pods to the same computing power node. Furthermore, this invention is not limited to Kubernetes and can also use other container orchestration management systems.

[0062] In summary, a large-scale parallel sampling environment that can be dynamically expanded, performs high-performance, and maintains high environmental fidelity can be successfully constructed, laying the foundation for efficient reinforcement learning training.

[0063] In one possible implementation, step S1 includes: step S11: obtaining multiple physical disks, wherein the number of physical disks is not less than ten; step S12: constructing an underlying storage pool based on the physical disks using RAID technology; step S13: storing the base image of the operating system, including Ubuntu and / or Windows, in the underlying storage pool in qcow2 format.

[0064] It should be noted that the specific implementation of step S1 is as follows: First, obtain no fewer than ten physical disks from the computing power nodes of the intelligent computing cloud platform; second, based on these physical disks, integrate them into a unified underlying storage pool using RAID technology; finally, store the base images of various operating systems, such as Ubuntu and / or Windows, in the constructed underlying storage pool using the specific file format qcow2.

[0065] Therefore, by aggregating at least ten physical disks and using RAID technology to combine their I / O capabilities, a high-bandwidth, low-latency shared storage infrastructure can be built. This infrastructure is specifically designed to efficiently store the base image of the operating system, which serves as the source of all virtual machine instances. This provides storage performance assurance for large-scale parallel sampling tasks of reinforcement learning that require running a large number of virtual machine instances.

[0066] In one possible implementation, step S2 includes: Step S21: Create an independent, initially empty write layer file for each virtual machine instance to be generated; Step S22: Mount the operating system base image in read-only mode and combine it with the write layer file to form a system disk view; Step S23: Start and generate the corresponding virtual machine instance based on the system disk view; wherein, when a data write request occurs during the operation of a virtual machine instance, the data to be modified is copied from the read-only base image to the corresponding write layer file for modification through a copy-on-write mechanism, so as to achieve state isolation between multiple virtual machine instances.

[0067] It should be noted that the specific implementation of step S2 is as follows: First, a separate dedicated write layer file with an initial empty state is created for each virtual machine instance to be generated; then, the base image of the operating system already stored in the underlying storage pool is mounted in a strictly read-only manner, and this read-only base image is logically combined with the separate write layer file created for the corresponding virtual machine instance to form a complete system disk view that can be used to start the virtual machine instance; finally, based on this combined system disk view, the corresponding virtual machine instance can be started and generated.

[0068] The key to this process lies in the copy-on-write mechanism. When any virtual machine instance sends a data write request to the system disk during operation, this mechanism is triggered, copying the data block to be modified from the base image, which is in a read-only state, to the write layer file dedicated to that virtual machine instance, and then performing the modification operation within the write layer file.

[0069] Therefore, all virtual machine instances share the same read-only base image at startup, enabling the generation of virtual machine instances in seconds, which greatly improves the deployment efficiency of virtual machine instances. During the operation of virtual machine instances, all data write operations of each virtual machine instance are directed to its own independent write layer file, ensuring that disk modification behavior between virtual machine instances is completely isolated and does not affect each other. This fundamentally guarantees the independence and reproducibility of the state of each virtual machine instance in the large-scale parallel sampling environment of reinforcement learning.

[0070] In one possible implementation, step S3 includes: step S31, loading and running the KVM kernel module in the operating system kernel of the computing node to enable the invocation of the hardware virtualization extension capabilities of the CPU of the computing node; step S32, using the KVM kernel module to put the CPU of the computing node into a mode capable of executing instructions of a virtual machine instance to create a virtualization environment; step S33, starting the QEMU process in the virtualization environment and using the QEMU process to simulate the hardware devices of the computing node; step S34, loading and running a virtual machine instance based on the QEMU process.

[0071] It should be noted that step S3 specifically involves: loading and running the KVM kernel module inherent in the operating system kernel of the computing node, thereby activating and enabling low-level calls to the hardware virtualization extension capabilities of the computing node's central processing unit (CPU), which is the foundation for achieving efficient virtualization. Secondly, through the loaded KVM kernel module, the computing node's CPU is placed in a working mode capable of directly and safely executing instructions issued by virtual machine instances, thus creating a basic virtualization environment at the hardware level.

[0072] After the virtualization environment is successfully created, a QEMU process needs to be started. This QEMU process is responsible for simulating a complete set of virtual hardware devices necessary for the virtual machine instance to run, including a virtual CPU, virtual memory, a virtual storage controller, and a virtual network interface card. Finally, based on this QEMU process configured with simulated hardware devices, the virtual machine instance generated in step S2 is loaded and run.

[0073] Thus, by closely coordinating the KVM kernel module's direct management of CPU hardware virtualization capabilities and the QEMU process's full-function simulation of peripheral hardware devices, a high-performance and fully functional virtualization environment is constructed, enabling each virtual machine instance to run its own independent operating system kernel and graphical user interface like an independent physical computer, thereby providing a highly faithful sampling environment for reinforcement learning.

[0074] In one possible implementation, after step S3, the method further includes: Step S5: Performing reproducible state management on the virtual machine instance. Step S5 includes: Step S51: Based on the system disk view corresponding to the virtual machine instance, creating a state snapshot of the virtual machine instance at a specific point in time, wherein the state snapshot records the complete state of the base image and write layer files at the specific point in time; Step S52: Storing the state snapshot in the underlying storage pool; Step S53: Reproducing the sampling environment corresponding to the state snapshot by rolling back the current state of the virtual machine instance to the state snapshot.

[0075] It should be noted that after completing the creation and operation of the virtual machine instance (step S3), step S5 can be executed: reproducible state management of the virtual machine instance. This step aims to solve the problem of how to accurately backtrack and repeat experiments on the environment state during reinforcement learning training. The specific operation process is as follows: First, based on the system disk view corresponding to the virtual machine instance, which is composed of a read-only base image and independent write layer files, a state snapshot of the virtual machine instance is created at a specific point in time. This state snapshot completely records all the data states of the base image and write layer files at this moment, forming a frozen checkpoint that can be restored later. Then, this state snapshot containing complete state information is securely stored in an underlying storage pool built by multiple physical disks through RAID technology to ensure its persistence and reliability. When it is necessary to reproduce a specific sampling environment, the current running state of the virtual machine instance can be rolled back to any previously saved state snapshot, thereby accurately reproducing the complete sampling environment at the time corresponding to the snapshot.

[0076] In one possible implementation, after step S4, the method further includes: step S6: performing full lifecycle management on the virtual machine instance, which includes: power-on management, operation management, and release management; Boot management includes: allocating computing power to virtual machine instances based on the type of operating system corresponding to the virtual machine instance, and starting the virtual machine instance; specifically, for virtual machine instances configured to run the Ubuntu desktop operating system, 2 CPUs and 5GB of memory are allocated; for virtual machine instances configured to run the Windows operating system, 4 CPUs and 8GB of memory are allocated. Operation management includes: performing simulated mouse operations and / or simulated keyboard operations on the virtual machine instance during its operation, and capturing the corresponding screen image of the virtual machine instance; Release management includes shutting down virtual machine instances and reclaiming the computing power used by virtual machine instances.

[0077] Simulated mouse operations include, but are not limited to, simulating the following mouse operations: mouse click, mouse double click, mouse scroll, mouse drag, mouse move, etc. Simulated keyboard operations include, but are not limited to, simulating the following keyboard operations: shortcut keys (such as Ctrl+C), function key operations, input operations, etc.

[0078] Capture can be a single frame or multiple frames, and it can also record video of the screen image corresponding to the virtual machine instance.

[0079] It should be noted that the full lifecycle management described in step S6 is the operational support for achieving stable and efficient operation of the reinforcement learning large-scale parallel sampling environment. By managing the entire lifecycle of virtual machine instances, a complete closed loop from virtual machine instance creation, use to recycling is systematically covered, ensuring that reinforcement learning training tasks can be executed continuously and orderly.

[0080] Specifically, the boot management allocates computing power differently based on the inherent needs of different operating systems (e.g., allocating 2 CPUs / 5GB to Ubuntu and 4 CPUs / 8GB to Windows), thereby optimizing the configuration of basic resources while ensuring the smoothness of the graphical interface. During instance operation, the operation management assumes the key function of training interaction. Finally, the release management is responsible for safely shutting down the virtual machine instance and reclaiming its occupied computing power after the task is completed, ensuring that valuable computing power can be quickly rescheduled to serve subsequent new tasks. This provides a reliable and efficient automated framework for the continuous creation, interaction, and reclamation of thousands of reinforcement learning massively parallel sampling environments, enabling large-scale reinforcement learning parallel training tasks to be carried out continuously and automatically on the intelligent computing cloud platform.

[0081] In addition, this invention supports elastic scaling of tens of thousands of concurrent rollouts (sampling trajectories). Based on Kubernetes' elastic scaling mechanism, it can automatically and in parallel deploy or reclaim up to tens of thousands of Pods by monitoring cluster resource metrics, so as to dynamically respond to the large-scale requirements of reinforcement learning tasks for the sampling environment.

[0082] This enables the large-scale parallel sampling environment for reinforcement learning, built on the same underlying technical architecture, to dynamically adapt to the different requirements of various reinforcement learning tasks for operating system type and computing power. It can provide appropriate computing power for the lightweight Ubuntu desktop environment to ensure the smooth operation of its graphical interface, and also provide sufficient computing power support for more complex Windows operating systems and applications.

[0083] In this invention, step S1 fully utilizes the computing power of the intelligent computing cloud platform and employs RAID technology to construct the underlying storage pool, significantly improving the concurrent read I / O performance of the base image and effectively solving the disk bottleneck problem of the traditional qcow2 image in high-concurrency scenarios. Step S2 combines read-only mounting and copy-on-write mechanisms to achieve second-level fast cloning and state isolation of virtual machine instances, ensuring the independence and reproducibility of the environment. Step S3, based on KVM and QEMU's complete hardware virtualization technology, provides a real OS environment with an independent kernel and graphical interface, overcoming the inherent defects of container technology in this regard. Finally, step S4 containerizes the QEMU process and its runtime environment and utilizes Kubernetes to intelligently schedule the computing power of the intelligent computing cloud platform, achieving elastic scaling and load balancing of resources across computing power nodes, breaking through the single-machine resource limitation, and supporting a parallel sampling environment of tens of thousands.

[0084] In summary, through the synergistic effect of the above technologies and based on the computing power of the intelligent computing cloud platform, a high-performance, high-fidelity, and elastically scalable large-scale parallel sampling environment for reinforcement learning has been successfully constructed.

[0085] On the other hand, in the field of reinforcement learning agent training, the method of constructing a large-scale parallel sampling environment for reinforcement learning using the intelligent computing cloud platform described in this application can significantly increase the training efficiency of reinforcement learning agents.

[0086] Figure 2 This invention illustrates a device for constructing a large-scale parallel sampling environment for reinforcement learning using computing power on an intelligent computing cloud platform, according to an embodiment of this application. Figure 2 As shown, device 20 includes: The acquisition module 201 is used to perform step S1: acquire multiple physical disks of multiple computing power nodes of the intelligent computing cloud platform, construct multiple physical disks into an underlying storage pool through RAID technology, and store the base image of the operating system in the underlying storage pool in qcow2 format; Execution module 202 is used to execute step S2: based on the operating system base image, multiple state-isolated virtual machine instances are generated through read-only mounting and copy-on-write mechanism; Step S3: Create a virtualization environment based on the KVM kernel module integrated in the computing node and the hardware virtualization extension capability of the computing node's CPU. In the virtualization environment, use the QEMU process to simulate the hardware device of the computing node to run virtual machine instances. Step S4: Based on container technology, the QEMU process that hosts the virtual machine instance and the runtime environment of the QEMU process are encapsulated into a container. According to the disk topology relationship between multiple underlying storage pools on multiple computing power nodes of the intelligent computing cloud platform and the real-time I / O load, Kubernetes is used to perform distributed scheduling of the containers.

[0087] In one possible implementation, step S1 includes: step S11: obtaining multiple physical disks, wherein the number of physical disks is not less than ten; step S12: constructing an underlying storage pool based on the physical disks using RAID technology; step S13: storing the base image of the operating system, including Ubuntu and / or Windows, in the underlying storage pool in qcow2 format.

[0088] In one possible implementation, step S2 includes: Step S21: Create an independent, initially empty write layer file for each virtual machine instance to be generated; Step S22: Mount the operating system base image in read-only mode and combine it with the write layer file to form a system disk view; Step S23: Start and generate the corresponding virtual machine instance based on the system disk view; wherein, when a data write request occurs during the operation of a virtual machine instance, the data to be modified is copied from the read-only base image to the corresponding write layer file for modification through a copy-on-write mechanism, so as to achieve state isolation between multiple virtual machine instances.

[0089] In one possible implementation, step S3 includes: step S31, loading and running the KVM kernel module in the operating system kernel of the computing node to enable the invocation of the hardware virtualization extension capabilities of the CPU of the computing node; step S32, using the KVM kernel module to put the CPU of the computing node into a mode capable of executing instructions of a virtual machine instance to create a virtualization environment; step S33, starting the QEMU process in the virtualization environment and using the QEMU process to simulate the hardware devices of the computing node; step S34, loading and running a virtual machine instance based on the QEMU process.

[0090] In one possible implementation, the execution module 202 is further configured to perform step S5 after step S3: perform reproducible state management on the virtual machine instance.

[0091] In one possible implementation, step S5 includes: step S51: based on the system disk view corresponding to the virtual machine instance, create a state snapshot of the virtual machine instance at a specific point in time, wherein the state snapshot records the complete state of the base image and write layer files at the specific point in time; step S52: store the state snapshot in the underlying storage pool; step S53: by rolling back the current state of the virtual machine instance to the state snapshot, the sampling environment corresponding to the state snapshot is reproduced.

[0092] In one possible implementation, the execution module 202 is further configured to execute step S6 after step S4: perform full lifecycle management of the virtual machine instance, which includes: power-on management, operation management, and release management; Boot management includes: allocating computing power to virtual machine instances based on the type of operating system corresponding to the virtual machine instance, and starting the virtual machine instance; specifically, for virtual machine instances configured to run the Ubuntu desktop operating system, 2 CPUs and 5GB of memory are allocated; for virtual machine instances configured to run the Windows operating system, 4 CPUs and 8GB of memory are allocated. Operation management includes: performing simulated mouse operations and / or simulated keyboard operations on the virtual machine instance during its operation, and capturing the corresponding screen image of the virtual machine instance; Release management includes shutting down virtual machine instances and reclaiming the computing power used by virtual machine instances.

[0093] In this invention, step S1 fully utilizes the computing power of the intelligent computing cloud platform and employs RAID technology to construct the underlying storage pool, significantly improving the concurrent read I / O performance of the base image and effectively solving the disk bottleneck problem of the traditional qcow2 image in high-concurrency scenarios. Step S2 combines read-only mounting and copy-on-write mechanisms to achieve second-level fast cloning and state isolation of virtual machine instances, ensuring the independence and reproducibility of the environment. Step S3, based on KVM and QEMU's complete hardware virtualization technology, provides a real OS environment with an independent kernel and graphical interface, overcoming the inherent defects of container technology in this regard. Finally, step S4 containerizes the QEMU process and its runtime environment and utilizes Kubernetes to intelligently schedule the computing power of the intelligent computing cloud platform, achieving elastic scaling and load balancing of resources across computing power nodes, breaking through the single-machine resource limitation, and supporting a parallel sampling environment of tens of thousands.

[0094] In summary, through the synergistic effect of the above technologies and based on the computing power of the intelligent computing cloud platform, a high-performance, high-fidelity, and elastically scalable large-scale parallel sampling environment for reinforcement learning has been successfully constructed.

[0095] The technical effects of the method for constructing a large-scale parallel sampling environment for reinforcement learning using computing power in the intelligent computing cloud platform described in this invention are explained in detail below. The large-scale parallel sampling environment for reinforcement learning constructed by this invention, through the deep collaboration of RAID, qcow2, KVM, and Kubernetes (four technologies), produces a significant synergistic effect that transcends the simple superposition of technologies, specifically manifested in the following five aspects: First, it has achieved a qualitative leap in I / O performance. By aggregating disk bandwidth with RAID technology and the read-only sharing mechanism of qcow2 format, the time for tens of thousands of environments to start concurrently has been shortened from minutes to seconds, and the overall disk throughput performance has been improved by 300%, completely solving the I / O bottleneck in high-concurrency scenarios.

[0096] Secondly, resource utilization efficiency is optimized to the extreme. Real-world testing shows that only 2 CPUs and 5GB of memory (2C5G) are needed to run the Ubuntu desktop environment with a graphical interface smoothly; while 4 CPUs and 8GB of memory (4C8G) can effectively support the interaction of Windows operating system and applications. This lightweight configuration allows a single physical server to stably support more than 50 complete environments.

[0097] Furthermore, the complete virtualization technology based on KVM ensures strong isolation and high fidelity of the environment. Each parallel environment is a complete virtual machine with an independent kernel, fundamentally avoiding the state pollution problems that may be caused by container technology sharing the host kernel, and providing a clean and reliable interactive platform for reinforcement learning.

[0098] Furthermore, it possesses seamless horizontal scaling capabilities. By leveraging Kubernetes for cluster-level intelligent scheduling of containers encapsulating the QEMU process, it achieves dynamic resource allocation and load balancing, stably supporting elastic deployment and management of environments with a scale of tens of thousands.

[0099] Ultimately, the combined effects of these advantages significantly improve the stability of reinforcement learning training. The environment reset operation, based on qcow2 snapshot technology, ensures 100% state consistency, guaranteeing the reliability of the training process and the reproducibility of experimental results.

[0100] Please refer to Figure 3 The present invention also provides an electronic device 30, including a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the computer program is executed by the processor 301, it implements the steps of the above-mentioned method for constructing a large-scale parallel sampling environment for reinforcement learning by the intelligent computing cloud platform through computing power, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0101] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps of the aforementioned method for constructing a large-scale parallel sampling environment for reinforcement learning using computing power in an intelligent computing cloud platform, achieving the same technical effect. To avoid repetition, this will not be elaborated further. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0102] The present invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the above-mentioned method for constructing a large-scale parallel sampling environment for reinforcement learning by the intelligent computing cloud platform through computing power, and can achieve the same technical effect. To avoid repetition, it will not be described in detail here.

[0103] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0104] Through the above description of the embodiments, those skilled in the art can clearly understand that the above methods can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the present invention.

[0105] The present invention has been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other modifications under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these modifications are within the protection scope of the present invention.

Claims

1. A method for constructing a large-scale parallel sampling environment for reinforcement learning using computing power in an intelligent computing cloud platform, characterized in that, The method includes: Step S1: Obtain multiple physical disks from multiple computing nodes of the intelligent computing cloud platform, construct the multiple physical disks into an underlying storage pool using RAID technology, and store the base image of the operating system in the underlying storage pool in qcow2 format; Step S2: Based on the base image of the operating system, generate multiple state-isolated virtual machine instances through read-only mounting and copy-on-write mechanisms; Step S3: Create a virtualization environment based on the KVM kernel module integrated in the computing node and the hardware virtualization extension capability of the CPU of the computing node, and use the QEMU process to simulate the hardware device of the computing node in the virtualization environment to run the virtual machine instance. Step S4: Based on container technology, the QEMU process that hosts the virtual machine instance and the runtime environment of the QEMU process are encapsulated into a container. According to the disk topology relationship between multiple underlying storage pools on multiple computing power nodes of the intelligent computing cloud platform and the real-time I / O load, the container is distributed and scheduled using Kubernetes.

2. The method according to claim 1, characterized in that, Step S1 includes: Step S11: Obtain multiple physical disks, wherein the number of physical disks is no less than ten; Step S12: Based on the physical disk, construct the underlying storage pool using RAID technology; Step S13: Store the base image of the operating system, including Ubuntu and / or Windows, in the underlying storage pool in qcow2 format.

3. The method according to claim 1, characterized in that, Step S2 includes: Step S21: Create a separate, initially empty write layer file for each virtual machine instance to be generated; Step S22: Mount the base image of the operating system in read-only mode and combine it with the write layer file to form a system disk view; Step S23: Based on the system disk view, start and generate the corresponding virtual machine instance; wherein, when the virtual machine instance has a data write request during operation, the data to be modified is copied from the read-only base image to the corresponding write layer file for modification through the copy-on-write mechanism, so as to achieve state isolation between multiple virtual machine instances.

4. The method according to claim 1, characterized in that, Step S3 includes: Step S31: Load and run the KVM kernel module in the operating system kernel of the computing node to enable the invocation of the hardware virtualization extension capability of the CPU of the computing node; Step S32: Using the KVM kernel module, the CPU of the computing node is placed in a mode that enables it to execute instructions of the virtual machine instance, thereby creating the virtualization environment; Step S33: In the virtualization environment, start the QEMU process and use the QEMU process to simulate the hardware device of the computing node; Step S34: Load and run the virtual machine instance based on the QEMU process.

5. The method according to claim 3, characterized in that, After step S4, the method further includes: Step S5: Perform reproducible state management on the virtual machine instance.

6. The method according to claim 5, characterized in that, Step S5 includes: Step S51: Based on the system disk view corresponding to the virtual machine instance, create a state snapshot of the virtual machine instance at a specific point in time, wherein the state snapshot records the complete state of the base image and the write layer file at the specific point in time; Step S52: Store the state snapshot in the underlying storage pool; Step S53: By rolling back the current state of the virtual machine instance to the state snapshot, the sampling environment corresponding to the state snapshot is reproduced.

7. The method according to claim 3, characterized in that, After step S4, the method further includes: Step S6: Perform full lifecycle management on the virtual machine instance, which includes: power-on management, operation management, and release management; The boot management includes: allocating computing power to the virtual machine instance according to the type of operating system corresponding to the virtual machine instance, and starting the virtual machine instance; wherein, for a virtual machine instance configured to run the Ubuntu desktop operating system, 2 CPUs and 5GB of memory are allocated; for a virtual machine instance configured to run the Windows operating system, 4 CPUs and 8GB of memory are allocated. The operation management includes: performing simulated mouse operations and / or simulated keyboard operations on the virtual machine instance during its operation, and capturing the screen image corresponding to the virtual machine instance. The release management includes: shutting down the virtual machine instance and reclaiming the computing power occupied by the virtual machine instance.

8. A device for constructing a large-scale parallel sampling environment for reinforcement learning using computing power in an intelligent computing cloud platform, characterized in that, The device includes: The acquisition module is used to execute step S1: acquire multiple physical disks of multiple computing nodes of the intelligent computing cloud platform, construct the multiple physical disks into an underlying storage pool through RAID technology, and store the base image of the operating system in the underlying storage pool in qcow2 format; The execution module is used to execute step S2: based on the base image of the operating system, multiple state-isolated virtual machine instances are generated through read-only mounting and copy-on-write mechanism; Step S3: Create a virtualization environment based on the KVM kernel module integrated in the computing node and the hardware virtualization extension capability of the CPU of the computing node, and use the QEMU process to simulate the hardware device of the computing node in the virtualization environment to run the virtual machine instance. Step S4: Based on container technology, the QEMU process that hosts the virtual machine instance and the runtime environment of the QEMU process are encapsulated into a container. According to the disk topology relationship between multiple underlying storage pools on multiple computing power nodes of the intelligent computing cloud platform and the real-time I / O load, the container is distributed and scheduled using Kubernetes.

9. A server, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of a method for constructing a large-scale parallel sampling environment for reinforcement learning via computing power on an intelligent computing cloud platform as described in any one of claims 1-7 are implemented.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a method for constructing a large-scale parallel sampling environment for reinforcement learning using computing power on an intelligent computing cloud platform as described in any one of claims 1-7.

11. A computer program product, characterized in that, It includes computer instructions, which, when executed by a processor, implement the steps of a method for constructing a large-scale parallel sampling environment for reinforcement learning via computing power on an intelligent computing cloud platform as described in any one of claims 1-7.

Images