Material structure stream generation and adaptive neural network asynchronous non-blocking routing system based on actor elastic scheduling

Abstract

The application provides a material structure stream generation and adaptive neural network asynchronous non-blocking routing system based on actor elastic scheduling, belongs to the technical field of computer and material engineering integration, and comprises a feedback learning and optimization control unit, a stream generation and intelligent packaging unit, a zero-copy dynamic routing unit, an elastic execution and adaptive arrangement unit and a storage and memory management unit. The application breaks the serial barrier of structure generation and model reasoning by introducing a pipeline parallel mechanism, establishes a stream generation full-asynchronous link of generation-distribution-reasoning, realizes linear growth of throughput, constructs an extremely fast IO channel by using a zero-copy transmission technology in view of the fragmentation characteristics of material structure files, and realizes direct transmission in a kernel state in combination with NIO multiplexing.

Description

Technical Field

[0001] This invention relates to the field of computer and materials engineering integration technology, and in particular to a material structure streaming generation and an adaptive neural network asynchronous non-blocking routing system based on Actor elastic scheduling. Background Technology

[0002] With the advancement of the Materials Genome Initiative, high-throughput computing screening has become a core method for discovering new materials. However, the existing technology system has the following fundamental defects: (1) Rigid pipeline bottleneck: The traditional generation-storage-reading-computation process forms a synchronous blockage. The structure generator must wait for the previous batch of calculations to be completed before it can continue, resulting in periodic idleness of high-performance computing resources and low overall efficiency. (2) Small file metadata disaster: Each material configuration is usually saved as a 1-10KB CIF or POSCAR file. When the processing scale reaches millions or even hundreds of millions, the file system needs to manage a massive number of inodes, and directory lookup and metadata updates become the main performance bottlenecks, and I / O throughput drops sharply. (3) Heterogeneous resource scheduling mismatch: Different neural network models (such as models used to predict band gap, elastic modulus, and stability) have huge differences in their demand for computing resources. Static task allocation strategies cannot be dynamically adjusted according to real-time load, resulting in uneven GPU utilization, with some nodes overloaded and others idle. (4) Lack of closed-loop intelligence: Current systems mostly use open-loop computation, and the screening results cannot be fed back in real time to guide the next round of structure generation. They cannot achieve proactive exploration based on prediction results, resulting in low search efficiency. Existing methods, such as MPI-based batch processing tasks, simple message queues (such as RabbitMQ), or static DAG schedulers, cannot solve the above problems simultaneously, especially in dealing with continuous computation tasks on the scale of hundreds of millions that last for several days or even weeks. Therefore, it is necessary to design a streaming generation system for millions of material structures and an adaptive neural network asynchronous non-blocking routing system based on Actor elastic scheduling. Summary of the Invention

[0003] The purpose of this invention is to provide a material structure streaming generation and an adaptive neural network asynchronous non-blocking routing system based on Actor elastic scheduling, which solves the problem of coordination between computationally intensive tasks and data-intensive pipelines in the exploration of billions of material spaces.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0005] The material structure streaming generation and Actor-based adaptive neural network asynchronous non-blocking routing system includes a feedback learning and optimization control unit, a streaming generation and intelligent encapsulation unit, a zero-copy dynamic routing unit, an elastic execution and adaptive orchestration unit, and a storage and memory management unit. The feedback learning and optimization control unit is connected to both the streaming generation and intelligent encapsulation unit and the storage and memory management unit. The feedback learning and optimization control unit is used for training and optimization, and for analyzing and identifying feedback results. The streaming generation and intelligent encapsulation unit is connected to the zero-copy dynamic routing unit. This unit is used to compute stream abstraction and memory-resident data packets, and reports the generation rate, the currently explored material space region, and the estimated value of the generated structure in real time through the generator cluster, followed by encapsulation. The zero-copy dynamic routing unit is connected to the elastic execution and adaptive orchestration unit. The zero-copy dynamic routing unit is used to establish a partitioned circular buffer and distribute tasks through a multi-level routing decision engine. The elastic execution and adaptive orchestration unit identifies in real time through a predictor model and automatically starts the corresponding model. The storage and memory management unit stores data and feeds back the identification results.

[0006] Furthermore, the feedback learning and optimization control unit includes a feedback controller module, several objective Bayesian optimization modules, an active learning strategy module, a central coordinator module, a real-time result analysis pipeline module, a multi-dimensional performance space clustering module, and a potential material island identification module. The feedback controller module, through the multi-objective Bayesian optimization module, the active learning strategy module, and the central coordinator module, analyzes the characteristic patterns of potential materials, dynamically adjusts generator parameters, increases the sampling density of potential regions, reduces the exploration of barren regions, and introduces a mutation strategy to escape local optima. The several objective Bayesian optimization modules are used for material space exploration optimization, automatically identify performance bottleneck links, dynamically adjust pipeline parallelism, and when material characterization is found to be a bottleneck, automatically allocate more resources to that link while reducing the generation rate.

[0007] The real-time results analysis pipeline module is connected to the potential material island identification module via a multi-dimensional performance space clustering module. It stream processes the inference results, extracts key indicators, establishes a multi-dimensional material performance space, and identifies potential material islands in real time through clustering.

[0008] Furthermore, the streaming generation and intelligent encapsulation unit includes a structural fragment intelligent caching module, an optimization generator, a random generator, an enumeration generator, and an intelligent encapsulation module. The structural fragment intelligent caching module is connected to the optimization generator, the random generator, and the enumeration generator, respectively. The optimization generator, the random generator, and the enumeration generator are all connected to the intelligent encapsulation module. The structural fragment intelligent caching module is used to design MaterialStream objects, encapsulate each material configuration into a self-contained binary data packet, pre-generate and cache frequently occurring structural fragments according to the material space exploration path, and generate new structures by assembling fragments, thereby improving speed. The optimization generator, the random generator, and the enumeration generator report their output rate, the currently explored material space region, and the estimated value of the generated structure in real time. The intelligent encapsulation module dynamically adjusts the output rate and target space of each generator according to the load of the downstream inference cluster to avoid generating useless structures that cause downstream congestion.

[0009] Furthermore, the zero-copy dynamic routing unit includes a partitioned ring memory pool module, several levels of routing decision engine modules, a first-level model type routing module, a second-level load-aware routing module, a third-level priority scheduling module, and several scalper algorithm modules. The partitioned ring memory pool module establishes a partitioned ring buffer in shared memory or RDMA network memory, dividing the memory pool into N equal-length sub-partitions. Each partition can hold one CrystalPacket. The router prefetches data packets to be distributed into the CPU cache based on historical access patterns to reduce cache misses. The first-level model type routing quickly hashes the data to the corresponding neural network model category based on the material composition. The second-level load-aware routing obtains the load status of all inference execution nodes in real time. The third-level priority scheduling module prioritizes the allocation of high-priority tasks, sets a soft deadline, and automatically degrades or reroutes timed-out tasks.

[0010] Furthermore, the elastic execution and adaptive orchestration unit includes an orchestrator, several predictor models, a resource autoscalorie module, a Raft consensus algorithm module, a cold backup model startup module, and a low-load sleep model. The orchestrator is connected to several predictor models, the resource autoscalorie module, and the Raft consensus algorithm module. The orchestrator maintains a global resource graph, including network bandwidth, latency, and NUMA topology between nodes. The resource autoscalorie module automatically starts the cold backup Actor for a model category when the task queue length is greater than the threshold Q_max. When GPU utilization is less than 20% for T minutes, it automatically puts some Actors into sleep to save energy. The Raft consensus algorithm module manages the Actor state using the Raft consensus algorithm. Faulty node tasks are automatically migrated, and checkpoints are automatically saved every 100 tasks.

[0011] Furthermore, the predictor model includes a mailbox SPMC queue module, a model manager module, a dynamic batch processing engine module, and a performance monitor module. The mailbox SPMC queue module is used for task reception and asynchronous buffering. The upper-layer routing layer acts as a single producer, asynchronously pushing task descriptors that need to be inferred to it. The internal dynamic batch processing engine acts as a consumer, pulling tasks from it, decoupling task production and consumption, effectively handling traffic spikes, and avoiding task loss. It is a key foundation for ensuring high concurrency and low latency of the system. The model manager module is used to load and manage the neural network models required for prediction. The dynamic batch processing engine module is used for computational performance optimization and maximizing resource utilization. The performance monitor module is used for runtime status awareness and health reporting.

[0012] Furthermore, the storage and memory management unit includes a partitioned ring memory pool module, a partitioning module, and a distributed result storage module. The partitioned ring memory pool module is used for memory partitioning, the partitioning module is used for storing data according to the partitioned areas, and the distributed result storage module is used as a permanent repository for prediction results and a data source for analysis.

[0013] The present invention, by adopting the above-described technical solution, has the following beneficial effects:

[0014] This invention breaks down the serial barrier between structure generation and model inference by introducing a pipelined parallel mechanism, establishing a fully asynchronous streaming generation link of generation-distribution-inference to achieve linear growth in throughput. Addressing the fragmented nature of material structure files, it utilizes zero-copy transmission technology to construct an ultra-fast I / O channel, combined with NIO multiplexing to achieve direct kernel-mode transmission, reducing the reading latency of millions of KB-level small files to microseconds and completely eliminating context switching overhead. It also constructs an adaptive orchestration engine based on the Actor collaborative model, employing a lock-free data exchange strategy to achieve elastic scheduling and fault self-healing of heterogeneous computing resources, enabling real-time generation of millions of material structures and millisecond-level neural network routing responses, significantly improving the efficiency of new material discovery. Attached Figure Description

[0015] Figure 1 This is a system block diagram of the present invention;

[0016] Figure 2 This is a block diagram of the feedback learning and optimization control unit module of the present invention;

[0017] Figure 3 This is a block diagram of the flow generation and intelligent packaging unit module of the present invention;

[0018] Figure 4 This is a block diagram of the zero-copy dynamic routing unit module of the present invention;

[0019] Figure 5 This is a block diagram of the flexible execution and adaptive orchestration unit module of the present invention;

[0020] Figure 6 This is a block diagram of the storage and memory management unit module of the present invention;

[0021] Figure 7 This is an internal block diagram of the predictor model of this invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and preferred embodiments. However, it should be noted that many details listed in the specification are merely to provide the reader with a thorough understanding of one or more aspects of the present invention, and these aspects of the invention can be implemented even without these specific details.

[0023] like Figure 1 As shown, the material structure streaming generation and Actor-based adaptive neural network asynchronous non-blocking routing system includes a feedback learning and optimization control unit, a streaming generation and intelligent encapsulation unit, a zero-copy dynamic routing unit, an elastic execution and adaptive orchestration unit, and a storage and memory management unit. The feedback learning and optimization control unit is connected to both the streaming generation and intelligent encapsulation unit and the storage and memory management unit. The feedback learning and optimization control unit is used for training and optimization, and for analyzing and identifying the feedback results. The streaming generation and intelligent encapsulation unit is connected to the zero-copy dynamic routing unit. This unit is used to compute stream abstraction and memory-resident data packets, and reports the generation rate, the currently explored material space region, and the estimated value of the generated structure in real time through the generator cluster, followed by encapsulation. The zero-copy dynamic routing unit is connected to the elastic execution and adaptive orchestration unit. The zero-copy dynamic routing unit is used to establish a partitioned circular buffer and distribute tasks through a multi-level routing decision engine. The elastic execution and adaptive orchestration unit identifies the model in real time through a predictor model and automatically starts the corresponding model. The storage and memory management unit stores the data and feeds back the identification results.

[0024] In embodiments of the present invention, such as Figure 2As shown, the feedback learning and optimization control unit includes a feedback controller module, several objective Bayesian optimization modules, an active learning strategy module, a central coordinator module, a real-time result analysis pipeline module, a multi-dimensional performance space clustering module, and a potential material island identification module. The feedback controller module, through the multi-objective Bayesian optimization module, the active learning strategy module, and the central coordinator module, analyzes the characteristic patterns of potential materials, dynamically adjusts generator parameters, increases the sampling density of potential regions, reduces the exploration of barren regions, and introduces a mutation strategy to escape local optima. The several objective Bayesian optimization modules are used for material space exploration optimization, automatically identify performance bottlenecks, dynamically adjust pipeline parallelism, and when material characterization is found to be a bottleneck, automatically allocate more resources to that stage while reducing the generation rate.

[0025] The real-time results analysis pipeline module is connected to the potential material island identification module via a multi-dimensional performance space clustering module. It stream processes the inference results, extracts key indicators, establishes a multi-dimensional material performance space, and identifies potential material islands in real time through clustering.

[0026] In embodiments of the present invention, such as Figure 3 As shown, the streaming generation and intelligent encapsulation unit includes a structural fragment intelligent caching module, an optimization generator, a random generator, an enumeration generator, and an intelligent encapsulation module. The structural fragment intelligent caching module is connected to the optimization generator, the random generator, and the enumeration generator, respectively. The optimization generator, the random generator, and the enumeration generator are all connected to the intelligent encapsulation module. The structural fragment intelligent caching module is used to design MaterialStream objects, encapsulate each material configuration into a self-contained binary data packet, pre-generate and cache frequently occurring structural fragments according to the material space exploration path, and generate new structures by assembling fragments, thereby improving speed. The optimization generator, the random generator, and the enumeration generator report their output rate, the currently explored material space region, and the estimated value of the generated structure in real time. The intelligent encapsulation module dynamically adjusts the output rate and target space of each generator according to the load of the downstream inference cluster to avoid generating useless structures that cause downstream congestion.

[0027] A computational flow abstraction is introduced. Instead of outputting separate files, the generator encapsulates each material structure and its metadata into a self-describing, memory-contiguous StructurePacket data packet. This packet contains serialized data such as structure identifiers, atomic coordinates, and lattice vectors, and its size is comparable to the original file, but eliminates all file system overhead.

[0028] In embodiments of the present invention, such as Figure 4As shown, the zero-copy dynamic routing unit includes a partitioned ring memory pool module, several levels of routing decision engine modules, a first-level model type routing module, a second-level load-aware routing module, a third-level priority scheduling module, and several scalper algorithm modules. The partitioned ring memory pool module establishes a partitioned ring buffer in shared memory or RDMA network memory, dividing the memory pool into N equal-length sub-partitions. Each partition can hold one CrystalPacket. The router prefetches data packets to be distributed into the CPU cache based on historical access patterns to reduce cache misses. The first-level model type routing quickly hashes the data to the corresponding neural network model category based on the material composition. The second-level load-aware routing obtains the load status of all inference execution nodes in real time. The third-level priority scheduling module prioritizes the allocation of high-priority tasks, sets a soft deadline, and automatically degrades or reroutes timed-out tasks.

[0029] In networks that support shared memory or remote direct memory access, memory channels establish pre-allocated memory pools. Once a StructurePacket is generated, it is directly placed into a designated location within the memory pool, enabling zero-copy writes by the producer.

[0030] The generator does not pass the data itself, but instead passes a Descriptor (containing address, length, and status) pointing to the memory location of the StructurePacket to the routing system.

[0031] NIO multiplexing routing: The routing hub uses an event-driven model to listen for descriptors emitted by multiple generators and asynchronously, non-blockingly dispatches them to downstream idle inference executors. The executor directly reads memory data through the descriptor, achieving zero-copy reading for consumers. Throughout the process, data remains in a fixed location in user space or kernel space, eliminating the need for copying between buffers.

[0032] In embodiments of the present invention, such as Figure 5 As shown, the elastic execution and adaptive orchestration unit includes an orchestrator, several predictor models, a resource autoscalor module, a Raft consensus algorithm module, a cold backup model startup module, and a low-load sleep model. The orchestrator is connected to several predictor models, the resource autoscalor module, and the Raft consensus algorithm module. The orchestrator maintains a global resource graph, including network bandwidth, latency, and NUMA topology between nodes. The resource autoscalor module automatically starts the cold backup Actor for a model category when the task queue length is greater than the threshold Q_max. When GPU utilization is less than 20% for T minutes, it automatically puts some Actors into sleep to save energy. The Raft consensus algorithm module manages the Actor state using the Raft consensus algorithm. Faulty node tasks are automatically migrated, and checkpoints are automatically saved every 100 tasks.

[0033] The intelligent Actor design treats each actor as an independent PredictorActor, encapsulating a computational unit and loading a specific neural network model (different models for bandgap prediction and stability classification). Each actor records its own health status (normal / busy / failed), current load, model type, performance profile, and average inference time. It receives Descriptor tasks from routing channels via its mailbox. Collaborative orchestration and elastic scheduling, along with heartbeats and registration, ensure that all PredictorActors actively register their capabilities and status with the orchestrator. An adaptive routing strategy dynamically selects the optimal, idle PredictorActor based on global load and task characteristics, pushing the Descriptor to its mailbox. Strategies can be based on minimum queuing time, minimum expected completion time, etc. Fault self-healing: If an Actor becomes unreachable or times out during inference, the orchestrator automatically marks it as faulty and reroutes the task to other Actors, generating an alarm. Lock-free data exchange: Task transfer between Actors is accomplished through Descriptors. This process is atomic, requiring no global locks, ensuring system scalability under high concurrency.

[0034] In embodiments of the present invention, such as Figure 7 As shown, the predictor model includes a mailbox SPMC queue module, a model manager module, a dynamic batch processing engine module, and a performance monitor module. The mailbox SPMC queue module is used for task reception and asynchronous buffering. The upper routing layer acts as a single producer, asynchronously pushing task descriptors that need to be inferred to it. The internal dynamic batch processing engine acts as a consumer, pulling tasks from it, decoupling task production and consumption, effectively handling traffic spikes, and avoiding task loss. It is a key foundation for ensuring high concurrency and low latency of the system. The model manager module is used to load and manage the neural network models required for prediction. The dynamic batch processing engine module is used for computational performance optimization and maximizing resource utilization. The performance monitor module is used for runtime status awareness and health reporting.

[0035] In embodiments of the present invention, such as Figure 6 As shown, the storage and memory management unit includes a partitioned ring memory pool module, a partitioning module, and a distributed result storage module. The partitioned ring memory pool module is used for memory partitioning, the partitioning module is used for storing data according to the partitioned areas, and the distributed result storage module is used as a permanent repository for prediction results and a data source for analysis.

[0036] Matters not covered in this invention are common knowledge.

[0037] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A material structure streaming generation and adaptive neural network asynchronous non-blocking routing system based on Actor elastic scheduling, characterized in that: It includes a feedback learning and optimization control unit, a streaming generation and intelligent encapsulation unit, a zero-copy dynamic routing unit, a flexible execution and adaptive orchestration unit, and a storage and memory management unit. The feedback learning and optimization control unit is connected to the streaming generation and intelligent encapsulation unit and the storage and memory management unit, respectively. The feedback learning and optimization control unit is used for training and optimization, and for analyzing and identifying feedback results. The streaming generation and intelligent encapsulation unit is connected to the zero-copy dynamic routing unit. The streaming generation and intelligent encapsulation unit is used to calculate the stream abstraction and memory-resident data packets, and reports its output rate, the currently explored material space region, and the estimated value of the output structure in real time through the generator cluster, and then encapsulates them. The zero-copy dynamic routing unit is connected to the flexible execution and adaptive orchestration unit. The zero-copy dynamic routing unit is used to establish a partitioned circular buffer and distribute tasks through several levels of routing decision engines. The flexible execution and adaptive orchestration unit identifies in real time through the predictor model and automatically starts the corresponding model. The storage and memory management unit is used to store data and feed back the identification results.

2. The material structure streaming generation and adaptive neural network asynchronous non-blocking routing system based on Actor elastic scheduling according to claim 1, characterized in that: The feedback learning and optimization control unit includes a feedback controller module, several objective Bayesian optimization modules, an active learning strategy module, a central coordinator module, a real-time result analysis pipeline module, a multi-dimensional performance space clustering module, and a potential material island identification module. The feedback controller module, through the multi-objective Bayesian optimization module, the active learning strategy module, and the central coordinator module, analyzes the characteristic patterns of potential materials, dynamically adjusts generator parameters, increases the sampling density of potential regions, reduces the exploration of barren regions, and introduces a mutation strategy to escape local optima. The several objective Bayesian optimization modules are used for material space exploration optimization, automatically identify performance bottlenecks, dynamically adjust pipeline parallelism, and when material characterization is found to be a bottleneck, automatically allocate more resources to that stage while reducing the generation rate. The real-time results analysis pipeline module is connected to the potential material island identification module via a multi-dimensional performance space clustering module. It stream processes the inference results, extracts key indicators, establishes a multi-dimensional material performance space, and identifies potential material islands in real time through clustering.

3. The material structure streaming generation and adaptive neural network asynchronous non-blocking routing system based on Actor elastic scheduling according to claim 1, characterized in that: The streaming generation and intelligent encapsulation unit includes a structure fragment intelligent caching module, an optimization generator, a random generator, an enumeration generator, and an intelligent encapsulation module. The structure fragment intelligent caching module is connected to the optimization generator, the random generator, and the enumeration generator, respectively. The optimization generator, the random generator, and the enumeration generator are all connected to the intelligent encapsulation module. The structure fragment intelligent caching module is used to design MaterialStream objects, encapsulate each material configuration into a self-contained binary data packet, pre-generate and cache frequently occurring structure fragments according to the material space exploration path, and generate new structures by assembling fragments, thereby improving speed. The optimization generator, the random generator, and the enumeration generator report their output rate, the currently explored material space region, and the estimated value of the generated structure in real time. The intelligent encapsulation module dynamically adjusts the output rate and target space of each generator according to the load of the downstream inference cluster to avoid generating useless structures that cause downstream congestion.

4. The material structure streaming generation and adaptive neural network asynchronous non-blocking routing system based on Actor elastic scheduling according to claim 1, characterized in that: The zero-copy dynamic routing unit includes a partitioned ring memory pool module, several levels of routing decision engine modules, a first-level model type routing module, a second-level load-aware routing module, a third-level priority scheduling module, and several scalper algorithm modules. The partitioned ring memory pool module establishes a partitioned ring buffer in shared memory or RDMA network memory, dividing the memory pool into N equal-length sub-partitions. Each partition can hold one CrystalPacket. The router prefetches the data packets to be distributed into the CPU cache based on historical access patterns to reduce cache misses. The first-level model type routing quickly hashes the data to the corresponding neural network model category based on the material composition. The second-level load-aware routing obtains the load status of all inference execution nodes in real time. The third-level priority scheduling module prioritizes the allocation of high-priority tasks, sets a soft deadline, and automatically degrades or reroutes timed-out tasks.

5. The material structure streaming generation and adaptive neural network asynchronous non-blocking routing system based on Actor elastic scheduling according to claim 1, characterized in that: The elastic execution and adaptive orchestration unit includes an orchestrator, several predictor models, a resource autoscalorie module, a Raft consensus algorithm module, a cold backup model startup module, and a low-load sleep model. The orchestrator connects to the predictor models, the resource autoscalorie module, and the Raft consensus algorithm module. The orchestrator maintains a global resource graph, including network bandwidth, latency, and NUMA topology between nodes. The resource autoscalorie module automatically starts a cold backup Actor for a model category when the task queue length exceeds the threshold Q_max. When GPU utilization is less than 20% for T minutes, it automatically puts some Actors into sleep to save energy. The Raft consensus algorithm module manages Actor states using the Raft consensus algorithm, automatically migrates tasks from failed nodes, and automatically saves checkpoints every 100 tasks.

6. The material structure streaming generation and adaptive neural network asynchronous non-blocking routing system based on Actor elastic scheduling according to claim 5, characterized in that: The predictor model includes a mailbox SPMC queue module, a model manager module, a dynamic batch processing engine module, and a performance monitor module. The mailbox SPMC queue module is used for task reception and asynchronous buffering. The upper-layer routing layer acts as a single producer, asynchronously pushing task descriptors that need to be inferred to it. The internal dynamic batch processing engine acts as a consumer, pulling tasks from it, decoupling task production and consumption, effectively handling traffic spikes, and avoiding task loss. It is a key foundation for ensuring high concurrency and low latency of the system. The model manager module is used to load and manage the neural network models required for prediction. The dynamic batch processing engine module is used for computational performance optimization and maximizing resource utilization. The performance monitor module is used for runtime status awareness and health reporting.

7. The material structure streaming generation and adaptive neural network asynchronous non-blocking routing system based on Actor elastic scheduling according to claim 1, characterized in that: The storage and memory management unit includes a partitioned ring memory pool module, a partitioning module, and a distributed result storage module. The partitioned ring memory pool module is used for memory partitioning, the partitioning module is used for storing data according to the partitioned areas, and the distributed result storage module is used as a permanent repository for prediction results and a data source for analysis.

Images