New energy power generation prediction parallel communication efficiency optimization method, system and device based on supercomputing and medium

Abstract

The invention relates to the technical field of high-performance calculation and new energy power prediction, in particular to a new energy power generation prediction parallel communication efficiency optimization method, system and device based on supercomputing, and a medium. Computing nodes of a supercomputer are divided into an assembly line layer, a computing island layer and a summary scheduling layer; based on a pipeline layer, estimating time consumption of the data blocks by adopting a machine learning model, and dynamically adjusting transmission parameters of the data blocks according to the estimated maximum time consumption so as to synchronize calculation and communication; on the basis of the pipeline layer after calculation and communication synchronization, generating original prediction data of the main node to the sub-region in the calculation island layer, and compressing the original prediction data; and the compressed prediction data is sent to the summarizing scheduling layer, the summarizing scheduling layer carries out processing according to the type of the received prediction data and carries out intelligent decision making by using a resource scheduler, and rapid convergence and intelligent scheduling of the data are realized through multi-mode data synchronization optimization data transmission.

Description

Technical Field

[0001] This invention relates to the fields of high-performance computing and new energy power prediction technology, and in particular to a method, system, device and medium for optimizing parallel communication efficiency in new energy power generation prediction based on supercomputing. Background Technology

[0002] Currently, when performing large-scale renewable energy power generation forecasting on supercomputing platforms, a master-slave parallel architecture based on the Message Passing Interface (MPI) is typically adopted. This architecture is the most common and fundamental parallel computing paradigm.

[0003] Traditional methods typically employ a centralized master-slave architecture. There is usually only one master process, responsible for controlling and coordinating all processes, including task decomposition, data broadcasting, and receiving and integrating the results from all slave processes. It serves as the scheduling center and communication hub for the entire operation. Multiple slave processes exist, such as in a forecasting task within a wind farm or solar power plant. These slave processes receive public data broadcast by the master process, perform local calculations, and then send the complete raw calculation results back to the master process.

[0004] Traditional architectures suffer from several drawbacks. First, severe synchronization waits and communication bottlenecks exist. The main process must wait for all slave processes to complete their computations and return results before it can begin final result integration. As shown in the diagram, due to the varying computational complexity of different sub-regions (Slave3 computes faster, SlaveN computes slower), faster processes must wait for slower processes (T_wait), resulting in significant idle computing resources. Simultaneously, all slave processes sending large amounts of result data to the main process at the same time causes network bandwidth contention and communication congestion, greatly increasing the overall job time. Second, static task allocation leads to unbalanced loads. Tasks are typically assigned by the main process at the start of the job. Since the actual computational load of each subtask cannot be known in advance, nor can it be dynamically adjusted based on the running status, it easily leads to some processes being overloaded while others are idle and waiting, reducing overall efficiency. Finally, redundant data communication occurs. The main process sends complete raw data to all slave processes, but each slave process only needs its relevant portion of the data. Furthermore, slave processes often return raw results containing many details to the main process, rather than compressed or simplified data, resulting in wasted network bandwidth.

[0005] The core problem with existing methods is centralized control and synchronous communication. The main process becomes the single bottleneck for performance and reliability, and global synchronization operations and a large amount of redundant data transmission lead to poor parallel efficiency and scalability. As the scale of computation and the complexity of the problem increase, these problems become more severe, making it difficult to effectively utilize the massive computing resources of supercomputers. Summary of the Invention

[0006] In view of the aforementioned existing problems, the present invention is proposed.

[0007] Therefore, this invention provides a method, system, device, and medium for optimizing parallel communication efficiency in supercomputing-based new energy power generation prediction. This addresses the issue that the main process becomes the single bottleneck for performance and reliability, and that global synchronization operations and large amounts of redundant data transmission lead to poor parallel efficiency and scalability. These problems become more severe as the scale of computation and the complexity of the problem increase, making it difficult to effectively utilize the large-scale computing resources of supercomputing.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a method for optimizing the parallel communication efficiency of new energy power generation prediction based on supercomputing, comprising: The computing nodes of a supercomputer are divided into a pipeline layer, a computing island layer, and a summary scheduling layer. Based on the pipeline layer, a machine learning model is used to estimate the data block latency, and the transmission parameters of the data block are dynamically adjusted according to the estimated maximum latency to synchronize computation and communication. Based on the pipeline layer after the synchronization of computation and communication, the master node generates the original prediction data for the sub-region within the computation island layer, and compresses the original prediction data. The compressed forecast data is sent to the aggregation scheduling layer, which processes the received forecast data type and uses the resource scheduler to make intelligent decisions, optimizing data transmission through multi-mode data synchronization.

[0009] As a preferred embodiment of the parallel communication efficiency optimization method for predicting new energy power generation based on supercomputing described in this invention, the method includes: using a machine learning model to estimate the data block latency based on a pipeline layer, and dynamically adjusting the data block transmission parameters according to the estimated maximum latency to synchronize computation and communication, including: Based on the pipeline layer, feature vectors of data blocks are extracted in real time. Based on the extracted feature vectors, a machine learning model is used to estimate the time taken to process the data blocks. The theoretical bottleneck duration is determined based on the estimated maximum latency, and an adjustment strategy for data block transmission parameters is selected based on the theoretical bottleneck duration. Based on the selected data block transmission parameter adjustment strategy, the data block transmission parameters are dynamically adjusted to synchronize computation and communication.

[0010] As a preferred embodiment of the parallel communication efficiency optimization method for new energy power generation prediction based on supercomputing described in this invention, the method includes: based on the pipeline layer after computation and communication synchronization, within the computation island layer, generating the original prediction data of the master node for the sub-region, and compressing the original prediction data, including: Based on the pipeline layer after the synchronization of computation and communication, the original prediction data of the master node for the sub-region is generated within the computation island layer. The selection of compression strategies for raw prediction data is based on the actual application scenario. Compression strategies include feature extraction and compression through feature extraction models reconstructed from data. Data compression is performed based on the selected compression strategy for the original forecast data.

[0011] As a preferred embodiment of the parallel communication efficiency optimization method for new energy power generation prediction based on supercomputing described in this invention, the compressed prediction data is sent to the aggregation scheduling layer. The aggregation scheduling layer processes the received prediction data type and makes intelligent decisions using a resource scheduler. Data transmission is optimized through multi-mode data synchronization, including: The compressed prediction data is sent to the aggregation and scheduling layer via asynchronous communication, and a handle is returned immediately after the data is sent. The master node saves the returned handle to the pending request pool and immediately processes the next computation task. The pending request pool continuously checks the request status and determines whether to release related resources based on the request status; The aggregation scheduling layer processes the received predicted data types and uses the resource scheduler for intelligent decision-making. Optimize data transmission through multi-mode data synchronization.

[0012] As a preferred embodiment of the parallel communication efficiency optimization method for new energy power generation prediction based on supercomputing described in this invention, the method includes: extracting feature vectors of data blocks in real time based on a pipeline layer; and estimating the data block latency using a machine learning model based on the extracted feature vectors, including: When data blocks About to enter the stage At that time, the scheduler built into the pipeline layer will extract data blocks in real time. eigenvectors ; The prediction is performed using a linear regression model, and the mathematical expression is: in, For the predicted data blocks In the stage The computation time, For the stage Parameters of a model trained separately. , and As weight, For bias terms, For the size of the data block, The estimated complexity of the data block. This is a data block type.

[0013] The beneficial effects of this preferred technical solution are as follows: By introducing a time-consuming prediction mechanism based on feature vectors, real-time evaluation and dynamic scheduling of the computational load of different data blocks are realized. By extracting features such as the size, complexity, and type of data blocks, and using a linear regression model to predict the time consumption at each computation stage, resource regulation and load balancing can be achieved before task allocation.

[0014] As a preferred embodiment of the parallel communication efficiency optimization method for new energy power generation prediction based on supercomputing described in this invention, the method includes: determining the theoretical bottleneck duration based on the estimated maximum time consumption, including: To identify system bottlenecks, the scheduler continuously tracks the predicted time for processing a data block at each stage, and the current processing time. The theoretical bottleneck time for the assembly line is... The expression is: ; in, For the theoretical bottleneck duration, The time consumed in the prediction computation for processing the i-th data block in stage 0. The time consumed in the prediction computation for processing the i-th data block (Chunki) in the first stage. To determine the prediction computation time for processing the i-th data block (Chunki) in the second stage, a dynamic adjustment strategy is implemented, and the scheduler ensures that when... Calculating hour, Receiving via communication The result, and the communication operation in Complete before performing the calculation.

[0015] As a preferred embodiment of the parallel communication efficiency optimization method for new energy power generation prediction based on supercomputing described in this invention, the compressed prediction data is sent to the aggregation and scheduling layer via asynchronous communication, and a handle is returned immediately after sending, including: The computing island master node uses the non-blocking send function MPI_Isend. After the function is called, it immediately returns an MPI_Request handle. The actual data transmission is completed in the background by the MPI library. After the master node is called, it immediately processes the next computing task. The aggregation scheduling layer node uses the non-blocking receive function MPI_Irecv. After the node publishes a request to receive data, it returns immediately. The actual data reception is carried out in the background.

[0016] The beneficial effects of this preferred technical solution are as follows: By introducing a theoretical bottleneck identification and asynchronous communication mechanism based on maximum time estimation, the collaborative optimization of the computation and communication processes is achieved. During operation, bottlenecks can be identified based on the predicted time consumption of each stage, and task scheduling can be adjusted accordingly, allowing the computation and communication processes to overlap in time, thereby reducing waiting and blocking between stages.

[0017] Secondly, this invention provides a supercomputing-based parallel communication efficiency optimization system for predicting new energy power generation, comprising: The pipeline module uses a machine learning model to estimate the time taken to process data blocks and dynamically adjusts the transmission parameters of the data blocks based on the estimated maximum time taken, so that computation and communication are synchronized. The computing island module, based on the pipeline layer after the synchronization of computing and communication, generates the original prediction data of the master node for the sub-region within the computing island layer, and compresses the original prediction data; The aggregation scheduling module sends the compressed prediction data to the aggregation scheduling layer. The aggregation scheduling layer processes the received prediction data according to its type and uses the resource scheduler to make intelligent decisions, optimizing data transmission through multi-mode data synchronization.

[0018] Thirdly, the present invention provides an electronic device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of a parallel communication efficiency optimization method for predicting new energy power generation based on supercomputing.

[0019] Fourthly, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the supercomputing-based parallel communication efficiency optimization method for predicting new energy power generation.

[0020] Compared with existing technologies, the beneficial effects of this invention are as follows: By constructing a multi-layered collaborative parallel communication optimization mechanism in the supercomputing system, this invention achieves dynamic coordination and efficient operation of the computation and communication processes in new energy power generation prediction tasks. By introducing a feature vector-based time-consuming prediction model and bottleneck identification mechanism at the pipeline layer, the system can achieve forward scheduling of computational load and dynamic adjustment of transmission parameters according to the size, complexity, and type of different data blocks, ensuring that computation and communication remain synchronized in time. The computation island layer performs partitioned compression processing on the raw prediction data to reduce the communication load; the aggregation and scheduling layer achieves rapid data aggregation and intelligent scheduling through asynchronous communication and a multi-mode data synchronization mechanism. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the overall process of a parallel communication efficiency optimization method for predicting new energy power generation based on supercomputing, according to one embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram of a hybrid parallel architecture for a supercomputing-based method for optimizing parallel communication efficiency in predicting new energy power generation, as described in one embodiment of the present invention.

[0024] Figure 3 This is a complete flowchart illustrating the non-blocking communication overlap between the master node and the aggregation node of the computing island in a parallel communication efficiency optimization method for predicting new energy power generation based on supercomputing, as described in one embodiment of the present invention.

[0025] Figure 4 This is a schematic diagram illustrating the implementation process of an event-driven synchronization mechanism for a supercomputing-based parallel communication efficiency optimization method for predicting new energy power generation, as described in one embodiment of the present invention.

[0026] Figure 5 This is a schematic diagram of the dynamic pipeline scheduling timing of a parallel communication efficiency optimization method for predicting new energy power generation based on supercomputing, as described in one embodiment of the present invention. Detailed Implementation

[0027] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0028] Example 1, referring to Figure 1 As an embodiment of the present invention, a method for optimizing the parallel communication efficiency of new energy power generation prediction based on supercomputing is provided, comprising: To address the issue of the main process becoming the sole bottleneck for performance and reliability, and the poor parallel efficiency and scalability resulting from global synchronization operations and massive redundant data transmission, this invention provides a method for optimizing parallel communication efficiency in supercomputing-based new energy power generation prediction. This method aims to address the challenge of effectively utilizing the massive computing resources of supercomputers, as these problems worsen with increasing computational scale and problem complexity.

[0029] S1: Divide the computing nodes of the supercomputer into pipeline layer, computing island layer and summary scheduling layer.

[0030] S2: Based on the pipeline layer, a machine learning model is used to estimate the data block latency, and the transmission parameters of the data block are dynamically adjusted according to the estimated maximum latency to synchronize computation and communication.

[0031] S3: Based on the pipeline layer after the synchronization of computation and communication, the master node generates the original prediction data for the sub-region within the computation island layer, and compresses the original prediction data.

[0032] S4: The compressed prediction data is sent to the aggregation scheduling layer. The aggregation scheduling layer processes the received prediction data type and makes intelligent decisions using the resource scheduler, optimizing data transmission through multi-mode data synchronization.

[0033] Therefore, by layering nodes, predicting time consumption, and synchronizing scheduling, the computation and communication processes are coordinated; the predicted data is compressed at the computation island layer to reduce the transmission burden; and multi-mode data synchronization and optimized resource allocation are achieved at the aggregation and scheduling layer.

[0034] Example 2, refer to Figures 1-4 This is one embodiment of the present invention. Based on the above embodiment, a method for optimizing the parallel communication efficiency of new energy power generation prediction based on supercomputing is provided.

[0035] In this embodiment of the application, step S1 divides the supercomputer's computing nodes into a pipeline layer, a computing island layer, and a summary scheduling layer, including: Reference Figure 2 A hierarchical-pipeline hybrid parallel architecture was established, logically dividing the supercomputing nodes into three layers: The pipeline layer breaks down the prediction process into multiple stages, including data preprocessing, feature extraction, core prediction, and result integration. Each stage is handled by a dedicated process group, forming a pipeline operation.

[0036] The compute island layer further divides the core prediction process group into multiple logical compute islands. Each compute island contains a master node and multiple compute nodes, responsible for the prediction task of a complete sub-region. The compute islands internally use a hybrid parallel mode of MPI+OpenMP or MPI+CUDA. That is, the compute island layer contains one or more pipeline layers; dividing different pipeline layers according to geographical location results in the compute island layer.

[0037] The aggregation and scheduling layer consists of a master aggregation node and an intelligent task scheduler. It receives the results from all computing islands and performs overall aggregation, while dynamically scheduling all tasks using artificial intelligence technology.

[0038] In this embodiment of the application, step S2 uses a machine learning model based on the pipeline layer to estimate the data block latency, and dynamically adjusts the data block transmission parameters according to the estimated maximum latency to synchronize computation and communication, including A1-A3: A1: Based on the pipeline layer, feature vectors of data blocks are extracted in real time. Based on the extracted feature vectors, a machine learning model is used to estimate the time taken to process the data blocks.

[0039] A2: Determine the theoretical bottleneck duration based on the estimated maximum time consumption, and select an adjustment strategy for data block transmission parameters based on the theoretical bottleneck duration.

[0040] To identify system bottlenecks, the scheduler continuously tracks the predicted time for processing a data block at each stage, and the current processing time. The theoretical bottleneck time for the assembly line is... The expression is: ; in, The time consumed in the prediction computation for processing the i-th data block in stage 0. The time consumed in the prediction computation for processing the i-th data block (Chunki) in the first stage. To determine the prediction computation time for processing the i-th data block (Chunki) in the second stage, a dynamic adjustment strategy is implemented, and the scheduler ensures that when... Calculating hour, Receiving via communication The result, and the communication operation must be Complete before performing the calculation.

[0041] A3: Based on the selected data block transmission parameter adjustment strategy, dynamically adjust the data block transmission parameters to synchronize computation and communication.

[0042] There are two adjustment methods: dynamically adjusting the data block size and dynamically adjusting the transmission timing; Dynamically adjust the data block size if The current data block's computation time is significantly longer than the bottleneck time of the previous data block, indicating a large computational load. To avoid excessively long wait times in subsequent stages, the size of subsequent data blocks should be reduced, bringing their computation time closer to that of the current block. This makes the pipeline smoother; conversely, if the system remains idle, the data block size can be increased to reduce the number of communications and improve efficiency. The adjustment formula is as follows: ; in, For the adjusted data block size, The current size of the data block. This is the target calculation duration, generally set to recent. The moving average, The computation time for the next data block in the target phase, as given by the prediction model; Dynamically adjusting transmission timing, i.e., intelligent task scheduler control. Will The result was sent to The start time, the core principle is take over Communication operations must be performed Start calculation Ideally, communication operations should be completed beforehand. calculate This will be done simultaneously. The specific strategy is... Finish The calculations are initiated immediately (or slightly earlier based on the prediction) to... Asynchronous communication operation for sending results. (Through...) The predicted values ​​are used to estimate when communication should begin, ensuring that it is completed before the next phase of calculations begins.

[0043] In this embodiment of the application, the machine learning model used to estimate the data block time in step S2 specifically employs a linear regression model: When data blocks About to enter the stage At that time, the scheduler built into the pipeline layer will extract data blocks in real time. eigenvectors ,vector include: Data block size Estimated complexity of data blocks For example, the turbulence intensity index and data sparsity of meteorological data, and the type of data blocks. For example, stable weather, storm fronts, and clear-air radiation; A linear regression model is used for prediction. The model is trained offline using historical execution data and supports lightweight online fine-tuning. The mathematical expression is: in, For the predicted data blocks In the stage The computation time, For the stage Parameters of a model trained separately. , and As weight, The bias term, weights, and bias terms are obtained through training on historical data. For the size of the data block, The estimated complexity of the data block. This is a data block type.

[0044] During the training phase, the system collects data from historical assignments. Processing data blocks with different features Actual time spent , forming a dataset , used to train the model.

[0045] In an optional implementation, the time taken to estimate the data block using a machine learning model in step S2 can also be achieved using a multivariate multinomial regression model. When there is a non-linear relationship between the data block features and the computation time, the model obtains parameters by fitting the sample data using the least squares method, which is used to predict the computation time under different scales and complexities during pipeline operation.

[0046] In another optional implementation, the time taken to estimate the data block using a machine learning model in step S2 can also be achieved using a decision tree regression model. When the data block types differ significantly, the model generates a tree structure by hierarchically dividing the input features, with each leaf node representing a typical computational pattern and its average time taken.

[0047] In this embodiment of the application, in step S3, based on the pipeline layer after computation and communication synchronization, the master node generates the original prediction data for the sub-region within the computation island layer, and compresses the original prediction data, including B1-B3: B1: Based on the pipeline layer after the synchronization of computation and communication, the master node generates the original prediction data for the sub-region within the computation island layer.

[0048] B2: Select a compression strategy for the original prediction data based on the actual application scenario. The compression strategy includes feature extraction and compression through a feature extraction model reconstructed from the data.

[0049] Feature extraction: The prediction results of the sub-regions by the main node within the island (such as the power value per second for the next 24 hours) ) Perform calculations to generate a feature vector : ; in, It is the average power. That is the maximum power. It is the minimum power. It is the point in time when maximum power is reached. It is the maximum uphill gradient. , It is the maximum downhill ramp rate , To determine availability (the ratio of power generation to rated capacity), after characterization, traditional compression algorithms (such as ZLib and LZ4) are used to perform lossless compression on the feature vector or index vector, further reducing the data volume by several percentage points.

[0050] B3: Compress the data according to the selected compression strategy for the original forecast data.

[0051] In this embodiment of the application, the compression of the feature extraction model through data reconstruction in step B2 specifically adopts an autoencoder neural network; Encoding process: For a time series power data via encoder Compressed into a low-dimensional feature vector Z: ; in, It is an encoder The output contains the core information of the original data X. Represents the encoder function itself. It is an input variable. It is the set of weight parameters for all layers in the encoder neural network. It is the weight matrix connecting the input layer X to the first hidden layer. It is the weight matrix connecting the first hidden layer to the second hidden layer (i.e., the output layer). It is the bias vector of the first layer of neurons. It is the bias vector of the second layer (output layer) neurons. It is an activation function.

[0052] Decoding process: After receiving the low-dimensional feature vector Z, the summarization layer uses the corresponding decoder. Perform data reconstruction: ; in, It is the reconstructed data. It is a decoder function. These are the parameters of the decoder network.

[0053] In an optional implementation, the compression of the feature extraction model through data reconstruction in step B2 can also be achieved by principal component analysis. When the linear features are obvious and the computational burden is small, the original high-dimensional data can be projected into a low-dimensional space by performing covariance matrix decomposition on the time series power data and selecting the principal component direction, thereby achieving data compression and feature preservation.

[0054] In another alternative implementation, the compression of the feature extraction model through data reconstruction in step B2 can also be performed using a variational autoencoder. When the power data has uncertainty and multi-peak distribution, a probability distribution hypothesis is introduced on the basis of the autoencoder, and the unified modeling of compression and reconstruction is achieved by learning the potential distribution of the data.

[0055] It should be noted that by introducing a multi-layered data compression mechanism at the computing island layer, the transmission and storage efficiency of new energy power generation prediction results in the supercomputing system has been improved. After generating the original prediction data for the sub-region, the master node can select an appropriate compression strategy according to the application scenario: on the one hand, by extracting power sequence features, the continuous time series is transformed into a representative statistical feature vector, and lossless compression algorithms are combined to reduce the amount of data; on the other hand, an autoencoder neural network is used to achieve deep compression based on data reconstruction, which reduces data redundancy while retaining core feature information.

[0056] In this embodiment of the application, step S4 sends the compressed prediction data to the aggregation scheduling layer. The aggregation scheduling layer processes the received prediction data type and uses the resource scheduler to make intelligent decisions, optimizing data transmission through multi-mode data synchronization, including C1-C4: C1: Sends the compressed prediction data to the aggregation scheduling layer via asynchronous communication, and returns a handle immediately after sending.

[0057] C2: The master node saves the returned handle to the pending request pool and immediately processes the next computation task.

[0058] C3: The pending request pool continuously checks the request status and determines whether to release related resources based on the request status.

[0059] refer to Figure 3 The computation island completes sub-region prediction, and the master node performs feature extraction and compression on the results. It then calls MPI_Isend to initiate asynchronous transmission. The MPI library takes over the communication task, transmitting data over the network in the background. The master node saves the returned communication handle to the pending request pool and immediately returns to continue processing the next computation task. This achieves overlap between computation and communication.

[0060] At the receiving end, the aggregation node pre-publishes an MPI_Irecv non-blocking receive request to receive data. During data transmission, parallel processing tasks (such as integrating already received results) are performed.

[0061] The entire process also includes periodic request pool checks, continuously monitoring the status of each request. If the communication corresponding to a request has been completed, the relevant resources are released; otherwise, it continues to wait. In this way, the system can efficiently manage all background communication tasks.

[0062] C4: The aggregation scheduling layer processes the received predicted data types and uses the resource scheduler for intelligent decision-making.

[0063] C5: Optimizes data transmission through multi-mode data synchronization.

[0064] Reference Figure 4 The data transmission employs a multi-mode data synchronization strategy, specifically including: Event-driven synchronization mechanisms for dynamic data. This strategy is primarily used for data that updates rapidly and changes unpredictably. In the traditional approach, regardless of whether the data has changed, all data is sent to all computing islands at fixed time intervals (e.g., once per second), or the computing islands periodically query the data. This method generates a lot of unnecessary communication.

[0065] This event-driven approach uses a publish-subscribe model. The data source (such as a numerical weather forecast service) acts as the publisher, and the computing island acts as the subscriber. The publisher only notifies the subscriber when a meaningful change occurs in the data (i.e., an event is triggered).

[0066] Reference Figure 4 The implementation process of the event-driven synchronization mechanism is as follows: When new dynamic data is generated, the system first calculates and compares the differences between the new and old data; if the data has not changed, the data packet is discarded and no action is taken; if the data has changed, an event-driven mechanism is triggered. The system analyzes which areas are affected by the data changes. The system checks each computing island one by one to determine the area each computing island is responsible for. If there is overlap with the area that has changed: if there is no overlap, skip the computation island and do not send data; if there is overlap, add the incremental data to the asynchronous sending queue and send it to the affected computation island.

[0067] Change detection and incremental generation: the data source will save the latest version number of the data. and hash value .

[0068] When new data Upon arrival, compared to the previous version Compare the data with the old data and calculate the difference, i.e., the increment: ; in, For increments, The function can be selected based on the data type: for grid data, a bitmap is used to mark the cells that have changed; for arrays, run-length encoding or binary difference algorithms such as bsdiff are used. For new data, This is data from the previous version.

[0069] Only when The subsequent synchronization steps will only be initiated when the change is not empty.

[0070] Impact range analysis shows that not all data changes affect every computational island. A key step is analyzing spatial correlations.

[0071] The judgment rule is that for each computation island j, the geographical area it is responsible for is... Data update The area of ​​influence is Only when Updates are only pushed to the compute island when these two regions overlap. This ensures that updates are only sent to the compute islands that actually need them, thus minimizing communication overhead.

[0072] A periodic synchronization and caching mechanism for static / slow-changing data. Used for updating background information that is updated infrequently but in large volumes.

[0073] Leveraging the long-term stability of the data, a complete data broadcast is performed at the start of the job, and a local data copy (cache) is established on each computation island. Throughout the job's execution, all computations directly read the locally cached data.

[0074] When the job starts, the master node broadcasts static data. and the latest version number Each computing island receives and caches data. .

[0075] During job execution, the computing island receives data, stores it in its local cache, and then... Record it as the local version number .at this time, This indicates that the cache is up-to-date, and the compute island is directly using local resources. Perform the calculation.

[0076] Periodic or event-triggered checks are performed. When it's necessary to check the validity of the cache (e.g., before a new computation begins on a compute island), a query request containing the local version number is sent to the master node. .

[0077] The master node compares the version number, and the master node will receive... With my latest The two version numbers are compared. If they match, it means the data hasn't been updated, and the master node notifies the compute island to continue using the cache. If they don't match, it means the data on the master node has been updated, and the compute island's cache has expired. The master node then initiates an update process, adding the new data (or the changed parts) and the new... Send it to this computing island. After the computing island updates its cache, it will... Set as new : .

[0078] in, This is the latest version number. This is the local version number.

[0079] Characterized compression transmission of intermediate and result data, which processes data generated within the system and needs to be exchanged between different nodes, is the most direct way to reduce communication volume.

[0080] Before leaving the computing node, data is encoded and transformed from its original form into a feature form. The feature form has higher information density and lower dimensionality, thus significantly reducing the data volume.

[0081] Step 1: Calculate the island to generate raw data The computing island completes the prediction calculations for the sub-region it is responsible for, generating high-dimensional and high-precision raw result data Y (e.g., power prediction data for the next 72 hours, one point every 5 minutes, which is a large amount of data).

[0082] Step 2: Data Compression and Characterization The master node within the computing island does not directly send the raw data Y, but intelligently selects one or more of the following compression methods based on the data's final purpose: Key indicator extraction (lossy compression): When the ultimate goal is to generate statistical reports or to monitor and alert, key features are extracted using statistical functions to obtain a low-dimensional feature vector F.

[0083] ; Where F is the low-dimensional feature vector, It is the average power. That is the maximum power. It is the minimum power. It is the point in time when maximum power is reached. It is the maximum uphill gradient. , It is the maximum downhill ramp rate , Availability (the ratio of power generation to rated capacity) Model-based compression (lossless / lossy) uses a pre-trained encoder model when it is necessary to approximately recover the original data sequence at the summarization layer for subsequent analysis.

[0084] in, It is an encoder The output contains the core information of the original data X. Represents the encoder function itself. It is an input variable. It is the set of weight parameters for all layers in the encoder neural network.

[0085] Step 3: Initiate asynchronous non-blocking communication The master node of the computing island immediately calls MPI_Isend to asynchronously send the compressed or characterized data to the aggregation scheduling layer.

[0086] The function call immediately returns an MPI_Request handle, and the actual data transfer task is completed in the background by the MPI library.

[0087] Step 4: Return immediately and process the new task. After the master node saves the communication handle to the pool of pending requests, it returns immediately without waiting for the communication to complete, and continues processing the next computation task. This allows computation and communication to occur simultaneously.

[0088] Step 5: Data Reception and Reconstruction at the Aggregation Layer The nodes in the aggregation layer receive data non-blockingly via MPI_Irecv.

[0089] Upon receiving the data, process it accordingly based on its data type: If the received data is a feature vector F, it is directly stored in the report database without needing to be restored.

[0090] If the received data is the encoded feature Z, the decoder model that is paired with the encoder is called to reconstruct the data.

[0091] In this embodiment of the application, the compressed prediction data is sent to the aggregation and scheduling layer via asynchronous communication in step C1. Specifically, non-blocking transmission is used. The computing island master node uses the non-blocking transmission function MPI_Isend. After the function is called, it immediately returns an MPI_Request handle. The actual data transmission is completed in the background by the MPI library. The master node does not need to wait after the call and immediately processes the next computing task. The aggregation and scheduling layer node uses the non-blocking reception function MPI_Irecv. After the node publishes a request to receive data, it returns immediately. The actual data reception is performed in the background.

[0092] The MPI_Request handle represents a specific send and receive operation. The handle is a unique identifier used later to query the operation status and wait for the operation to complete.

[0093] In an optional implementation, sending the compressed prediction data to the aggregation scheduling layer via asynchronous communication in step C1 can also be done using persistent communication based on MPI. When there is a fixed communication mode or repeated data exchange between the computing island layer and the aggregation scheduling layer, the two communicating parties establish a persistent communication channel during the initialization phase. Subsequently, only MPI_Start needs to be called to activate the communication, without having to repeatedly establish the communication context.

[0094] In another alternative implementation, the compressed prediction data can be sent to the aggregation and scheduling layer asynchronously in step C1 using one-sided communication based on MPI. When the aggregation and scheduling layer node needs to directly access the memory data of the computing island layer, the sending end can directly write the data into the memory window predefined by the receiving end through the remote memory access mechanism, without the need for explicit receiving operation.

[0095] In this embodiment of the application, the intelligent decision-making using the resource scheduler in step S4 specifically employs an intelligent task scheduler: The intelligent task scheduler adopts a task scheduling mechanism based on the principle of maximizing the utilization of the entire supercomputing cluster's resources and minimizing the total job completion time. By continuously monitoring the system status and making intelligent decisions, it solves the problems of dynamic load and system fluctuations that traditional static scheduling methods cannot handle.

[0096] Multi-dimensional real-time status monitoring (perception phase): The intelligent task scheduler continuously collects a large amount of real-time data from the entire cluster, which forms the basis for its decision-making. Monitoring content includes: The system monitors the load status and metrics, including CPU utilization, GPU utilization, memory utilization, and cache hit rate for all nodes within each compute island. The monitoring objective is to accurately understand the current workload and remaining computing capacity of each compute island.

[0097] Network communication status is monitored by metrics such as the current bandwidth, latency, and congestion level of the communication link between the computing island and the aggregation node. The purpose of monitoring is to identify communication bottlenecks. Even if a computing island is idle, assigning it new tasks may not be the best option if its network connection is congested.

[0098] Task and data status monitoring metrics include the number of tasks in the current task queue for each compute island, the duration of task execution, and the size and characteristics of pending data blocks. The purpose of monitoring is to understand the future load of each compute island and determine when it will likely become idle.

[0099] Node health status is monitored using metrics such as node heartbeat signals, temperature, and hardware error logs. The purpose of monitoring is to detect potential hardware failures or performance degradation issues as early as possible. All this monitoring data is aggregated to form a global, time-varying system state vector. .

[0100] Intelligent decision-making based on reinforcement learning (decision phase): The scheduler uses reinforcement learning as its core decision-making method, making it suitable for solving dynamic scheduling problems. The specific mechanism is as follows: The reinforcement learning framework models the system, with the state represented by the system state vector S(t). Actions are the operations performed by the scheduler. The core action is to predict the next available sub-region. To which computing island is it assigned? .

[0101] The reward function is the objective function used to guide the AI ​​agent's learning. The core objective of the reward function is to minimize the total task time. For example: A positive reward is given when a task is completed quickly.

[0102] When a computing island is idle (load imbalance) or communication delays occur due to network congestion, a negative reward (penalty) is given.

[0103] When a new task needs to be assigned, the scheduler will use the current system state. The input is fed into the already trained reinforcement learning policy network.

[0104] The policy network, based on experience, evaluates the long-term benefit (called the Q-value) of each potential action (which island to assign the task to) and outputs an optimal action. ,Right now: ; in, The action chosen by the agent at time t. This means traversing all possible computation islands j to find the one that satisfies the function The value of j that reaches the maximum is taken as the result. The Q-function in reinforcement learning, also known as the action value function, For specific actions, this refers to the operation of assigning the task to computing island j.

[0105] This decision-making process takes into account both current load and long-term benefits. It may not always assign tasks to the currently idle node, but rather to a node that is currently slightly busy but has good network connectivity and will soon become idle, in order to achieve the best overall effect.

[0106] For model training, reinforcement learning models are usually trained offline using historical job data. Methods such as imitation learning or deep Q-learning can be used to learn the scheduling strategies of senior engineers, or to find the optimal strategy through a large number of simulation experiments.

[0107] Once the model is trained and put into use, it can be continuously fine-tuned through an online learning mechanism to adapt to changes in the cluster environment.

[0108] Fault detection and task migration (fault tolerance phase): Fault detection is performed. The scheduler continuously monitors signals and performance metrics from the compute islands. If a node does not respond within a set time, or its performance metrics remain abnormal and fall below normal thresholds, it will be marked as a suspected fault. Detection methods include simple timeout checks or more complex statistical anomaly detection algorithms. Task migration is then performed. If a compute island is confirmed to have failed or experienced a severe performance degradation, the scheduler will immediately initiate the migration process. In the task queue, mark all unfinished tasks on this faulty computing island as needing to be migrated.

[0109] Based on the reinforcement learning decision-making mechanism, tasks are reassigned to the healthiest computing island that is expected to complete the task earliest.

[0110] Transfer the partially calculated results from the faulty node to the new node to avoid starting the task from scratch.

[0111] The process is usually completed automatically without human intervention, thus ensuring that the system can continue to run and complete the job even if some nodes fail.

[0112] In an optional implementation, the intelligent decision-making using the resource scheduler in step S4 can also employ rule-based and weight-based adaptive scheduling. When the system is large in scale, the state changes frequently, but the computing tasks are relatively independent, the scheduler sets a weight function based on indicators such as node CPU utilization, network latency, and expected task execution time, dynamically calculates a comprehensive score, and assigns the task to the node with the highest score.

[0113] In another optional implementation, the intelligent decision-making using the resource scheduler in step S4 can also employ global optimization scheduling based on a genetic algorithm. When the task scale is large, the resource distribution is complex, and there are multi-objective optimization requirements, the task allocation scheme is encoded as chromosomes, a fitness function is designed, and the globally optimal scheduling scheme is approximated through multiple generations of evolution via operations such as selection, crossover, and mutation. Genetic algorithms can explore globally near-optimal solutions in large-scale heterogeneous resources and are suitable for periodic batch scheduling.

[0114] In summary, this invention achieves collaborative optimization of computation and communication in new energy power generation prediction under a supercomputing environment by constructing a hierarchical-pipeline hybrid parallel architecture and an asynchronous communication mechanism. In the pipeline layer, a feature vector-based computation time prediction model and a dynamic bottleneck identification mechanism adaptively adjust task scheduling and transmission parameters according to the complexity and size of data blocks, enabling computation and communication to overlap in time and reducing waiting and blocking. The computation island layer utilizes feature extraction and self-encoding compression to reduce data transmission volume, and the aggregation scheduling layer achieves efficient aggregation of prediction results and dynamic resource allocation through non-blocking communication, multi-mode data synchronization, and intelligent scheduling mechanisms.

[0115] Example 3, Reference Figure 5 To verify the beneficial effects of the present invention, simulation experiments were conducted.

[0116] After dynamically adjusting the size of data blocks and the timing of transmission, the ideal state of the system is as follows: Figure 5 As shown. From Figure 5 As can be seen, computational operations (orange) and communication operations (green) are completely parallel in time. Long computational wait times are used to transmit data, and communication time is completely hidden, thereby significantly improving the overall efficiency and resource utilization of the pipeline.

[0117] Example 4: This invention runs on a supercomputing cluster connected via a high-speed Infiniband network. The software used includes the MPI library, Python / C++ language, PyTorch / TensorFlow framework, and a reinforcement learning scheduling framework based on Ray or a self-developed framework.

[0118] Assuming a supercomputing platform has 512 computing nodes, it needs to predict the power generation of a wind farm containing 1,000 wind turbines for the next 72 hours.

[0119] Logically, the 512 nodes are divided into 16 nodes as the pipeline layer (Group 0 to Group 3), 1 node as the aggregation and scheduling layer, and the remaining 495 nodes as the computing island layer. These 495 nodes are further divided into 15 computing islands, each with 33 nodes (including 1 master node and 32 computing nodes). The entire wind farm's geographical area is then divided into 15 sub-regions, with each computing island responsible for one sub-region.

[0120] Group0 receives the raw numerical weather forecast data, decodes and performs quality checks, and then passes it to Group1.

[0121] Group1 extracts key meteorological features (such as wind speed, wind direction, and turbulence intensity) and then sends the data to the master node of Group2.

[0122] The intelligent task scheduler continuously monitors the status of all computing islands and dynamically allocates computing tasks from the 15 sub-regions to the 15 computing islands.

[0123] Each computing island runs a computational fluid dynamics or artificial intelligence prediction model in parallel within itself to complete the power prediction of the sub-region it is responsible for.

[0124] The master node of the computing island performs feature processing on the prediction results (such as extracting key indicators like future power curves, maximum values, and minimum values), and then immediately uploads them to the aggregation node asynchronously.

[0125] The aggregation node receives all feature data, and Group3 integrates this data into a full-field prediction report and performs subsequent calibration processing.

[0126] The terrain data is broadcast to all compute islands at once and cached locally.

[0127] When new numerical weather forecast data arrives, the system only sends the changed wind field data to the computing island corresponding to the sub-region affected by the weather changes.

[0128] Through the above implementation methods, the present invention effectively solves the communication bottleneck problem and significantly improves the overall efficiency of new energy power generation prediction.

[0129] Example 5 illustrates a schematic scheme for optimizing the parallel communication efficiency of new energy power generation prediction based on supercomputing. It should be noted that the technical solution of this system for optimizing the parallel communication efficiency of new energy power generation prediction based on supercomputing is based on the same concept as the aforementioned method for optimizing the parallel communication efficiency of new energy power generation prediction based on supercomputing. Details not described in detail in the system for optimizing the parallel communication efficiency of new energy power generation prediction based on supercomputing in this embodiment can be found in the description of the aforementioned method for optimizing the parallel communication efficiency of new energy power generation prediction based on supercomputing.

[0130] This embodiment also provides a supercomputing-based parallel communication efficiency optimization system for predicting new energy power generation, including: The pipeline module uses a machine learning model to estimate the time taken to process data blocks and dynamically adjusts the transmission parameters of the data blocks based on the estimated maximum time taken, so that computation and communication are synchronized. The computing island module, based on the pipeline layer after the synchronization of computing and communication, generates the original prediction data of the master node for the sub-region within the computing island layer, and compresses the original prediction data; The aggregation scheduling module sends the compressed prediction data to the aggregation scheduling layer. The aggregation scheduling layer processes the received prediction data based on its type and uses the resource scheduler for intelligent decision-making.

[0131] This embodiment also provides an electronic device applicable to the optimization of parallel communication efficiency for supercomputing-based new energy power generation prediction, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the supercomputing-based new energy power generation prediction parallel communication efficiency optimization method proposed in the above embodiment.

[0132] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the parallel communication efficiency optimization method for predicting new energy power generation based on supercomputing, as proposed in the above embodiment.

[0133] The storage medium proposed in this embodiment belongs to the same inventive concept as the parallel communication efficiency optimization method for supercomputing-based new energy power generation prediction proposed in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0134] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0135] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for optimizing the parallel communication efficiency of new energy power generation prediction based on supercomputing, characterized in that, include: The computing nodes of a supercomputer are divided into a pipeline layer, a computing island layer, and a summary scheduling layer. Based on the pipeline layer, a machine learning model is used to estimate the data block latency, and the transmission parameters of the data block are dynamically adjusted according to the estimated maximum latency to synchronize computation and communication. Based on the pipeline layer after the synchronization of computation and communication, the master node generates the original prediction data for the sub-region within the computation island layer, and compresses the original prediction data. The compressed forecast data is sent to the aggregation scheduling layer, which processes the received forecast data type and uses the resource scheduler to make intelligent decisions, optimizing data transmission through multi-mode data synchronization.

2. The method for optimizing parallel communication efficiency in supercomputing-based new energy power generation prediction as described in claim 1, characterized in that, The pipeline-based approach employs a machine learning model to predict data block latency and dynamically adjusts data block transmission parameters based on the estimated maximum latency to synchronize computation and communication. This includes: Based on the pipeline layer, feature vectors of data blocks are extracted in real time. Based on the extracted feature vectors, a machine learning model is used to estimate the time taken to process the data blocks. The theoretical bottleneck duration is determined based on the estimated maximum latency, and an adjustment strategy for data block transmission parameters is selected based on the theoretical bottleneck duration. Based on the selected data block transmission parameter adjustment strategy, the data block transmission parameters are dynamically adjusted to synchronize computation and communication.

3. The method for optimizing parallel communication efficiency in supercomputing-based new energy power generation prediction as described in claim 2, characterized in that, The pipeline layer based on synchronized computation and communication generates raw prediction data for sub-regions for the master node within the computation island layer, and compresses the raw prediction data, including: Based on the pipeline layer after the synchronization of computation and communication, the original prediction data of the master node for the sub-region is generated within the computation island layer. The selection of compression strategies for raw prediction data is based on the actual application scenario. Compression strategies include feature extraction and compression through feature extraction models reconstructed from data. Data compression is performed based on the selected compression strategy for the original forecast data.

4. The method for optimizing parallel communication efficiency in supercomputing-based new energy power generation prediction as described in claim 3, characterized in that, The compressed prediction data is sent to the aggregation and scheduling layer, which processes the received prediction data type and uses a resource scheduler for intelligent decision-making. This optimization of data transmission through multi-mode data synchronization includes: The compressed prediction data is sent to the aggregation and scheduling layer via asynchronous communication, and a handle is returned immediately after the data is sent. The master node saves the returned handle to the pending request pool and immediately processes the next computation task. The pending request pool continuously checks the request status and determines whether to release related resources based on the request status; The aggregation scheduling layer processes the received predicted data types and uses the resource scheduler for intelligent decision-making. Optimize data transmission through multi-mode data synchronization.

5. The method for optimizing parallel communication efficiency in supercomputing-based new energy power generation prediction as described in claim 4, characterized in that, The process of extracting feature vectors from data blocks in real time based on the pipeline layer, and estimating the data block latency using a machine learning model based on the extracted feature vectors, includes: When data blocks About to enter the stage At that time, the scheduler built into the pipeline layer will extract data blocks in real time. eigenvectors ; The prediction is performed using a linear regression model, and the mathematical expression is: in, For the predicted data blocks In the stage The computation time, For the stage Parameters of a model trained separately. , and As weight, For bias terms, For the size of the data block, The estimated complexity of the data block. This is a data block type.

6. The method for optimizing parallel communication efficiency in supercomputing-based new energy power generation prediction as described in claim 5, characterized in that, The determination of the theoretical bottleneck duration based on the estimated maximum time consumption includes: To identify system bottlenecks, the scheduler continuously tracks the predicted time for processing a data block at each stage, and the current processing time. The theoretical bottleneck time for the assembly line is... The expression is: in, For the theoretical bottleneck duration, The time consumed in the prediction computation for processing the i-th data block in stage 0. The time consumed in the prediction computation for processing the i-th data block (Chunki) in the first stage. To determine the prediction computation time for processing the i-th data block (Chunki) in the second stage, a dynamic adjustment strategy is implemented, and the scheduler ensures that when... Calculating hour, Receiving via communication The result, and the communication operation in Complete before performing the calculation.

7. The method for optimizing parallel communication efficiency in supercomputing-based new energy power generation prediction as described in claim 6, characterized in that, The step of sending the compressed prediction data to the aggregation and scheduling layer via asynchronous communication, and immediately returning a handle after sending, includes: The computing island master node uses the non-blocking send function MPI_Isend. After the function is called, it immediately returns an MPI_Request handle. The actual data transmission is completed in the background by the MPI library. After the master node is called, it immediately processes the next computing task. The aggregation scheduling layer node uses the non-blocking receive function MPI_Irecv. After the node publishes a request to receive data, it returns immediately. The actual data reception is carried out in the background.

8. A supercomputing-based parallel communication efficiency optimization system for predicting new energy power generation, employing the method described in any one of claims 1-7, characterized in that, include: The pipeline module uses a machine learning model to estimate the time taken to process data blocks and dynamically adjusts the transmission parameters of the data blocks based on the estimated maximum time taken, so that computation and communication are synchronized. The computing island module, based on the pipeline layer after the synchronization of computing and communication, generates the original prediction data of the master node for the sub-region within the computing island layer, and compresses the original prediction data; The aggregation scheduling module sends the compressed prediction data to the aggregation scheduling layer. The aggregation scheduling layer processes the received prediction data according to its type and uses the resource scheduler to make intelligent decisions, optimizing data transmission through multi-mode data synchronization.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images