An efficient sea surface emissivity calculation method based on cross-language parallel scheduling

Abstract

This invention relates to an efficient sea surface emissivity calculation method based on cross-language parallel scheduling, comprising the following steps: a residual attention network model module normalizes the input physical parameters and extracts features; the residual attention network includes a baseline prediction branch, an attention gating branch, an explicit interactive feature branch, and a residual correction branch; a cross-language interactive interface connects the Python training environment and the Fortran numerical prediction program; the MPI parallel scheduling module adopts a master-slave architecture, implementing task distribution and result collection logic through a state machine; the batch of tasks to be processed in each working process is calculated and distributed; after receiving the task, each subprocess decodes the task identifier, calls the cross-language inference interface to complete the numerical calculation related to sea surface emissivity, and stores the sea surface emissivity calculation results in a local result array.

Description

Technical Field

[0001] This invention belongs to the field of meteorological numerical forecasting technology, and in particular relates to an efficient method for calculating sea surface emissivity based on cross-language parallel scheduling. Background Technology

[0002] Sea surface emissivity is a key physical parameter affecting satellite remote sensing brightness temperature inversion and sea surface wind field inversion. Traditional sea surface emissivity calculations mainly employ physical model methods, such as the two-scale model and the Durden model. These methods are based on the physical mechanism of electromagnetic wave-sea surface interaction, offering high accuracy but requiring enormous computational resources. Experiments show that traditional physical models take approximately 70 seconds to calculate a single sample, which is insufficient to meet the operational needs of real-time weather forecasting.

[0003] In recent years, deep learning technology has provided a new technical approach for calculating sea surface emissivity. By training neural network models using deep learning frameworks such as PyTorch in a Python environment, a rapid mapping from physical parameters to sea surface emissivity can be achieved. Existing schemes for integrating deep learning models into Fortran numerical weather prediction programs mainly fall into two categories: the first is a pure physical model scheme (pBRDF-E), and the second is a Python training to Fortran porting scheme (OceanEM_F90). The latter hardcodes the model weights into Fortran DATA statements. Experiments show that this scheme has a single-sample computation time of approximately 0.00003 seconds, which is about 2.3 million times faster than traditional physical models.

[0004] However, the second type of solution has the following technical limitations: First, computational methods based on physical models (such as pBRDF-E) have high computational complexity, with a single inference taking approximately 70 seconds, which cannot meet the timeliness requirements of numerical weather prediction scenarios. Second, existing solutions for porting deep learning models to the Fortran environment (such as OceanEM_F90) hardcode model weights into DATA statements, requiring the entire numerical weather prediction program to be recompiled for model updates, resulting in iteration cycles lasting several weeks. Third, traditional solutions lack support for complex model structures and cannot introduce advanced modules such as attention mechanisms and feature interactions, thus hindering further improvements in model accuracy. Fourth, the single-batch serial computation mode cannot fully utilize the parallel computing capabilities of HPC clusters, resulting in persistently high total computation time when calculating large-scale grid points. To address these technical issues, there is an urgent need to develop a new technical solution that maintains the flexibility of the Python training environment, seamlessly integrates with Fortran numerical weather prediction programs, and simultaneously supports complex model structures and MPI parallel acceleration. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a method for calculating sea surface emissivity based on cross-language parallel scheduling. Its overall technical framework is as follows: Figure 1 As shown, this method comprises four core modules: a residual attention network model module, a cross-language interaction interface module, an MPI parallel scheduling module, and a distributed inference execution module. The data flow and control flow relationships between these modules are as follows: Input parameters are first preprocessed through data normalization, and then passed to the C++ inference engine via the cross-language interface. The C++ engine calls the TorchScript model to perform forward computation, and the computation results are returned to the Fortran main program through the interface. The MPI main process is responsible for task distribution and result aggregation.

[0006] To achieve the above objectives, this application discloses an efficient sea surface emissivity calculation method based on cross-language parallel scheduling, characterized by the following steps: The residual attention network model module normalizes the input physical parameters and extracts features. The residual attention network contains four parallel branches: baseline prediction branch, attention gating branch, explicit interaction feature branch, and residual correction branch. The baseline prediction branch adopts a standard multilayer perceptron structure to learn the nonlinear mapping relationship from the input physical parameters to the sea surface emissivity. The attention gating branch learns the weight distribution of the input features through a self-attention mechanism. The explicit interaction feature branch uses a factorization machine mechanism to capture the second-order interaction effect between the input features. The residual correction branch adopts a deep residual learning framework to learn fine-grained correction amounts based on the baseline prediction. A cross-language interactive interface connects the Python training environment with the Fortran numerical prediction program. The preprocessed features are then used by the C++ inference engine. The C++ engine calls the TorchScript model to perform forward computation, and the computation results are returned to the Fortran main program through the interface. The MPI parallel scheduling module adopts a master-slave architecture and uses a state machine to implement task distribution and result collection logic. In the distributed inference execution module, the main process dynamically calculates the batch of tasks to be processed by each working process and completes the allocation and distribution. After receiving the task, each subprocess decodes the task identifier, matches it to obtain the corresponding azimuth interval, calls the cross-language inference interface to complete the numerical calculation of sea surface emissivity, and stores the complete set of sea surface emissivity calculation results, including isotope emission components and anisotropy correction coefficients, into the local result array.

[0007] Furthermore, for the input feature vector as It includes seven features: frequency f, salinity SSS, zenith angle θ, sea surface temperature SST, wind speed v, Fresnel vertical emissivity ev_f, and Fresnel horizontal emissivity eh_f; the mathematical expression for baseline prediction by the baseline prediction branch is: ; in , , This is the weight matrix. , , For the corresponding bias vector; The formula for calculating the attention gating branch is: ; in This is the attention weight matrix. The bias vector, the softmax(·) function normalizes the real vector into a probability distribution, and the attention weight vector is... ; Gated feature vector ,in This represents element-wise multiplication; the fused feature vector is... ; The formula for calculating the second-order interaction term of the explicit interaction feature branch is: ; in Let be the latent vector representation of the i-th feature. Let k be the latent vector representation of the j-th feature, and k be the dimension of the latent vector. , These are the normalized values ​​of the i-th and j-th input features, respectively. The residual calculation process for the residual correction branch is as follows: ; The input to the residual network is , and These are the weight matrix and bias vector of the output layer, respectively. To ensure that the network performance does not degrade in the early stage of training, the output layer adopts zero initialization constraint; the final network output is the sum of the baseline prediction and the residual correction. The training loss function adopts the weighted MSE loss function based on physical dimensions, and the physical RMSE is used as the basis for early stopping during the validation phase.

[0008] Furthermore, the cross-language interaction interface adopts a three-layer architecture: the C++ wrapper layer is responsible for encapsulating TorchScript inference calls, the Fortran-C interoperability layer is responsible for parameter type conversion and memory management, and the Fortran scheduling layer is responsible for task orchestration and result processing; the C++ wrapper layer loads TorchScript models and performs inference through the libtorch C++ API; the Fortran-C interoperability layer uses the ISO_C_BINDING standard to implement lock-free memory sharing; the Fortran scheduling layer passes array parameters by value or reference, and the C++ side uses pointers to receive them, with the memory layout adopting a column-first strategy to match the Fortran convention.

[0009] Furthermore, in the MPI parallel scheduling module, the main process uses a state machine to maintain a conditional jump structure to implement the scheduling decision logic. The state machine is defined as follows: S_MAIN ∈ {S_IDLE, S_SCAN, S_DISPATCH, S_COLLECT, S_TERMINATE} S_IDLE is the initialization state, S_SCAN is responsible for scanning the working process status, S_DISPATCH performs task distribution, S_COLLECT collects calculation results, and S_TERMINATE is the termination state. Each worker process maintains an independent sub-state machine: ; The state transitions of the worker process state machine are driven by the control messages of the main process and the local computation completion event. SS_FREE is the idle state, SS_WORKING is the working state, which performs cross-language computation, SS_RESPONSE is the response state, which sends the result ready message, and SS_WAIT is the waiting state. The MPI parallel scheduling module uses a dynamic task allocation strategy based on throughput prediction to allocate tasks. The current batch size of worker process w is... The dynamic adjustment coefficient is The actual amount of work allocated is: =

[0010] in This is the batch flexibility coefficient. The dynamic fairness coefficient is adjusted based on factors including the historical throughput of the working process, the current system load, and the task priority. The MPI parallel scheduling module uses a credit-based fair scheduling mechanism, where worker process w has a credit value C. V Follow these rules: ; in For credit recovery rate, For the target credit score, This is the timeout penalty coefficient. The timeout number is t, where t is the current scheduling round. This is the upper limit of the process credit value. The credit value directly affects the task allocation priority. The higher the credit value, the more tasks will be allocated to it. Attached Figure Description

[0011] Figure 1 This is a flowchart of the method in this application. Detailed Implementation

[0012] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0013] This application comprises four core modules: a residual attention network model module, a cross-language interaction interface module, an MPI parallel scheduling module, and a distributed inference execution module. The data flow and control flow relationships between these modules are as follows: Input parameters are first preprocessed through data normalization, and then passed to the C++ inference engine via the cross-language interface. The C++ engine calls the TorchScript model to perform forward computation, and the computation results are returned to the Fortran main program through the interface. The MPI main process is responsible for task distribution and result aggregation.

[0014] In one embodiment, the Residual Attention Network (RAN) designed in this application is a composite deep learning architecture comprising four parallel branches: a baseline prediction branch, an attention gating branch, an explicit interaction feature branch, and a residual correction branch. The functional definitions and technical implementations of each branch are as follows: 1. Baseline Prediction Branch: The baseline prediction branch employs a standard multilayer perceptron (MLP) structure, responsible for learning the nonlinear mapping relationship from input physical parameters to sea surface emissivity. For an input feature vector of... It includes seven features: frequency f, salinity SSS, zenith angle θ, sea surface temperature SST, wind speed v, Fresnel vertical emissivity ev_f, and Fresnel horizontal emissivity eh_f; the mathematical expression for baseline prediction by the baseline prediction branch is: ; in , , This is the weight matrix. , , This corresponds to the bias vector. The baseline branch output has a 12-dimensional dimension, corresponding to 2 isotopic components and 10 anisotropic correction coefficients.

[0015] 2. Attention-gated branch: The attention-gated branch learns the weight distribution of input features through a self-attention mechanism, achieving adaptive feature selection. Let the calculation formula for the attention-gated branch be: ; in This is the attention weight matrix. The bias vector, the softmax(·) function normalizes the real vector into a probability distribution, and the attention weight vector is... ; Gated feature vector ,in This represents element-wise multiplication; the fused feature vector is... .

[0016] 3. Explicit Interaction Feature Branch: The explicit interaction feature branch uses a factorization machine (FM) mechanism to capture second-order interaction effects between input features. The formula for calculating all second-order interaction terms is: ; in Let be the latent vector representation of the i-th feature. Let k be the latent vector representation of the j-th feature, and k be the dimension of the latent vector. , These are the normalized values ​​of the i-th and j-th input features, respectively, where k is typically between 8 and 16. The concatenated interaction feature vectors have a 22-dimensional dimension (21 interaction terms plus 1 FM aggregation scalar).

[0017] 4. Residual Correction Branch: The residual correction branch employs a deep residual learning framework to learn fine-grained corrections based on the baseline predictions. Let the input to the residual network be... =[x, , The residual calculation process is as follows: ; To ensure that network performance does not degrade during the initial training phase, the output layer employs a zero-initialization constraint: = 0, = 0. The final network output is the sum of the baseline prediction and the residual correction: = + Δy.

[0018] In one embodiment, the training loss function and optimization strategy of the present invention adopt a weighted MSE loss function based on physical dimensions to address the problem of large differences in the numerical ranges of different output components. Assuming the number of training samples is N and the output dimension is 12, the loss function is defined as: ; Among them, weight The physical value range of each output component is determined. Physical RMSE is used as the basis for early stopping during the validation phase. The model employs the Adam optimizer, with an initial learning rate of 0.001, a batch size of 256, a maximum training epoch of 500, and an early stopping strategy (patience=30).

[0019] Cross-language interaction interfaces are a crucial bridge connecting Python training environments and Fortran numerical prediction programs. In one embodiment, the interface designed in this invention adopts a three-layer architecture: a C++ wrapper layer is responsible for encapsulating TorchScript inference calls, a Fortran-C interoperability layer is responsible for parameter type conversion and memory management, and a Fortran scheduling layer is responsible for task orchestration and result processing.

[0020] The C++ wrapper layer loads TorchScript models and performs inference through the libtorch C++ API. The main interface functions are defined as follows: void inference_forward(float* input, float* output, int batch_size) Here, `input` is a pointer to the input data (with a dimension of `batch_size × 7`), and `output` is a pointer to the output buffer (with a dimension of `batch_size × 12`). The model weights are stored in TorchScript format in the file system and are dynamically loaded via the `dlopen` mechanism.

[0021] The Fortran-C interoperability layer uses the ISO_C_BINDING standard to implement lock-free memory sharing. Key data structures are defined as follows: type, bind(C) :: em_input real(C_FLOAT), dimension(7) :: paramsinteger(C_INT) :: batch_id end type em_input In Fortran, array parameters are passed by value or reference, while in C++, they are received by pointers. Memory layout follows a column-major strategy to match Fortran conventions.

[0022] In one embodiment, the MPI parallel scheduling module adopts a master-worker architecture, using nested state machines to implement complex task distribution and result collection logic. The design goals of the scheduling mechanism include: maximizing load balancing, ensuring fairness in task allocation, dynamic adaptive adjustment, and fault tolerance and recovery.

[0023] 1. Main Process State Machine Design: The main process maintains a conditional jump structure to implement the scheduling decision logic. The state machine is defined as follows: S_MAIN ∈ {S_IDLE, S_SCAN, S_DISPATCH, S_COLLECT, S_TERMINATE}; S_IDLE represents the initialization state, S_SCAN scans the worker process status, S_DISPATCH performs task distribution, S_COLLECT collects computation results, and S_TERMINATE represents the completion state. State transition conditions involve multiple dimensions, including checking the task pool for non-empty states, worker process availability, timeout detection, and summary completion detection.

[0024] 2. Design of the working process state machine Each worker process maintains an independent sub-state machine: ; The state transitions of the worker process state machine are driven by both the control messages from the main process and the local computation completion event.

[0025] 3. Dynamic task allocation strategy Tasks are allocated using a dynamic task allocation strategy based on throughput prediction. The current batch size of worker process w is... The dynamic adjustment coefficient is The actual amount of work allocated is: = ; in This is the batch flexibility coefficient. This is a dynamic fairness factor (range 0.8~1.2). The dynamic parameter is adjusted based on factors including the historical throughput of the working process, the current system load, and the task priority.

[0026] 4. Credit Mechanism and Fair Scheduling: A credit-based fair scheduling mechanism is used. The credit value C of worker process w is... V The following evolutionary laws apply: ; in For credit recovery rate, For the target credit score, This is the timeout penalty coefficient. The timeout number is t, where t is the current scheduling round. This is the upper limit of the process's credit score. The credit score directly affects the priority of task allocation; the higher the credit score, the more tasks will be allocated to it.

[0027] In one embodiment, the construction and TorchScript format export of the residual attention network model includes: 1. Definition of Neural Network Model Structure The neural network input vector x is 7-dimensional and contains the following parameters: frequency f, salinity SSS, zenith angle θ, sea surface temperature SST, wind speed v, Fresnel vertical polarization emissivity ev_f, and Fresnel horizontal polarization emissivity eh_f. The neural network output vector y is 12-dimensional. The neural network employs a composite architecture, comprising a baseline prediction branch, an attention gating branch, an explicit interaction feature branch, and a residual correction branch. The hierarchical structure of each branch is defined as follows:

[0028] The input parameters are mapped to the [0,1] interval using the Min-Max normalization method.

[0029] 3. Attention gating mechanism formula: For the 7-dimensional input features, a self-attention mechanism is used to learn feature-level weights. The attention weight vector α = [α1, α2, ..., α7], and the weight parameters are learned through training; The element-wise product of the original input vector and the attention weight vector is used to concatenate the gated feature vector with the original feature vector.

[0030] 4. Explicit Interaction Feature Formula Generate all second-order product interaction terms using the second-order interaction mechanism of the factorization machine (FM) model: ; in Let be the latent vector representation of the i-th feature (obtained through training). Let k be the latent vector representation of the j-th feature, and k be the dimension of the latent vector. , These are the normalized values ​​of the i-th and j-th input features, respectively. Represents the dot product of implicit vectors. Scalar interaction strength; The cross term and the FM interaction term are concatenated to form a complete interaction feature vector.

[0031] 5. Residual Correction Network The original input, gated features, and interaction features are concatenated into the residual network input. The residual block uses a two-layer fully connected network. The final prediction is the sum of the baseline prediction and the residual correction. During the verification phase, the denormalized physical RMSE was used as the basis for early termination. Finally, call torch.jit.script to convert the PyTorch model to TorchScript format, and call torch.jit.save to save the TorchScript model as a .pt file.

[0032] In one embodiment, the standardized encapsulation of a cross-language interaction interface includes: 1. Fortran Interface Module Definition The ISO_C_BINDING internal module is imported into the Fortran source file to enable interoperability between Fortran and C; Declare the function signature corresponding to the C++ side in the INTERFACE block: Model initialization interface: SUBROUTINE init_iso_model(path) BIND(C, name='init_iso_model') Isotropic emissivity inference interface: SUBROUTINE compute_iso(freq, sss, zen, sst,wind, ev_iso, eh_iso) BIND(C, name='compute_iso') Anisotropic emissivity inference interface: SUBROUTINE compute_aniso(x7, y8) BIND(C, name='compute_aniso') 2. C++ middleware tier construction, including: Define global static device objects and tensor options: const torch::Device device(torch::kCPU) const torch::TensorOptions tensor_opts = torch::TensorOptions().dtype(torch::kFloat64).device(device) The torch::kCPU specifies the CPU device to be used, and torch::kFloat64 specifies the double-precision floating-point data type.

[0033] Using the `thread_local` storage class modifier to declare model instance variables ensures that each thread maintains the model state independently. thread_local torch::jit::script::Module g_model thread_local bool g_model_loaded = false Introduce a mutex lock to protect against race conditions during the initial load: std::mutex g_model_mutex 3. Model Delayed Loading Mechanism At the entry point of the inference function, check whether the model has been loaded; acquire the mutex lock and enter the critical section for model loading; Call torch::jit::load to load the model file from the specified path; switch the model to evaluation mode; set a flag after successful loading; if loading fails, catch the exception and set the failure flag.

[0034] 4. Data format conversion The double-precision scalar passed in from Fortran is constructed into a torch::Tensor object of shape [1].

[0035] Branching is performed based on the model's output format: if output.isTuple() is true, extract the values ​​of each element in the tuple; if output.isTensor() is true, extract the values ​​of each element in the tensor; for other formats, set the output to zero as a safe fallback.

[0036] In one embodiment, the initialization of the MPI parallel environment and the dynamic partitioning of the task pool include: 1. MPI Environment Initialization Initialize the MPI execution environment by calling the MPI_Init interface; obtain the unique identifier of the current process in the global communication domain through MPI_Comm_rank, where myid is the rank value of the current process, 0 for the master process and 1 to N-1 for worker processes; The total number of processes can be obtained through MPI_Comm_size: where nproc is the total number of processes, including 1 main process and nproc-1 worker processes.

[0037] 2. Master-slave process role division The master / slave role is determined based on the process's rank value: if myid equals 0, it is the master process (rank=0); if myid is greater than 0, it is the worker process (rank=1~N-1). The main process is responsible for task distribution, result collection, and I / O operations, and maintains the worker process state table. The worker processes are responsible for receiving tasks and calling cross-language interfaces to perform numerical calculations.

[0038] 3. Computational grid initialization Set the calculation range and number of sampling points for the azimuth angle: azimuth_range = [0°, 360°], n_azimuth = 361 Generate an azimuth array: azimuth(i) = (i-1) × (360 / (n_azimuth-1)), i = 1, 2, ..., n_azimuth Where n_azimuth is the number of azimuth sampling points, and azimuth(i) is the angle value (in degrees) of the i-th azimuth sampling point.

[0039] 4. Work process status management The main process allocates and initializes the worker_table, a table for managing worker process status, to track the running status of each worker process. Worker process status worker_state(w): 0 = Idle, 1 = Busy, 2 = Suspected fault, 3 = Terminated current_tasks(w): The number of currently assigned but incomplete tasks. Last active timestamp last_active_ts(w): The last time a message was received from this worker process. Completed_count(w): The total number of tasks completed by this worker process. error_count(w): The number of consecutive errors. Average response time (response_time_avg(w): The average task response time (in seconds) for this worker process. success_streak(w): The number of consecutive successful task completions. timeout_count(w): The total number of times the worker process has timed out. Historical batch record `batch_size_hist(w)`: A record of the batch size of the last 10 tasks. Priority score (w): A comprehensive score used for task priority allocation. Throughput score (w): A predicted throughput value based on historical data. Load index (w) is a dynamically calculated load status indicator (0~1). Credit value (w): Credit used to ensure fair scheduling of work processes. Last scheduled time last_schedule_ts(w): The timestamp of the last time this worker process was scheduled. Predicted_finish_ts(w): The task completion timestamp predicted based on the performance model.

[0040] 5. Adaptive Batch Adjustment Algorithm The base batch size is determined based on the total number of tasks and the number of processes: ; Where total_tasks is the total number of tasks, and nproc is the total number of processes. Set the base batch size (make sure it is at least 1).

[0041] The load factor is dynamically calculated based on the current number of tasks in the workflow, the predicted completion time, and historical performance. ; in =0.4、 =0.35、 =0.25 is the weighting coefficient, curr_load(w) is the current load, pred_load(w) is the predicted load, and hist_load(w) is the historical average load; ; ; ; The performance factor comprehensively considers response time, historical batch change rate, and throughput prediction. ; in =0.35、 =0.25、 =0.4 is the weighting coefficient. =5.0 seconds =2 is the smoothing time constant, response_time_avg(w) is the average response time, batch_size_var(w) is the standard deviation of the batch size change rate, and throughput_score(w) is the throughput score.

[0042] Throughput forecasting based on Exponentially Weighted Moving Average (EWMA): ; ; in =0.3 is the smoothing coefficient of EWMA, actual_throughput is the actual throughput (number of tasks / second), and prev_throughput is the previous predicted throughput.

[0043] Dynamically adjust batch size based on historical performance and current status of the workflow: = ; in The actual batch size for worker process w. As a dynamic adjustment factor, This is the batch elasticity coefficient. As the scheduling factor; ;

[0044] Batch elasticity coefficient ρ n (w) is calculated as follows: ; It is the variance of historical batch size in the work process w. It is the global average batch size; Scheduling factor The calculation is as follows: ; in =10 is the kurtosis parameter of the Sigmoid curve. Priority score for worker process w, scheduling factor Make batch allocation smoother;

[0045] 7. State Machine Transition Logic The main process scheduling is controlled by a main state machine, which includes five main states and their sub-states:

[0046] Each worker process maintains an independent sub-state machine:

[0047] Within each state of the main state machine, nested conditional branches are executed based on the state of the child state machines: When the main process reaches the scanning state, it first checks whether the time interval between the current time and the last scheduling time has reached the preset scheduling cycle threshold. If it has, it iterates through all worker processes numbered from 1 to the total number of processes minus one, updating worker process metrics and predicting throughput one by one. If the current worker process sub-state is idle, it calculates process priority and adjusts batch elasticity coefficients, and jumps to the large batch task distribution, regular batch task distribution, or small batch task distribution branch based on the relationship between the process priority value and the high-priority threshold and low-priority threshold. If the worker process sub-state is in the work execution state, it performs task timeout detection. Once a task timeout is detected, it dynamically updates the process timeout threshold and jumps to the timeout handling stage. If the worker process sub-state is in the response ready state, it directly jumps to the result collection stage. After completing the traversal of all worker processes, it ends the current round of scheduling cycle verification and scanning, waiting to enter the next state logic. The decision tree is as follows: Level 1: main_state decision → S_SCAN / S_DISPATCH / S_COLLECT / S_TERMINATE Level 2: Scheduling Period Determination → Scheduling Period T s If c expires / not yet expired, skip the scan. Layer 3: worker_scan_loop → iterates through worker_id = 1 to nproc-1 Level 4: Sub-state(w) determination → SS_FREE / SS_WORKING / SS_RESPONSE / SS_ERROR Level 5: priority_score(w) interval determination → HIGH / NORMAL / LOW / SUSPECT Level 6: Batch size adjust strategy selection → ACCELERATE / STABLE / DECELERATE / PAUSE Level 7: Dispatch execution branch → DISPATCH_LARGE / DISPATCH_NORMAL / DISPATCH_SMALL / SKIP The first layer determines the main state, classifying the system's operating state into four core main states: scanning, task distribution, result collection, and termination. The second layer determines the scheduling cycle; when a preset scheduling cycle threshold expires, the scheduling scan process is initiated; otherwise, the current scan is skipped to reduce unnecessary computational overhead. The third layer enters a worker process traversal loop, sequentially traversing all valid worker processes from 1 to the total number of processes minus one. The fourth layer determines the sub-state of the currently traversed worker processes, classifying them into four sub-states: idle, working, ready, and error, to differentiate operating modes. The fifth layer calculates priority scores based on the operating indicators of idle processes, classifying processes into four levels: high priority, normal priority, low priority, and suspicious priority, providing a basis for differentiated task allocation. The sixth layer selects a corresponding batch size adjustment strategy based on process priority level and load conditions, including four control modes: accelerated distribution, stable distribution, decelerated distribution, and paused distribution. The seventh layer executes the final task distribution branch according to the adjusted batch strategy, performing large-batch task distribution, regular batch task distribution, small-batch task distribution, or skipping distribution operations, thereby completing a refined and adaptive distributed task scheduling process. 8-Priority Scoring Dynamic Algorithm The priority score for the work process is calculated by combining the load factor, speed factor, quality factor, and fairness factor. ; in =0.35、 =0.25, αq=0.15, αf=0.25 are weighting coefficients, load_factor(w) is the load factor (0~1), speed_factor(w) is the speed factor (0~1), quality_factor(w) is the quality factor (0~1), and fairness_factor(w) is the fairness factor. The value of priority_score(w) ranges from 0 to 1.

[0048] The velocity factor is calculated based on the ratio of the response time to the global average, and time decay is introduced: ; Where min_response is the global minimum average response time, and max_response is the global maximum average response time. This is the time since the last scheduling. =60 seconds is the time decay constant. The faster the response, the higher the rate factor, but it decays over time.

[0049] The quality factor is calculated based on the error rate and success rate of the work process. ; in =0.4, =0.3, =0.3 is the weighting coefficient, accuracy_rate(w) is the accuracy rate, success_rate(w) is the success rate, and error_rate(w) is the error rate.

[0050] The fairness factor is calculated based on the credit score and scheduling history of the worker processes to ensure that all worker processes receive a relatively fair task allocation: ; Where credit(w) is the current credit value of worker process w, η=0.5 is the waiting time weighting coefficient, wait_time(w) is the waiting time since the last scheduling, and T max This is the longest waiting time.

[0051] The credit score of a work process is dynamically adjusted based on its performance. ); in =100 is the credit benchmark value. =20 represents the credit penalty for each failure, and λc=0.1 represents the credit recovery rate for each success. The credit score is capped at [...]. / 2, 2× Within the range.

[0052] Based on priority scores, the work processes are divided into four categories, and different distribution strategies are applied accordingly:

[0053] 9. Timeout and Fault Handling Mechanism The main process maintains an independent timer for each dispatched task and dynamically calculates the deadline: The timeout threshold is dynamically adjusted based on runtime and system load: ; in =30 seconds as the base timeout threshold The timeout decay coefficient (gradually decreases with runtime), load_factor(w) is the load factor of worker process w, and timeout_count(w) is the number of timeouts for the worker process (the more timeouts, the more lenient the threshold).

[0054] The timeout determination mechanism employs a combination of multi-level thresholds and dynamic weights. When a task's execution time exceeds its preset deadline, a retry weight is calculated based on the task's waiting time and the system load factor. If the task's retries are less than the maximum number of retries and the retry weight is higher than the retry threshold, the task is marked as retrying, the original worker process's timeout history is updated, the original worker process is excluded through priority filtering, a target worker process is selected, the task batch size is adjusted according to the number of retries, and the tasks are redistributed to the target process. Then, the process returns to the scanning phase. If the task's execution time exceeds a severe timeout threshold, the task is directly marked as failed. The timeout count of the original working process is accumulated and the corresponding credit value is deducted, and the process jumps directly to the result collection stage. When the system detects that a working process is in a suspicious abnormal state, all orphan tasks assigned to that process are immediately obtained, the process is marked as failed, all orphan tasks are traversed, sorted by priority, and after excluding faulty processes, qualified working processes with credit values ​​greater than half of the benchmark credit value and in an idle state are selected. The optimal available working process is selected based on priority and load, and individual tasks are distributed to the selected target working process one by one. The status of the target process is updated to working, thereby realizing the dynamic transfer and fault-tolerant scheduling of faulty process tasks.

[0055] 10. Static partitioning of the task pool Divide the azimuth azimuth range evenly into N parts: Divide the azimuth azimuth range evenly into N parts: ; ; Where rank is the number of the working process, and azimuth_lo and azimuth_hi are the start and end angles of the azimuth range, respectively; Number of sampling points handled by each worker process: ; Where n_azimuth is the total number of azimuth sampling points, N is the number of working processes, and n_local is the number of sampling points allocated to each working process; Each worker process constructs its corresponding local azimuth subarray: .

[0056] In one embodiment, a distributed inference execution module that integrates cross-language computing performs the following steps: After the main process enters the task distribution state, it begins to traverse each worker process sequentially, executing branch logic based on the current worker process's sub-running state and scheduling context. If the currently traversed worker process is in an idle state, its running priority and allocable task batch are calculated sequentially, credit value constraints are applied, and fairness scheduling corrections are performed. When the calculated allocable task batch is greater than zero, the worker process's last scheduling time is updated, completing the current round of process processing and continuing to scan the next worker process. If the current worker process is in a work execution state, task timeout detection and status updates are performed. Once a task timeout is detected, the timeout threshold is dynamically updated, and the throughput prediction model is refreshed. After timeout processing is completed, the next worker process is scanned. If the current worker process is in a result response waiting state, it directly jumps to the result collection stage. After processing each worker process, the scanning pointer is incremented by one position. When the scanning pointer exceeds the total number of worker processes, it is reset to the starting position, the system's global running statistics are updated, and the process jumps to the result collection stage.

[0057] The batch size is dynamically calculated and distributed based on the current workflow, taking into account multiple factors: ; N_dispatch = min(N_dispatch, remaining_tasks) / / Ensure that the number of remaining tasks does not exceed the total number of tasks. N_dispatch = max(N_dispatch, 1) / / Ensure at least one task is dispatched in This is a retry task correction factor. If a task batch includes retry tasks, the batch size should be appropriately reduced to mitigate the risk.

[0058] The main process constructs and sends an array of task identifiers of length N_dispatch: Update the task status array and worker process status table.

[0059] 2. Task reception on the worker process side The worker process calls MPI_Recv and blocks while waiting for a task message from the main process. Determine if N_tasks is equal to -1: If N_tasks is equal to -1, it is considered a termination signal and the execution exits the sequence; if N_tasks is greater than 0, it is considered a valid task batch and execution continues.

[0060] The worker process calls MPI_Recv to receive the task identifier array: Store the received task identifier in the local task queue and update the sub-state machine.

[0061] 3. Calculation and execution on the work process side The workflow enters a task loop, processing the received tasks one by one: Decode the task identifier task_id and calculate the corresponding azimuth range: azimuth_idx = task_id / / Azimuth index corresponding to task ID azimuth_start = azimuth_lo + (azimuth_idx-1) × step / / Calculate the starting angle azimuth_end = azimuth_start + step / / Calculate the ending angle The worker process calls the cross-language inference interface compute_emissivity to store the computation results into a local result array: 4. Main process side result recycling The main process calls MPI_Probe to probe the message queue: Determine the message type based on the returned MPI_TAG field, and proceed to the corresponding processing branch: If it is a result ready message, process the result ready message; if it is a worker process status report, process the worker process status report; if it is a task batch return (abnormal), handle the abnormal situation.

[0062] The main process checks the termination condition "all tasks have been completed," and then the main state machine enters the termination state. The main process iterates through all worker processes and sends a termination signal.

[0063] Output the statistics for this scheduling: total number of tasks dispatched, total number of tasks completed, number of failed tasks, average response time, average throughput, retry rate, fairness index, and scheduling efficiency.

[0064] All processes call MPI_Finalize to exit the parallel environment.

[0065] The parallel speedup and efficiency metrics used in the experiments in this application are as follows:

[0066] Compared with the pure fully connected MLP method, this application uses a residual attention network with an attention mechanism to achieve batch inference, provides native integration, and offers MPI parallel scheduling with dynamic task scheduling, which can support large-scale parallelism and achieve large-scale computational expansion.

[0067] The beneficial effects of this invention are as follows: A residual attention network architecture is proposed, which organically integrates baseline prediction, attention gating, explicit interaction and residual correction, thereby improving the accuracy of sea surface emissivity calculation. The design is based on the ISO_C_BINDING cross-language interoperability standard, which enables seamless integration of the Python training environment with the Fortran numerical prediction program, reducing the model update cycle from weeks to hours. A nested state machine-based MPI parallel scheduling framework is constructed to support dynamic task allocation, credit mechanism and fault recovery, meeting the engineering deployment requirements in HPC environment; The concepts of batch elasticity coefficient and dynamic fairness coefficient are proposed to achieve adaptive balancing of workload and improve the system efficiency of large-scale parallel computing.

[0068] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for calculating sea surface emissivity based on cross-language parallel scheduling, characterized in that, Includes the following steps: The residual attention network model module normalizes the input physical parameters and extracts features. The residual attention network contains four parallel branches: baseline prediction branch, attention gating branch, explicit interactive feature branch, and residual correction branch. The baseline prediction branch adopts a standard multilayer perceptron structure and learns the nonlinear mapping relationship from the input physical parameters to the sea surface emissivity. The attention-gated branch learns the weight distribution of input features through a self-attention mechanism; the explicit interaction feature branch uses a factorization machine mechanism to capture the second-order interaction effect between input features; and the residual correction branch uses a deep residual learning framework to learn fine-grained corrections based on baseline predictions. A cross-language interactive interface connects the Python training environment with the Fortran numerical prediction program. The preprocessed features are then used by the C++ inference engine. The C++ engine calls the TorchScript model to perform forward computation, and the computation results are returned to the Fortran main program through the interface. The MPI parallel scheduling module adopts a master-slave architecture and uses a state machine to implement task distribution and result collection logic. In the distributed inference execution module, the main process dynamically calculates the batch of tasks to be processed by each working process and completes the allocation and distribution. After receiving the task, each subprocess decodes the task identifier, matches it to obtain the corresponding azimuth interval, calls the cross-language inference interface to complete the numerical calculation of sea surface emissivity, and stores the complete set of sea surface emissivity calculation results, including isotope emission components and anisotropy correction coefficients, into the local result array.

2. The efficient sea surface emissivity calculation method based on cross-language parallel scheduling according to claim 1, characterized in that, For the input feature vector is It includes seven features: frequency f, salinity SSS, zenith angle θ, sea surface temperature SST, wind speed v, Fresnel vertical emissivity ev_f, and Fresnel horizontal emissivity eh_f. The mathematical expression for baseline prediction by the baseline prediction branch is: ； in , , This is the weight matrix. , , For the corresponding bias vector; The formula for calculating the attention gating branch is: ； in This is the attention weight matrix. The bias vector, the softmax(·) function normalizes the real vector into a probability distribution, and the attention weight vector is... ; Gated feature vector ,in This represents element-wise multiplication; The fused feature vector is ; The formula for calculating the second-order interaction term of the explicit interaction feature branch is: ； in Let be the latent vector representation of the i-th feature. Let k be the latent vector representation of the j-th feature, and k be the dimension of the latent vector. , These are the normalized values ​​of the i-th and j-th input features, respectively. The residual calculation process for the residual correction branch is as follows: ； The input to the residual network is , and These are the weight matrix and bias vector of the output layer, respectively. To ensure that the network performance does not degrade in the early stage of training, the output layer adopts zero initialization constraint; the final network output is the sum of the baseline prediction and the residual correction. The training loss function adopts the weighted MSE loss function based on physical dimensions, and the physical RMSE is used as the basis for early stopping during the validation phase.

3. The efficient sea surface emissivity calculation method based on cross-language parallel scheduling according to claim 1, characterized in that, The cross-language interaction interface adopts a three-layer architecture: the C++ wrapper layer is responsible for encapsulating TorchScript inference calls, the Fortran-C interoperability layer is responsible for parameter type conversion and memory management, and the Fortran scheduling layer is responsible for task orchestration and result processing. The C++ wrapper layer loads TorchScript models and executes inference through the libtorch C++ API; the Fortran-C interoperability layer uses the ISO_C_BINDING standard to implement lock-free memory sharing; the Fortran scheduling layer passes array parameters by value or reference, and the C++ side uses pointers to receive them. The memory layout adopts a column-major strategy to match the Fortran convention.

4. The efficient sea surface emissivity calculation method based on cross-language parallel scheduling according to claim 1, characterized in that, In the MPI parallel scheduling module, the main process uses a state machine to maintain a conditional jump structure to implement the scheduling decision logic. The state machine is defined as follows: S_MAIN ∈ {S_IDLE, S_SCAN, S_DISPATCH, S_COLLECT, S_TERMINATE} S_IDLE is the initialization state, S_SCAN is responsible for scanning the working process status, S_DISPATCH performs task distribution, S_COLLECT collects calculation results, and S_TERMINATE is the termination state. Each worker process maintains an independent sub-state machine: ； The state transitions of the worker process state machine are driven by the control messages of the main process and the local computation completion event. SS_FREE is the idle state, SS_WORKING is the working state, which performs cross-language computation, SS_RESPONSE is the response state, which sends the result ready message, and SS_WAIT is the waiting state. The MPI parallel scheduling module uses a dynamic task allocation strategy based on throughput prediction to allocate tasks. The current batch size of worker process w is... The dynamic adjustment coefficient is The actual amount of work allocated is: = ； in This is the batch flexibility coefficient. The dynamic fairness coefficient is adjusted based on factors including the historical throughput of the working process, the current system load, and the task priority. The MPI parallel scheduling module uses a credit-based fair scheduling mechanism, where worker process w has a credit value C. V Follow these rules: ； in For credit recovery rate, For the target credit score, This is the timeout penalty coefficient. The timeout number is t, where t is the current scheduling round. This is the upper limit of the process credit value. The credit value directly affects the task allocation priority. The higher the credit value, the more tasks will be allocated to it.

5. The efficient sea surface emissivity calculation method based on cross-language parallel scheduling according to claim 1, characterized in that, The adaptive batch adjustment algorithm in the MPI parallel scheduling module is as follows: The base batch size is determined based on the total number of tasks and the number of processes: ； Where total_tasks is the total number of tasks, and nproc is the total number of processes. Base batch size; The load factor is dynamically calculated based on the current number of tasks in the workflow, the predicted completion time, and historical performance. ； in =0.4、 =0.35、 =0.25 is the weighting coefficient, curr_load(w) is the current load, pred_load(w) is the predicted load, and hist_load(w) is the historical average load; ； ； ； It represents the number of tasks currently being executed by the worker process w. This represents the maximum number of tasks that the worker process w can handle. The worker process w predicts the task completion timestamp. It is the current system timestamp. It is the global average execution time per task. is the historical batch size of worker process w in the i-th round, where i is the index of the historical scheduling round; The performance factor comprehensively considers response time, historical batch change rate, and throughput prediction. ； in =0.35、 =0.25、 =0.4 is the weighting coefficient. =5.0 seconds =2 is the smoothing time constant, response_time_avg(w) is the average response time, batch_size_var(w) is the standard deviation of the batch size change rate, and throughput_score(w) is the throughput score. Throughput prediction based on exponentially weighted moving average is calculated as follows: ； ； in `EWMA` is the smoothing coefficient for the exponentially weighted moving average (EWMA). `actual_throughput` is the actual throughput, and `prev_throughput` is the previous predicted throughput. This is the size of the completed task batch. It is the time elapsed since the last scheduling; Dynamically adjust batch size based on historical performance and current status of the workflow: = ； in The actual batch size for worker process w. As a dynamic adjustment factor, This is the batch elasticity coefficient. is the scheduling factor, where Adjusted by state factor Load adjustment factor Response adjustment factor Error adjustment factor and credit adjustment factor The product consists of: ； Batch elasticity coefficient ρ n (w) is calculated as follows: ； It is the variance of historical batch size in the work process w. It is the global average batch size; Scheduling factor The calculation is as follows: ； in =10 is the kurtosis parameter of the Sigmoid curve. Priority score for worker process w, scheduling factor This makes batch allocation smoother.

6. The efficient sea surface emissivity calculation method based on cross-language parallel scheduling according to claim 5, characterized in that, In the MPI parallel scheduling module, nested conditional branches are executed within each state of the main state machine based on the state of the sub-state machine: When the main process reaches the scanning state, it first checks whether the time interval between the current time and the last scheduling time has reached the preset scheduling cycle threshold. If the scheduling cycle has been reached, it iterates through all worker processes numbered from 1 to the total number of processes minus one, performing worker process metric updates and throughput prediction operations one by one. If the current worker process sub-state is idle, it sequentially completes process priority calculation and batch elasticity coefficient adjustment, and jumps to the large batch task distribution, regular batch task distribution, or small batch task distribution branch according to the relationship between the process priority value and the high priority threshold and low priority threshold. If the worker process sub-state is in the work execution state, it performs task timeout detection. Once a task timeout is determined, it dynamically updates the process timeout threshold and jumps to the timeout handling stage. If the worker process sub-state is in the response ready state, it directly jumps to the result collection stage. After completing the traversal of all worker processes, it ends the current round of scheduling cycle verification and scanning state, and waits to enter the next state logic. The decision tree's first layer determines the main state, classifying the system's operating state into four core main states: scanning state, task distribution state, result collection state, and termination state. The second layer determines the scheduling cycle; when a preset scheduling cycle time threshold expires, the scheduling scan process is initiated; otherwise, the current scan is skipped to reduce unnecessary computational overhead. The third layer enters a worker process traversal loop, sequentially traversing all valid worker processes from 1 to the total number of processes minus one. The fourth layer determines the sub-state of the currently traversed worker process, classifying it into four sub-states: idle, working, ready to respond, and abnormal / error-related, to differentiate operating modes. The fifth layer calculates priority scores based on the operating indicators of idle processes and classifies processes into four levels: high priority, normal priority, low priority, and suspicious priority, providing a basis for differentiated task allocation. The sixth layer selects the corresponding batch size adjustment strategy based on process priority and load, including four control modes: accelerated distribution, stable distribution, decelerated distribution, and paused distribution. The seventh layer executes the final task distribution branch according to the adjusted batch strategy, performing large batch task distribution, regular batch task distribution, small batch task distribution, or skipping distribution operations respectively, thereby completing the entire process of fine-grained and adaptive distributed task scheduling.

7. The efficient sea surface emissivity calculation method based on cross-language parallel scheduling according to claim 6, characterized in that, In the MPI parallel scheduling module, the priority score of the working process is calculated by combining the load factor, speed factor, quality factor, and fairness factor. ； in =0.35、 =0.25、 =0.15、 =0.25 is the weight coefficient, load_factor(w) is the load factor, speed_factor(w) is the speed factor, quality_factor(w) is the quality factor, fairness_factor(w) is the fairness factor, and priority_score(w) is the priority factor; The velocity factor is calculated based on the ratio of the response time to the global average, and time decay is introduced: ； Where min_response is the global minimum average response time. This is the average response time, and `max_response` is the global maximum average response time. This is the time since the last scheduling. =60 seconds is the time decay constant; The quality factor is calculated based on the error rate and success rate of the work process. ； in =0.4, =0.3, =0.3 is the weighting coefficient, accuracy_rate(w) is the accuracy rate, success_rate(w) is the success rate, and error_rate(w) is the error rate; The fairness factor is calculated based on the credit score and scheduling history of the worker processes to ensure that all worker processes receive a relatively fair task allocation: ； Where credit(w) is the current credit value of worker process w, η=0.5 is the waiting time weighting coefficient, wait_time(w) is the waiting time since the last scheduling, and T max This is the longest waiting time. The credit score of a work process is dynamically adjusted based on its performance. )； in =100 is the credit benchmark value. =20 is the credit penalty for each failure. =0.1 represents the credit recovery rate per success, failure_count(w) is the total number of failures in worker process w, and success_count(w) is the total number of successes in worker process w. The credit value is limited to [ , 2× Within the range; Based on priority scores, the work processes are divided into four categories, and different distribution strategies are applied accordingly: The rating range is For scores ≤ 1.0, the HIGH_PRIORITY category is selected, and the distribution strategy prioritizes large batches, with adjustments made in batches. The rating range is ≤score≤ For the NORMAL_PRIORITY category, the distribution strategy is normal batch distribution, with batch adjustment of N × 1.0, and the score range is 0 ≤ score ≤ For the LOW_PRIORITY category, the distribution strategy is to distribute in small batches and adjust in larger batches. If error > 0 at any time, the system is classified as SUSPECT, and the distribution strategy is to pause distribution. It is the process load weight. It is the historical performance weight of the process. It is the weight of the urgency of the mission joining the team. It is the task execution efficiency weight.

8. The efficient sea surface emissivity calculation method based on cross-language parallel scheduling according to claim 7, characterized in that, The main process maintains an independent timer for each dispatched task and dynamically calculates the deadline; the timeout threshold is dynamically adjusted based on the runtime and system load. ； in =30 seconds as the base timeout threshold Here, load_factor(w) is the load factor of worker process w, and timeout_count(w) is the number of timeouts for that worker process. The timeout determination mechanism employs a combination of multi-level thresholds and dynamic weights. When a task's execution time exceeds its preset deadline, a retry weight is calculated based on the task's waiting time and the system load factor. If the task's retries are less than the maximum number of retries and the retry weight is higher than the retry threshold, the task is marked as retrying, the original worker process's timeout history is updated, the original worker process is excluded through priority filtering, a target worker process is selected, the task batch size is adjusted according to the number of retries, and the tasks are redistributed to the target process. Then, the process returns to the scanning phase. If the task's execution time exceeds a severe timeout threshold, the task is directly marked as failed. The timeout count of the original working process is accumulated and the corresponding credit value is deducted, and the process jumps directly to the result collection stage. When the system detects that a working process is in a suspicious abnormal state, all orphan tasks assigned to that process are immediately obtained, the process is marked as failed, all orphan tasks are traversed, sorted by priority, and after excluding faulty processes, qualified working processes with credit values ​​greater than half of the benchmark credit value and in an idle state are selected. The optimal available working process is selected based on priority and load, and individual tasks are distributed to the selected target working process one by one. The status of the target process is updated to working, thereby realizing the dynamic transfer and fault-tolerant scheduling of faulty process tasks.

9. The efficient sea surface emissivity calculation method based on cross-language parallel scheduling according to claim 8, characterized in that, The static partitioning of the task pool includes: Divide the azimuth azimuth range evenly into N parts: ； ； Where rank is the number of the working process, and azimuth_lo and azimuth_hi are the start and end angles of the azimuth range, respectively; Number of sampling points handled by each worker process: ； Where n_azimuth is the total number of azimuth sampling points, N is the number of working processes, and n_local is the number of sampling points allocated to each working process; Each worker process constructs its corresponding local azimuth subarray: 。 10. The efficient sea surface emissivity calculation method based on cross-language parallel scheduling according to claim 9, characterized in that, The distributed inference execution module performs the following steps: After the main process enters the task distribution state, it begins to traverse each worker process sequentially, executing branch logic based on the current worker process's sub-running state and scheduling context. If the currently traversed worker process is in an idle state, its running priority and allocable task batch are calculated sequentially, credit value constraints are applied, and fairness scheduling corrections are performed. When the calculated allocable task batch is greater than zero, the worker process's last scheduling time is updated, completing the current round of process processing and continuing to scan the next worker process. If the current worker process is in a work execution state, task timeout detection and status updates are performed. Once a task timeout is detected, the timeout threshold is dynamically updated, and the throughput prediction model is refreshed. After timeout processing is completed, the next worker process continues to be scanned. If the current worker process is in a result response waiting state, it directly jumps to the result collection stage. After processing each worker process, the scanning pointer is incremented by one position. When the scanning pointer exceeds the total number of worker processes, the scanning pointer is reset to the starting position, the system's global running statistics are updated, and the process jumps to the result collection stage. The batch size is dynamically calculated and distributed based on the current workflow, taking into account multiple factors: ； in The retry task correction factor is set up by task_batch, which is the current batch of tasks to be dispatched, and remaining_tasks, which is the total number of remaining uncalculated tasks globally. If the task batch contains retry tasks, the batch size is reduced to mitigate the risk.

Images