Efficient parallel data extraction and preprocessing method, model and medium for nuclear emergency

Abstract

The invention discloses a high-efficiency parallel data extraction and preprocessing model for nuclear emergency, which realizes high-efficiency preprocessing of meteorological and ocean data in a GRIB format by constructing a parallel data processing method based on task fragmentation. Firstly, an original GRIB file is logically divided through a data fragmentation step to form task fragments capable of being independently processed, and then the task fragments are uniformly distributed to a local task queue through a task initialization step in combination with multi-core system resources. Furthermore, a core emergency parallel task stealing algorithm is adopted, and cross-processing-unit task collaboration and stealing scheduling are realized on the basis of local priority execution, so that the resource utilization rate and the task throughput capacity of the multi-core system are improved. In the whole process, an error monitoring and fault-tolerant mechanism is combined, automatic rescheduling and retry can be carried out when processing fails, stability and integrity of data processing are guaranteed, and processing results are rearranged into structured output according to a time sequence and a space structure by means of a result integration step.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of large-scale data parallel processing, and particularly relates to a high-efficiency parallel data extraction and preprocessing method and model for nuclear emergency. BACKGROUND

[0002] After a nuclear accident occurs, in order to evaluate the diffusion behavior of radioactive substances in the atmosphere and the ocean and develop a scientific and effective emergency response plan, high-resolution and time-sensitive meteorological and ocean data must be relied on as input to provide boundary conditions and initial conditions for subsequent atmospheric and seawater transport and diffusion simulation models.

[0003] As an internationally accepted standard format for meteorological and ocean data, GRIB (GRIdded Binary) has the advantages of compact structure and rich data types, and is widely used in global and regional weather forecasting, ocean prediction, environmental monitoring and disaster response systems. GRIB data contains key variables such as wind speed, temperature and humidity, and pressure, and is an indispensable data source for nuclear emergency analysis.

[0004] However, due to the large size and complex structure of GRIB data files, and the highly compressed encoding mechanism, a large amount of computing resources is required for decoding and preprocessing in actual applications to extract the required variables and convert them into a data format recognizable by numerical simulation models. In the context of nuclear emergency response, the delay in data processing time will directly affect the timeliness and accuracy of the pollutant diffusion prediction results, thereby negatively affecting the scientificity of emergency decision-making. Therefore, efficient, stable and fault-tolerant preprocessing of GRIB data is one of the key prerequisites for ensuring the efficiency of nuclear accident emergency response.

[0005] In the prior art, GRIB data decoding and preprocessing often uses a serial processing mode or parallel processing based on a simple static task allocation strategy. These methods have the following shortcomings when faced with large-scale data: Low processing efficiency: traditional serial processing methods are slow in parsing GRIB data, and cannot meet the demand for minute-level or even second-level response speed in the context of nuclear accidents; although some parallel schemes introduce multi-threading mechanisms, due to uneven task allocation, resource utilization may be low and processing bottlenecks may occur.

[0006] Lack of dynamic load balancing mechanism: most parallelization schemes use static task allocation strategies and cannot dynamically adjust task allocation according to the actual running state of each processing unit, resulting in idle computing cores and overloading of some cores, and the overall performance of the system cannot be optimized.

[0007] Lack of fault tolerance for data anomalies: In high concurrency processing conditions, processing units may fail due to memory overflow, format errors or system failures. Existing solutions often use interruption or manual intervention methods, which are difficult to achieve automatic fault tolerance and recovery, reducing system stability.

[0008] Inadequate compatibility for multi-version GRIB format: As the format of meteorological and oceanographic data continues to evolve, different versions of GRIB files differ in storage structure, variable naming and compression methods. Current tool model chains or algorithms have poor compatibility for different versions, often requiring manual parameter adjustment or additional scripts to adapt, increasing system operation and maintenance costs.

[0009] Lack of flexible model interface adaptation capability: Decoded data often needs to be further converted into a format acceptable to numerical weather prediction and diagnostic models (such as WRF, CALMET, CFD, etc.). Existing methods often lack a unified interface and standardized output mechanism, making it difficult to meet the actual needs of multi-model, multi-task collaborative computing.

[0010] Therefore, there is an urgent need for a GRIB data parallel decoding and preprocessing technology solution for nuclear emergency application scenarios, with high efficiency, strong adaptability, high robustness and good scalability. The ideal solution should support multi-dimensional data slicing, dynamic scheduling, task stealing, automatic fault tolerance, format standardization output, and be able to adapt to a variety of hardware platforms including multi-core CPUs, GPUs and distributed clusters, ultimately providing stable, efficient and scalable data support for nuclear accident consequence simulation and emergency decision-making systems.

[0011] The present invention is proposed in the above background, aiming to solve the real-time, adaptability and stability problems of GRIB data processing in nuclear emergency applications, and proposes a high-efficiency parallel data extraction and preprocessing model and method for nuclear emergency. By constructing a modular task scheduling structure and an adaptive data processing strategy, the processing performance and system reliability are significantly improved. SUMMARY

[0012] The present invention aims to provide a high-efficiency parallel data extraction and preprocessing method and model for nuclear emergency, which can solve the above problems by constructing a modular task scheduling structure and an adaptive data processing strategy, significantly improving the processing performance and system reliability. In addition, the present invention has good heterogeneous platform adaptation capability and can stably run on multi-core CPUs, GPU acceleration cards, FPGAs and other hardware environments and edge computing nodes to meet the deployment needs of high-intensity computing in nuclear emergency scenarios.

[0013] The present invention discloses a high-efficiency parallel data extraction and preprocessing method for nuclear emergency, comprising the following steps: Step S1, data slicing: the input GRIB format meteorological or ocean data file is parsed to extract its time step, variable type and spatial grid information, and logically divided according to time, space and variable dimensions to form multiple independently processable task fragments; in the data slicing process, the system constructs a three-dimensional index system containing time, space and variable dimensions, and defines independent task fragments according to the combination of each time step, spatial region and variable type. Each task fragment has an independent data reading boundary, and the decoding processes of the processing units do not affect each other. At the same time, the system dynamically adjusts the task fragment granularity when dividing tasks by combining factors such as historical processing data, task size and data density, to ensure that the subsequent processing tasks are reasonably distributed in the multi-core system and do not produce bottlenecks.

[0014] Step S2, task initialization: according to the number and state of the multi-core processing units available in the system, the task fragments are initialized and distributed to the corresponding local task queues, and a task index information table is constructed for each task, which includes the offset address, time period, variable type, spatial range, target processing unit number and task state of the task fragment; preferably, a load estimation strategy is used during task initialization to avoid overloading or idling of a single processing unit; the constructed task index information records the starting position, data volume, variable type and current state of the task fragment, and is stored in shared memory or a scheduling database to support subsequent dynamic scheduling and fault recovery.

[0015] Step S3, parallel task scheduling: based on the improvement of the NEPTSA task stealing algorithm, the present application introduces the PPO deep reinforcement learning method to optimize the NEPTSA task stealing strategy online / offline: each processing unit executes the task fragment from the head of the local task queue first, and when the local queue is empty, the scheduler encodes the current cluster state (CPU / GPU queue length, near-accident space-time high-priority fragment ratio, fragment average time consumption estimation, device type and performance vector, etc.) into a state vector, inputs it into the PPO policy network trained to output the "stealing source processing unit identifier" and "stealing granularity parameter" (corresponding to the number / ratio of fragments stolen from the tail of the source queue). The scheduler acquires the task fragment that has not started from the tail of the target processing unit queue according to this, and dynamically adjusts the granularity of the subsequent fragments in combination with the execution feedback (actual time consumption of the fragment, whether the target unit produces backlog, change of system idle rate). If it is detected that the granularity is too large to cause overload or too small to cause frequent scheduling overhead, the strategy network or heuristic fallback rule automatically corrects the granularity of the next stealing. This process is executed in a loop among multiple processing units to maximize resource utilization and load balancing, and minimize the makespan; The preset priority is determined according to the characteristics of the nuclear accident, including: ① priority is given to the time close to the occurrence of the nuclear accident, ② priority is given to the space near the nuclear accident point, ③ priority is given to the wind direction, temperature, humidity, potential height and precipitation related parameters that have the greatest impact on the spread of the nuclear accident; the process dynamically adjusts the task granularity in combination with the task processing feedback, and realizes the load balancing among multiple cores; during the scheduling process, a task execution time feedback mechanism is introduced, and the granularity of the subsequent task fragments is dynamically adjusted according to the actual processing time of different task types. For example, if a certain type of task (multi-height layer physical quantity analysis) takes a long time to process, the system will further refine this type of task into smaller fragments (split height layer) in subsequent division to improve parallel processing efficiency. Through the above dynamic scheduling strategy, the maximum utilization of processing resources is realized, and the phenomenon of long-term idling or overloading of some processing units is avoided.

[0016] Step S4, data preprocessing: each processing unit decodes, formats and extracts variables for the allocated task fragments to generate intermediate data; specifically including: calling the GRIB decoder to analyze the corresponding data segment for the task fragment; extracting the target meteorological or oceanic variables and performing necessary unit conversion or physical quantity conversion; storing the extracted variable data in a standard intermediate data format (such as a structured array or a temporary NetCDF file).

[0017] Step S5, error monitoring and fault tolerance: real-time monitoring of task status and resource usage during task execution, if an exception or task failure is detected, the task status is marked as failed and rescheduled to other idle processing units for retry, a maximum of three retries, and failure information is recorded in the error log; to improve system stability, the scheduling system also monitors decoding results, execution time, and memory overflow, file reading errors and other abnormal conditions. Once a task fails, the system automatically selects a currently idle or low-load processing unit to re-execute the task. The fault tolerance mechanism allows a maximum of three retries for failed tasks, and records error information (task number, failure reason, processing unit ID, retry number, etc.) to the central error log for post-performance analysis and optimization.

[0018] Step S6, result integration: reorganize all the intermediate data according to the time sequence and spatial structure of the original data, and integrate to generate a structured output file, which can be directly called by an environmental simulation model and supports dynamic configuration of formats such as NetCDF, Json, HDF5 or CSV. Among them, the result reorganization sorts the intermediate data according to the original time, space and variable order, ensuring that the spatio-temporal correspondence of the output data is consistent with the source data. The integrated result file can be output in formats such as NetCDF, Json, HDF5 or CSV according to user configuration, and the data format and variable naming comply with the interface specifications of WRF, CALMET, CFD and other meteorological and ocean prediction and diagnosis models, realizing seamless docking and calling of the model.

[0019] Further, the data slicing step adopts a three-dimensional task mapping mechanism to divide the data; by constructing a three-dimensional data model containing time index, geographic grid coordinates and variable type dimensions, the GRIB meteorological and ocean data are mapped into several task fragments with independent processing boundaries, and the data amount contained in each task fragment is dynamically adjusted in combination with historical processing time consumption distribution, data density and spatial continuity and other factors, to ensure the balance and scheduling friendliness of the task granularity.

[0020] Further, the task initialization step includes "unprocessed", "processing", "completed" and "failed", and the system realizes tracking, scheduling and recovery of tasks through task index state. Further, the NEPTSA parallel task stealing algorithm adopts a tail-first stealing strategy: when a processing unit has executed all the task fragments in its local task queue, it acquires a task fragment that has not started execution from the tail of the task queue of another processing unit according to a preset priority; the algorithm schedules the fragments according to the following priority selection rules based on the characteristics of nuclear accidents: ① similar time before and after the nuclear accident, ② space near the nuclear accident point, ③ wind direction, temperature, humidity, potential height and precipitation related parameters that have the greatest impact on the diffusion of the nuclear accident; and introduces a deep reinforcement learning scheduling strategy based on proximal policy optimization (PPO), so that the stealing decision outputs "stealing source processing unit" and "stealing granularity" at the same time, and dynamically adjusts the granularity of the stolen task according to the actual execution time consumption feedback of each task fragment, avoiding overloading or repeated processing of the processing unit due to improper task size, thereby improving the parallel efficiency and load balancing level of the overall system.

[0021] The scheduler is a core control module in the data processing system of the application, responsible for the unified arrangement of the allocation, migration (stealing), granularity adjustment and end integration of "task fragments" on a heterogeneous computing cluster (CPU / GPU hybrid). It aims to minimize the overall completion time (makespan) and maximize load balancing, and makes scheduling decisions by combining "rule constraints (tail priority, core accident space-time and key factor priority)" and "AI strategy PPO"; The state vector of the scheduling strategy includes: the device type and performance vector of each processing unit, the queue length, the statistical summary of the tail stolen fragments in the accident space-time proximity and the key factor, the fragment size and the estimated time consumption, and the system idle rate and the load imbalance degree; The PPO strategy is trained in a simulation environment: the simulation environment randomly generates task fragments according to the core accident time and point, and sets the priority and initial allocation of the fragments according to the accident space-time proximity and key factor weight. In the training process, the negative value of the total completion time is used as the terminal reward, and the step-by-step rewards reflecting the reduction of idle rate, the improvement of balance degree and the migration overhead penalty are added. The clipped policy gradient and generalized advantage estimation are used to update the policy and value network parameters.

[0022] Further, the intelligent parallel task scheduling can optimize the scheduling strategy, and the application can also integrate other AI assisted scheduling algorithms: using other machine learning models to predict the optimal task allocation scheme according to the execution time of historical tasks, queue length change and other data. For example, a deep learning classification or other reinforcement learning agent model can be deployed to learn the impact of different scheduling actions on system throughput and latency online, so as to make intelligent decisions on task stealing and task redistribution, and reduce unnecessary waiting and competition. This AI assisted mechanism enables the scheduling strategy to be continuously optimized with dynamic changes in workloads, improving the real-time response performance of the system in complex situations.

[0023] Further, the deep reinforcement learning model in the application is realized by (PPO, Proximal Policy Optimization). PPO algorithm can exhibit significant advantages in multi-core heterogeneous computing environment. The agent (here referring to the scheduler) constructed by PPO algorithm can dynamically interact with the environment and continuously adjust and optimize the strategy according to the reward feedback, adapt to the performance difference and load change of different resource nodes. PPO algorithm uses neural network to approximate the state-action value function to cope with the high-dimensional complexity of the scheduling state space. Through this deep reinforcement learning model, the scheduler can learn effective task stealing decision strategy for complex core emergency scenarios, that is, intelligent decision making when and from which processing unit to steal how many task fragments to minimize the overall completion time and maintain load balancing. The specific model element design is as follows: State: State describes the load and task queue situation of each processing unit in the current heterogeneous computing cluster, including the type of each processing unit (CPU or GPU), the length of the local task queue, and the attribute summary of the task fragments in the queue, etc. For example, the remaining task fragment number of each processing unit, the average or highest priority of its task fragments (based on accident spatiotemporal proximity and key factors), and the computing performance index of the processing unit can be represented by a vector or matrix. Through state characterization, the intelligent scheduler can perceive the load distribution and key task fragment location of the entire system; Action: Action defines the decision of the scheduler, including selecting which processing unit as the source of stealing and deciding the size (number or granularity) of the stolen task fragments. The tail-priority stealing strategy means that the action generally initiates from the tail of the task queue of a busy processing unit. In order to simplify, the action space can be designed as several discrete options: for example, "steal k task fragments from the tail of processing unit i". Through this action definition, the agent can decide both the stealing object and the stealing workload, realizing the joint optimization of load balancing and task granularity; Reward: The reward signal encourages the scheduler behavior to shorten the overall computation time and optimize the load balancing. Specifically, the negative value of the total task completion time (Makespan) can be taken as the final reward, or an incremental reward can be given according to the system running situation after each stealing decision. For example, when a stealing operation significantly reduces the idle time of other processing units or reduces the total time of task completion, a higher positive reward is given; on the contrary, if the stealing is improper and leads to increased overhead or processing unit overload, a penalty is given. Through the design of the reward function in this way, the reinforcement learning agent will tend to adopt the scheduling strategy that can minimize the overall processing time to the greatest extent. This actually enables the agent to learn to minimize both the task completion time and the waiting time of each task, thereby improving the system performance.

[0024] Further, the error monitoring and fault tolerance step includes: Abnormality detection mechanism: based on the time consumption of task execution, resource occupancy rate, and correctness of returned results, the task is judged for abnormality, and when the execution time is significantly longer than the threshold, memory overflow, file reading error, or result verification failure occurs, etc. The task is determined to be abnormal; failure task rescheduling mechanism: When the task is detected to be abnormal or failed, the scheduling system automatically allocates the task fragment to the currently idle or lower load processing unit for re-execution, and writes the information of the abnormal task, the failure reason, the original processing unit ID, the retry number, etc. in real time into the central error log for system analysis and performance optimization; if the same task fragment fails for three consecutive retries, the fault is recorded and the retry is stopped to prevent resource waste.

[0025] Further, the structured data file format output by the result integration step can be dynamically specified by user demand through a configuration interface, supporting multiple data formats including NetCDF, Json, HDF5, and CSV; the integrated output data file is compatible with the interface specifications of mainstream atmospheric and ocean environmental simulation models, and can be automatically registered and pushed to WRF, RAMS, PMTM, CALMET, CFD, and other nuclear emergency dispersion simulation models for subsequent calling.

[0026] The application also discloses a high-efficiency parallel data extraction and preprocessing model for nuclear emergency, comprising: A data slicing module is configured to parse input GRIB format meteorological or ocean data files, logically divide data contents according to time steps, spatial regions, and variable types, and divide original data into multiple task segments. A task initialization module is configured to initialize and distribute task segments to corresponding local task queues according to the number and state of available processing units in the system, and construct task index information for subsequent scheduling management. A parallel task scheduling module is configured to control each processing unit to preferentially execute tasks from the local task queue based on a nuclear emergency parallel task stealing algorithm, and acquire unfinished tasks from the task queues of other processing units through a stealing mechanism combined with an AI strategy when the local task is completed, so as to realize dynamic load balancing. A data integration module is configured to format and integrate intermediate result data obtained after all task segments are preprocessed, and reorganize the data into a structured output file according to the time sequence and spatial structure of the original data; the output file can be directly called by an environmental simulation model. An error monitoring and fault tolerance module is configured to detect the task execution state in real time, reschedule failed tasks to idle processing units for automatic retry when a fault or error occurs, and record error logs for system analysis and optimization.

[0027] Further, the data slicing module adopts a multi-dimensional slicing task division mechanism, divides the GRIB data into several task segments with independent processing boundaries by constructing a three-dimensional data mapping model containing time index, geographic grid coordinates, and meteorological and ocean variable types, and can adjust the data size of each task segment according to the number of currently available processing units, the average execution time of task history, data density distribution, and other dynamic parameters, so as to ensure reasonable task granularity and meet the load balancing requirements of subsequent scheduling; the data slicing module also supports compatible analysis of different versions of GRIB format data, can automatically configure corresponding decoding parameters according to the file format version, and thus improves the adaptability of the model to multiple sources of meteorological and ocean data.

[0028] Furthermore, the parallel task scheduling module adopts a tail-first intelligent optimization task stealing strategy. When the local task queue of any processing unit is empty, the processing unit is allowed to obtain unprocessed task segments from the tail of the task queues of other processing units for execution. The algorithm prioritizes scheduling segments based on the characteristics of nuclear accidents, such as segments from the time before and after a nuclear accident, segments near the nuclear accident site, and segments related to wind vectors, temperature, humidity, geopotential height, and precipitation parameters that have the greatest impact on the spread of the nuclear accident. The scheduling module also dynamically adjusts the granularity of the stolen task segments based on real-time feedback information from task execution, avoiding resource overload or processing duplication due to improper task granularity, thereby improving the overall parallel processing efficiency and load balancing level of the system. The scheduling module can integrate multiple AI-assisted scheduling algorithms to improve scheduling efficiency: different machine learning models are trained based on information such as the execution time and queue length changes of historical tasks of the same type of GRIB data to be processed. Through iterative training, the model can eventually predict the optimal task allocation scheme for the scheduling module.

[0029] Furthermore, the data integration module has result reorganization and interface adaptation functions, which can reorder and merge intermediate results obtained from the preprocessing of multiple task segments according to the time order and spatial structure of the original data to generate an output data file that meets the preset format requirements. The output data file can be directly read and called by various meteorological, oceanographic or environmental simulation models, including WRF, CALMET, PMTM, RAMS, CFD, etc., and the output format can be dynamically set to NetCDF, Json, HDF5 or CSV format according to user needs through the configuration interface of the data integration module.

[0030] The present invention also discloses a computer-readable storage medium storing computer program instructions executable by one or more processors; when the computer program instructions are executed, the processor performs the steps of the above-described efficient parallel data extraction and preprocessing method for nuclear emergency response.

[0031] The beneficial effects of this invention are: 1. This invention significantly improves data preprocessing efficiency through a modular parallel scheduling process. Specifically, this invention achieves efficient preprocessing of GRIB format meteorological and oceanographic data by constructing an intelligent parallel data processing method based on task fragmentation. First, this invention logically divides the original GRIB file into independently processable task fragments through a data fragmentation step. Then, combining multi-core system resources, the task fragments are evenly distributed to the local task queue through a task initialization step. Furthermore, a core-emergency intelligent parallel task-stealing algorithm is adopted to achieve cross-processing unit task collaboration and stealing scheduling based on local priority execution, thereby improving the resource utilization and task throughput of the multi-core system.

[0032] Throughout the process, error monitoring and fault tolerance mechanisms are integrated to automatically reschedule and retry in case of processing failures, ensuring the stability and integrity of data processing. Finally, a result integration step rearranges the processing results into a structured output based on temporal and spatial order, ensuring it can be directly used by nuclear emergency environment simulation models. This entire process is encapsulated within a unified methodology, working collaboratively through process concatenation, load balancing, and error recovery mechanisms to significantly improve the preprocessing efficiency of GRIB data, making it particularly suitable for applications in nuclear emergency scenarios with extremely high requirements for processing speed and stability.

[0033] 2. Compared to existing technologies, the data sharding module proposed in this invention constructs a three-dimensional task mapping model by combining time indexing, spatial grids, and variable types, thereby achieving data sharding with independent processing boundaries. This module not only supports mapping GRIB data into parallel-processable task fragments, but also dynamically adjusts the sharding granularity based on task history processing time, data density, and system processing unit status, ensuring that the generated tasks are small enough for easy distribution without excessive fragmentation leading to frequent scheduling overhead. Furthermore, the parallel task scheduling module analyzes the load of each processing unit in real time during task execution and dynamically reallocates unfinished tasks through a tail-first task-stealing strategy, effectively improving the overall system's resource utilization and task balance. This multi-dimensionally aware task sharding and scheduling mechanism significantly enhances the system's flexibility and scalability when facing high-intensity data processing tasks.

[0034] 3. To address the requirements for robust data processing and accurate model integration in nuclear emergency applications, this invention constructs an error monitoring and fault-tolerance module, possessing capabilities for task execution status monitoring, anomaly type identification, and automatic rescheduling. When a task failure is detected, the system can retry the task, update its status, and record the anomaly without manual intervention, significantly improving the overall stability and emergency response capabilities of the processing system. Simultaneously, the data integration module supports multi-format output, structuring the processed data into formats conforming to the requirements of mainstream meteorological and marine environmental simulation models such as WRF, RAMS, PMTM, CALMET, and CFD. The results can be output to standard formats such as NetCDF, JSON, HDF5, or CSV via a user-configurable interface. This flexible format adaptation mechanism gives the system high portability and engineering application value, enabling rapid embedding into various nuclear emergency simulation platforms and decision support systems. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating the efficient parallel data extraction and preprocessing method for nuclear emergency response in this invention. Figure 2 To further compare key system performance indicators before and after the implementation of the deep learning AI scheduling optimization module ("before optimization" refers to not using deep learning AI predictive scheduling, but only relying on NEPTSA dynamic scheduling; "after optimization" refers to combining deep learning AI to optimize the scheduling strategy). Figure 3 Training curves (loss and accuracy) for the deep learning AI scheduling optimization module. Detailed Implementation

[0036] The present invention will now be described in detail with reference to the accompanying drawings: Example 1 This embodiment provides a highly efficient parallel data extraction and preprocessing method for nuclear emergencies. It enables rapid, efficient, and stable preprocessing of large-scale GRIB format meteorological or marine forecast data during a nuclear accident, providing support for nuclear accident consequence assessment models, nuclear proliferation trend assessment models, and environmental simulation models. This method can run on multi-core computing platforms (such as multi-core CPU servers, GPU servers, or high-performance clusters), and the process is as follows: Step S1: Data Sharding Using a large number of continuous GRIB format meteorological data files as input, the file structure information is first parsed to extract key metadata, including: Time step, spatial grid division information, and types of variables included.

[0037] Based on the above information, a three-dimensional index system (time × space × variable) is constructed, and logically divided as follows: First, each time step is divided, then multiple spatial regions are defined within each time step, and finally, the variable types within each region are defined. The indexed information after division is combined into a "task fragment"; each task fragment has an independent data reading boundary, and decoding does not interfere with each other.

[0038] When dividing tasks, the system dynamically adjusts the granularity of task segments based on factors such as historical processing data preferences, task size, and data density, ensuring that subsequent processing tasks are distributed reasonably in the multi-core system and do not create bottlenecks.

[0039] Step S2: Task Initialization After data sharding is completed, the system enters the task initialization phase. This phase includes the following operations: Detect the number of available processing units and their operating status in the current system; Task segments are initially allocated to the local task queues of each processing unit, and a load prediction strategy is used to avoid single-core overload or idleness. Build task index information for each task segment. Preferably, record the task's start position, size, data type, and status (unprocessed, processing, completed) to provide a basis for dynamic scheduling and task allocation.

[0040] Task index information is recorded in shared memory or a scheduling database to support subsequent scheduling, status updates, and fault tolerance.

[0041] Step S3: Parallel Task Scheduling After entering the task execution phase, each processing unit starts and executes the following scheduling logic: Local priority execution strategy: Each processing unit first extracts task fragments from its local task queue for processing in sequence; Tail-end task stealing mechanism: If a processing unit's local queue is empty, the NEPTSA algorithm triggers a task stealing mechanism to "steal" task segments that have not yet started execution from the tail of other processing units' task queues. Time-step related segments near the time of a given nuclear accident occurrence are prioritized for retrieval, spatial region segments near the point of a given nuclear accident occurrence are prioritized for retrieval, and among variable-related segments, segments containing wind vectors, temperature, humidity, geopotential height, and precipitation-related parameters are prioritized for retrieval. Dynamic granularity adjustment: The scheduling system adjusts the granularity of subsequent task slices based on the actual task processing time feedback mechanism to ensure efficiency and balance. For example, if a certain type of task takes too long to process, the system will further refine it into smaller granular tasks in subsequent partitioning. Through the above dynamic scheduling mechanism, the utilization of processing resources can be maximized, avoiding the phenomenon of some cores being idle or overloaded for a long time, and achieving true load balancing.

[0042] Specific implementation of the AI-assisted (PPO) scheduling module: S31. Model and Features (1) State (Sampling at scheduling time t): Device layer: Type and performance vector of each processing unit (e.g., CPU / GPU identifier, single-core / SM throughput estimate, memory bandwidth metric), queue length, and average time taken for the last 3 completed segments.

[0043] Task layer: Statistical summary (mean / variance) of the segments to be stolen at the tail of each processing unit queue: spatiotemporal proximity (time difference from the time of the incident, spherical distance from the location of the incident); weights of key meteorological factors (comprehensive importance index of wind field, temperature, humidity, geopotential height, and precipitation); segment size estimation (number of grids, number of variables, compression ratio) and estimated time.

[0044] System layer: global idle rate, global load imbalance (such as the Gini / variance index of queuing time for each processing unit).

[0045] The heterogeneous features mentioned above are normalized and then concatenated into a fixed-length vector; the number of devices can be achieved by a fixed upper limit plus filling empty spaces with 0 or by pooling.

[0046] (2) Actions : , where i is the index of the source processing unit being stolen (selected from the set of busy nodes). The continuous granularity parameter (representing the number of items stolen from the tail of the source queue = max(1, ), (This is the length of the source queue). If discretization is required, it can be... Bucketing (e.g., {10%, 20%, 33%, 50%}) can reduce the complexity of the action, but continuous output is recommended to take full advantage of PPO.

[0047] (3) Rewards (Combining step-by-step and endgame elements): Finale: Makespan's final reward is given at the end of the episode.

[0048] Step-by-step: Encourage immediate improvements that are "more balanced and more efficient," such as: in To reduce system idle rate, To reduce the amount of load imbalance, Migration / scheduling overhead (including data movement and scheduling thread wake-up). Penalties include excessively long queues for target units. Weighting Calibrate in simulation using grid search or Bayesian optimization.

[0049] (4) Network structure (Actor-Critic): Share two layers of MLP (e.g., width 256 / 128, ReLU or GELU). Actor Header Output: Source Node Distribution π(i|s): The selection probability of each busy node is obtained through softmax; Granularity Parameter γ outputs the mean and variance (Gaussian strategy) or Beta distribution parameters, which are then pruned to (0,1).

[0050] The Critic head outputs the state value V(s).

[0051] Training uses the PPO-Clip objective, advantage estimation uses GAE(λ), and entropy regularization is added to encourage exploration.

[0052] S32. Simulation Training Without Historical Data (1) Simulation task generator: Random sampling accident time With geographical location Generate multiple batches of GRIB fragments according to the set distribution (FNL / GFS / GRAPES parameter templates can be reused), and assign each fragment the following: Timestamp spatial distance Key factor weights (Wind vector, temperature, humidity, geopotential height, precipitation); computational scale (grid × variables × hierarchy) and device affinity (GPU / CPU advantage coefficient).

[0053] Priority rule quantification (consistent with weight 4): in It is a monotonically decaying or piecewise function; Ensure that "time proximity" is prioritized over "spatial proximity," which in turn is prioritized over "critical factors." Initialization: Segments are placed into the queues of each device according to the strategy of "high priority at the head of the queue and low priority at the tail of the queue," forming an initial allocation based on the on-site rules.

[0054] (2) Simulation actuator: Device performance model: Define the baseline and variance of unit fragment time for CPU / GPU; consider I / O and migration overhead (estimated by fragment size and inter-node bandwidth).

[0055] Time-based advancement: Each step reduces the remaining time in the execution segment; when a device is idle → triggers a strategy decision. →Execute the theft → Update status and rewards.

[0056] Termination condition: All segments complete; makespan and process log are recorded.

[0057] (3) Key Indicators and Path to Achievement: Objectives: Reduce makespan, improve CPU / GPU utilization, and reduce the imbalance variance to migration overhead ratio.

[0058] Process guarantee: (Tail-first + regularized priority + PPO continuous granularity) output → jointly drives the strategy to converge to "fast and stable" in simulation and online feedback.

[0059] Schedule execution pseudocode: Step S4: Data Preprocessing After receiving the task fragment, the processing unit performs the following data preprocessing operations: S41. Use the GRIB decoder (GRIB.API, WGRIB, and other development packages) to decode the data segment at the specified offset position; S42. Extract the target variable and perform necessary unit conversions and physical quantity transformations; S43. Store variable data in a standard intermediate data format, such as a structured array or a temporary NetCDF.

[0060] Each processing unit executes different segment decoding tasks in parallel, greatly improving the overall speed of data preprocessing.

[0061] Step S5: Error Monitoring and Fault Tolerance To improve system stability and reliability, the task scheduling system continuously monitors the execution status of each task, including but not limited to: whether decoding was successful; whether the execution time was abnormal; and whether there were system anomalies such as memory overflow or file read errors.

[0062] When a task fails, the system will: S51. Mark the task status as "failed"; S52. Automatically locate currently idle or low-load processing units; S53. Reschedule the failed task for execution, retrying up to three times; S54. Record error information (task number, failure reason, processing unit ID, number of retries) to the central error log for system performance analysis and improvement.

[0063] Step S6: Results Integration After all task segments have been processed, the system will collect all intermediate result files and perform the following operations: S61. Reordering: Reorganize the results according to the original time, space, and variable order of the data; S62. Format Integration: Based on user or system preset configuration, the integration result is output in a specified data format, supporting NetCDF, Json, HDF5 or CSV formats; S63. Model Adaptation: The structure of the results file and the naming conventions of variables conform to the standard interface requirements of meteorological or oceanic diffusion simulation models such as WRF, RAMS, PMTM, CALMET and CFD, so as to achieve "seamless calling".

[0064] The entire process of result integration and format conversion is completed automatically by the system without manual intervention.

[0065] Example 2 1. Basic Implementation In nuclear emergency response systems, large-scale GRIB format forecasts and reanalysis files from multiple meteorological and oceanographic data sources typically require real-time processing. Using traditional single-threaded serial decoding, decoding and preprocessing a 50GB GRIB data file takes an average of 30 minutes, and there is a risk of decoding interruptions due to high-frequency data loss or processing errors. Other improved multi-threaded parallel decoding methods (such as parallel allocation strategies based on the number of files) basically rely on the system's own allocation strategy. Since the system cannot allocate all resources to the threads used for decoding, and does not employ a fine-grained task-stealing strategy, its decoding and preprocessing time also requires 26 minutes. The efficient parallel data extraction and preprocessing model of this invention, combined with the NEPTSA algorithm and a multi-core task allocation mechanism, can significantly improve data decoding and processing speed.

[0066] Experimental environment: Hardware configuration: 64-core CPU server, 512GB memory, GPU acceleration unit.

[0067] Data file: 50GB GRIB file containing 14 days of global, fully variable GFS, FNL, and grapes meteorological data.

[0068] Test content: Compare the differences in processing efficiency, load balancing, and fault tolerance between the traditional serial method, other improved multi-threaded parallel methods, and the NEPTSA algorithm.

[0069] Implementation steps: Data fragmentation: Using this invention, GRIB files are fragmented according to time series, geographical region, and variable types under various height layers. The size of each task fragment is set to less than 3MB to ensure appropriate task granularity.

[0070] Task initialization and allocation: All task fragments are allocated sequentially to 64 processing units, and each processing unit initializes its local task queue.

[0071] Parallel decoding and task stealing: Processing units use the NEPTSA dynamic scheduling algorithm to balance the load of each task. If a processing unit completes a task, it will automatically steal high-priority tasks from the tail of other queues based on the characteristics of nuclear accident cases and historical task processing and allocation experience, ensuring that no unit is idle throughout the entire decoding process.

[0072] Error monitoring and result integration: Monitor the task execution status in real time, summarize the data after decoding each task segment into shared storage, and integrate the results into a unified format.

[0073] Test results: Processing time has been reduced from 30 minutes for serial processing and 26 minutes for parallel processing to approximately 5 minutes.

[0074] Resource utilization reaches over 95%, achieving load balancing.

[0075] The data loss rate was 0%, and the accuracy was significantly improved.

[0076] Errors that occurred during processing were all reassigned, resulting in a 100% decoding success rate.

[0077] Through this embodiment, the NEPTSA algorithm of the present invention significantly improves data decoding efficiency and load balancing effect, providing an efficient and reliable solution for large-scale data preprocessing in nuclear emergency situations.

[0078] 2. Simplified Implementation In data processing scenarios with small-scale evaluation scales (short-term, small-scale evaluation range), such as when only a 5GB GRIB data file needs to be decoded, this invention can further reduce system resource consumption by simplifying task allocation and adjusting task granularity.

[0079] Simplified steps: Simplified sharding strategy: GRIB data is directly divided into 10 shards based on time series and spatial region data block size, eliminating the need for complex multi-dimensional sharding strategies and reducing task management and sharding overhead.

[0080] Fixed task allocation: Due to the small amount of data, tasks are allocated to 16 processing units, eliminating the need for frequent task stealing. Task stealing is only triggered one to five times when the task allocation is uneven.

[0081] Single-level monitoring: In simplified implementation, fault tolerance can be achieved through single-level monitoring. Faulty tasks can be redistributed once without the need for multiple checks.

[0082] Effect: In small-scale data processing, the simplification strategy reduces sharding and task management overhead by approximately 20%.

[0083] The system processing time is 2 minutes, and the resource utilization rate reaches 90%.

[0084] It is applicable to emergency response scenarios and systems with low-resource environments or limited resources and small to medium-scale evaluation scope.

[0085] 3. Alternative Embodiments In environments lacking GPU acceleration, this invention can achieve the same effect by replacing GPU parallel processing tasks with CPU multi-threaded task processing, thus ensuring processing efficiency.

[0086] Replacement strategy: CPU multi-threading replaces GPU acceleration: If the GPU is unavailable, CPU multi-threading technology is used to allocate each task to a CPU core for processing, and the task granularity is appropriately increased to reduce the burden of thread management.

[0087] Replace with sequential task stealing: When CPU task load is low, the task stealing strategy is changed from stealing high-priority kernel incident features to stealing sequentially first and then considering the priority of kernel incident features to reduce scheduling overhead.

[0088] Simplified error tolerance strategy: The error tolerance mechanism is simplified into a task restart mechanism. If a task fails, the task is restarted and exits after three failures.

[0089] Replacement effect: The alternative strategy of CPU multithreading achieved approximately 80% of the GPU acceleration processing speed.

[0090] The sequential task stealing strategy reduces task scheduling overhead by approximately 10% and improves load balancing performance.

[0091] The task restart mechanism simplifies the error handling process and ensures the reliability of data processing.

[0092] 4. Scalability Implementation Examples in Heterogeneous Environments In heterogeneous environments that support multi-node distributed computing, this invention can support large-scale distributed processing by appropriately adjusting the parallel task allocation strategy, thereby further improving the system's processing capabilities.

[0093] Extended steps: Distributed sharding and task allocation: In a multi-node distributed environment, the sharding strategy of this invention is executed independently on each node. Each node is responsible for a certain amount of data sharding and task initialization. Task fragments can be transmitted to other nodes through the network to achieve parallel decoding.

[0094] Distributed task stealing: Task stealing within each node follows the NEPTA mechanism, while tasks among nodes are dynamically adjusted according to the load to achieve cross-node task load balancing.

[0095] Distributed result integration: The results of each node are aggregated to the master node through message passing or a shared storage system to achieve unified data integration and storage.

[0096] Extended effects: The expansion of the multi-node distributed NEPTSA enables the system to process ultra-large-scale GRIB data files, meeting higher processing requirements such as the assessment of the consequences of medium- and long-term nuclear accidents and multi-mode comparisons in nuclear emergency situations.

[0097] By implementing load balancing and dynamic task adjustment in heterogeneous environments, the system's processing efficiency is improved by 33%.

[0098] The ability to acquire and process distributed data in real time during a nuclear accident emergency greatly enhances the system's adaptability and data processing capabilities.

[0099] Through the above embodiments, the present invention demonstrates its flexibility and efficiency in different application scenarios and hardware environments, and can meet various data processing needs from small-scale single nodes to large-scale distributed heterogeneous environments, providing an effective data preprocessing solution for fields such as nuclear accident emergency response, environmental monitoring and assessment, and pollution diffusion simulation.

[0100] Example 3 This embodiment provides performance testing and comparative analysis of the method of the present invention under large-scale data processing and multiple data sources. It mainly focuses on long-distance simulation scenarios on a global scale (such as global diffusion simulation of atmospheric pollutants in a nuclear emergency, using a global diffusion simulation model such as PMTM) and WRF model calculation scenarios within a small-scale region. The test focuses on decoding performance of different types of GRIB data, efficiency comparison with traditional parallel methods, and changes in scheduling performance before and after introducing a deep learning AI training optimization module.

[0101] Experimental environment: The tests were conducted on a high-performance server with a 64-core CPU, 512GB of memory, and a GPU acceleration unit. The software environment included the GRIB decoding library supporting multi-threaded parallelism and the task scheduling and fault-tolerance module described in this invention.

[0102] Test data: This example selects three representative atmospheric GRIB datasets for performance comparison: FNL's 1° analysis data: This is global final analysis data provided by NCEP, with a horizontal resolution of 1°×1°, encompassing meteorological elements (temperature, humidity, wind field, air pressure, etc.) across multiple isobaric surfaces from the surface to the stratosphere. The test file size is approximately 1.3GB (7-day reanalysis data at 6-hour intervals). GFS 1° forecast data: Output data from the US NCEP Global Forecast System, with a horizontal resolution of 1°×1°, including meteorological elements for multiple forecast lead times (e.g., 0-hour analysis and 3-hour interval forecasts to 24 hours). The test file size is approximately 2GB, covering 7-day forecast data.

[0103] GRAPES 0.125° high-resolution data: Forecast output from the China Meteorological Administration's GRAPES global model, with a horizontal resolution of 0.125° × 0.125°, containing finer grids and more forecast elements. The 10-day forecast data is selected, with a file size of approximately 50GB, representing an extremely large dataset.

[0104] Testing process: The following two processing methods are compared for each type of data: First, the traditional parallel decoding method, which uses hash indexing and multi-threaded scheduling to allocate decoding based on time steps, reading GRIB files record by record and extracting the required meteorological elements, scheduling all decoding tasks in a fixed order; second, the parallel preprocessing method of this invention, which utilizes task fragmentation, multi-core parallel scheduling, and the NEPTSA task-stealing algorithm for processing. The specific implementation steps are as follows: Traditional methods involve sequentially reading GRIB files using a multi-threaded program. For each meteorological variable field, parsing and decoding are performed according to a predefined index. This method cannot employ task-stealing and scheduling strategies when processing large numbers of GRIB files. Each processing unit can only begin decoding the next GRIB file after the previous one in the queue has been decoded. Furthermore, processing units with smaller workloads may remain idle for extended periods after completing their tasks. This approach carries risks of high memory usage at startup, excessively long overall processing time, and potential data loss or errors under high load.

[0105] The method of this invention processes GRIB data by dividing it into tasks and processing them in parallel using the model of this invention. Taking 50GB of GRAPES data as an example, it is first divided into 100 task segments (approximately 500MB per segment) according to time and space. These task segments are then evenly distributed into the local queues of 64 CPU processing units, with each unit initially receiving 1-2 segments. All processing units start decoding preprocessing in parallel, using the NEPTSA algorithm for dynamic scheduling: after any processing unit completes its own task, it immediately steals unstarted task segments from the tail of the queues of other busy processing units according to the priority of nuclear accident characteristics to continue processing, ensuring that no processing unit is idle. During scheduling, the system continuously adjusts the granularity of remaining tasks based on the actual time taken by each task; for example, tasks with longer processing times are split into smaller segments in the next round to maintain load balance. After decoding preprocessing is completed, each processing unit writes intermediate results to shared storage. Finally, the master scheduling node collects and summarizes all results, reassembles the data according to time and space order, and automatically converts the output into a specified format for model use. For GRAPES, FNL, and GFS data, the output is converted to a NetCDF format readable by the WRF model; for FNL and GFS data, the output is converted to the input format (CSV, JSON) required for PMTM global simulation. An error monitoring mechanism is also enabled throughout the parallel processing; if some task segments fail on the first execution, the system will automatically retry the assignment to ensure that all tasks are successfully decoded in the end.

[0106] Performance comparison results:

[0107] Table 1 Table 1 shows a performance comparison of decoding and format conversion for different types of GRIB data (64-core parallel environment). As can be seen from Table 1, the processing performance of the above three types of GRIB data under the traditional parallel method and the parallel method of this invention is compared, including indicators such as total task time, data processing throughput and output conversion efficiency.

[0108] As shown in Table 1, the method of this invention significantly outperforms traditional parallel methods on various types of data. For FNL 1° resolution data, the traditional method takes approximately 53 seconds, while the method of this invention completes the process in only 13 seconds, a speed increase of approximately 4.1 times. For ultra-large-scale GRAPES 0.125° resolution data, the traditional method takes approximately 26 minutes to decode 50GB of data, while the method of this invention reduces the processing time to approximately 5 minutes (an acceleration of approximately 5 times), demonstrating that the advantages of parallelism become more pronounced as data scale and task complexity increase. Furthermore, this invention demonstrates that the overall CPU utilization of the system remains above 95% under parallel processing, achieving efficient load balancing; in contrast, traditional parallel methods have lower multi-core resource utilization. In terms of data throughput, the traditional method processes GRAPES data at approximately 38MB / s, while the method of this invention reaches approximately 104MB / s, indicating that task fragmentation and parallelism effectively improve decoding throughput.

[0109] Performance Comparison Before and After Deep Learning AI Optimization of Scheduling Module: This invention also provides a scheduling module optimized by integrating deep learning AI algorithms to further improve the intelligence of task scheduling. By further introducing a deep learning model, and training on case data with dynamic queue adjustment based on historical similar GRIB data task execution time, a better initial task allocation scheme can be predicted. This embodiment compares the system performance changes before and after enabling deep learning AI optimization, mainly examining three indicators: scheduling efficiency, load balancing degree, and task failure rate. The results are as follows: Figure 2 As shown.

[0110] like Figure 2 As shown, enabling deep learning AI-assisted scheduling improved the system's scheduling efficiency and load balancing metrics. For example, under the same test data, the overall CPU utilization increased from approximately 95% before optimization to nearly 98%, indicating that the task allocation predicted by deep learning AI ensured that each processing unit almost always had tasks available for execution, further reducing scheduling waiting overhead. Regarding load balancing metrics (defined as the balance of working time across processing units), the workload of each core became more consistent after optimization: the difference in task execution time was significantly reduced, and the system as a whole utilized multi-core resources more evenly (from approximately 90% load balancing before optimization to approximately 98%). Furthermore, deep learning AI-optimized scheduling also reduced the task failure rate. Because the machine learning model can identify tasks that may take too long or have potential errors in advance and allocate or adjust them more reasonably, the number of tasks requiring retries was significantly reduced after optimization. In the test, without deep learning AI, approximately 5% of task segments experienced timeouts or errors during initial execution and required rescheduling, while with deep learning AI enabled, this proportion dropped to approximately 1%. This means that the failure retry rate was reduced by 80%, further improving the stability of the overall processing flow.

[0111] Figure 3 This demonstrates the convergence process of the deep learning AI scheduling optimization module over 80 training epochs. The left axis represents loss, and the right axis represents accuracy. As training progresses, the training / validation loss generally decreases monotonically and stabilizes further after the learning rate decays in the 40th epoch; the training / validation accuracy continuously increases, ultimately reaching nearly 95% validation accuracy. The curve marks the epoch corresponding to the "optimal validation loss," indicating that the model achieves optimal generalization at this point. Slight fluctuations in the later validation loss reflect that, without significant overfitting, the model has fully learned the mapping relationship between task execution time and queue dynamics, thus enabling more accurate prediction of initial task allocation and granularity adjustment strategies. After training, the scheduler integrating this model significantly improves multi-core utilization, balances computational load, and reduces failure retry rates under real-world load conditions, providing stable intelligent scheduling support for high-intensity computing environments requiring nuclear emergency response.

[0112] In summary, this embodiment quantifies the advantages of the method of the present invention over traditional methods under different data scales and task complexities through performance testing, and verifies the performance improvement effect of the deep learning AI training optimization module on the system. The present invention not only significantly improves the speed and efficiency of GRIB data preprocessing, but also demonstrates significant improvements in intelligent scheduling, load balancing, and fault tolerance reliability, providing strong data processing support for global environmental simulation in nuclear accident emergency response.

[0113] Example 4 This embodiment further optimizes the adaptive control mechanism for task granularity and the system's automatic compatibility with multiple versions of GRIB data.

[0114] In large-scale data processing scenarios, the processing time of a single task has a significant impact on overall performance. Based on historical task processing time statistics, the system dynamically calculates the mean and variance of task execution times and adaptively controls task granularity according to the following rules: If the processing time variance is large, increase the task partitioning precision (smaller segments); If the load balancing is poor, the task granularity will be halved after the theft is performed; If the system resource utilization is low, the granularity will be automatically increased.

[0115] Furthermore, considering that different organizations may use different versions of GRIB data, the system incorporates a version identification module during data parsing. This module automatically calls the corresponding decoder based on the index information and header identifiers in the GRIB file, and adapts to different compression methods, thereby achieving unified preprocessing support for different versions of GRIB files.

[0116] This extended functionality ensures that the system can operate normally in different regions, with different application units, and with different sources of meteorological data, thereby improving the system's adaptability and reliability in actual nuclear emergency deployments.

[0117] Example 5 This embodiment simplifies the scheduling mechanism for small to medium-sized deployment scenarios while retaining the key features of the present invention.

[0118] When the system starts up, it detects the number of available cores and memory resources, and adjusts the task partitioning granularity according to resource limits to prevent memory overload.

[0119] Processing units are not pre-allocated during the task initialization phase, but are dynamically allocated by the task scheduler during the execution phase, and are scheduled in a round-robin manner based on the current load of each core.

[0120] Error handling mechanisms are still retained, with a backup core taking over execution after a task fails, ensuring that the entire task does not fail due to a failure of a single core.

[0121] The integration module can be configured to output only CSV format, reducing the overhead of format conversion.

[0122] This embodiment is applicable to resource-constrained scenarios, such as mobile nuclear emergency units and regional emergency centers. The system is easy to deploy, has low maintenance costs, and is highly practical.

[0123] Example 6 A computer-readable storage medium stores computer program instructions executable by a processor. When the computer program instructions are executed by the processor, the processor performs steps S1 to S6 of the "High-Efficiency Parallel Data Extraction and Preprocessing Method for Nuclear Emergency Response" described in the above embodiment. The computer program instructions at least implement the following modular functions: calling a data sharding module to complete task fragment division; calling a task initialization module to complete task indexing and queue initialization; calling a parallel task scheduling module to perform NEPTSA task stealing and dynamic granularity adjustment; calling a data processing module (corresponding to the data preprocessing function in the claim) to perform GRIB decoding and variable extraction; calling an error monitoring and fault tolerance module to implement anomaly detection and failure retries; and calling a data integration module to complete result reordering and formatted output.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Those skilled in the art should understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for efficient parallel data extraction and pre-processing for nuclear emergencies, characterized in that, The method comprises the following steps: Step S1, parsing the input GRIB format meteorological and ocean data file, extracting its time step, variable type and spatial grid information, and logically dividing the data according to time, space and variable dimensions to form a plurality of independently processable task segments; Step S2, according to the number and state of the multi-core processing units available in the system, the task segments are initialized and distributed to the corresponding local task queue, and a task index information table is constructed for each task, which at least includes the offset address, time interval, variable type, spatial range, target processing unit identifier and task state of the task segment; Step S3, based on the core emergency parallel task stealing algorithm, each processing unit preferentially executes the task segment from the head of the local task queue, and when the local queue is empty, the scheduler acquires the unfinished task segment from the tail of the queue of other processing units through the task stealing mechanism optimized by deep reinforcement learning PPO; during the scheduling process, the granularity of the subsequent task segment is dynamically adjusted in combination with the task execution feedback, so as to realize the load balancing among the multiple processing units; Step S4, each processing unit performs preprocessing operations such as decoding, format conversion and target variable extraction on the assigned task segment to generate the corresponding intermediate data result; Step S5, the task state and resource usage are monitored in real time during the task execution process, if an exception or task failure is detected, the task state is marked as failed and automatically rescheduled to other idle processing units for retry execution, the failure retry is performed at most three times, and the related exception information is recorded to the central error log; Step S6, after all the task segments are preprocessed, all the intermediate result data are recombined and sorted according to the time sequence and spatial position of the original data to generate a structured output file directly for calling by the environmental simulation model.

2. The method for efficient parallel data extraction and pre-processing for nuclear emergencies as claimed in claim 1 wherein, In step S1, the data is divided based on a three-dimensional task mapping mechanism; by constructing a three-dimensional data model containing time index, geographic grid coordinates and variable type dimension, the GRIB meteorological and ocean data are mapped into a plurality of task segments with independent processing boundaries, and the data amount contained in each task segment is dynamically adjusted in combination with historical processing time consumption distribution, data density and spatial continuity and other factors to ensure the balance and scheduling friendliness of the task granularity.

3. The method for efficient parallel data extraction and pre-processing for nuclear emergency of claim 1, wherein, In the task initialization step, the task state corresponding to each task segment includes "unprocessed", "processing", "completed" and "failed", and the system tracks and manages the task execution process through the task state recorded in the task index information table; when a task segment execution failure is detected, automatic redistribution and recovery scheduling can be performed according to the task state information.

4. The method for efficient parallel data extraction and pre-processing for nuclear emergencies as claimed in claim 1 wherein, The NEPTSA parallel task stealing algorithm adopts a tail-first stealing strategy: when a processing unit has executed all the task segments in its local task queue, it acquires the task segments that have not started execution from the tail of the task queue of other processing units according to a preset priority; The algorithm schedules the fragment priority selection rules according to the characteristics of the nuclear accident: ① the time close to the nuclear accident, ② the space near the nuclear accident point, ③ the wind direction, temperature, humidity, potential height and precipitation related parameters that have the greatest impact on the diffusion of the nuclear accident; and introduces a deep reinforcement learning scheduling strategy based on the proximal policy optimization PPO, so that the stealing decision outputs the "stealing source processing unit" and the "stealing granularity" at the same time, and dynamically adjusts the granularity of the stolen task according to the actual execution time feedback of each task fragment.

5. The method for efficient parallel data extraction and pre-processing for nuclear emergencies as claimed in claim 1 wherein, The error monitoring and fault tolerance step includes: Abnormality detection mechanism: based on the time consumption of task execution, resource occupancy rate and correctness of returned results, the task is judged for abnormality, and when the execution time is obviously beyond the threshold, memory overflow, file reading error or result verification failure occurs, etc., the task is determined to be abnormal; Failed task rescheduling mechanism: when the task is detected to be abnormal or failed, the scheduling system automatically allocates the task fragment to the current idle or low-load processing unit for re-execution, and writes the identification of the abnormal task, the failure reason, the original processing unit ID, the retry number and other information into the central error log in real time for system analysis and performance optimization; if the same task fragment fails for three consecutive times, record the fault and stop retrying to prevent resource waste.

6. The method for efficient parallel data extraction and pre-processing for nuclear emergencies as claimed in claim 1 wherein, The structured data file format output by the result integration step can be dynamically specified by user demand through a configuration interface, supporting multiple data formats including NetCDF, Json, HDF5 and CSV; the integrated output data file is compatible with the interface specification of mainstream atmospheric and ocean environmental simulation models, and can be automatically registered and pushed to WRF, RAMS, PMTM, CALMET, CFD and other nuclear emergency diffusion simulation models for subsequent calling.

7. A high efficient parallel data extraction and pre-processing model for nuclear emergency, characterized in that, It includes: A data slicing module for parsing input GRIB format meteorological data files and logically dividing the data content according to time step, spatial region and variable type information, dividing the original data into multiple independently processable task fragments; A task initialization module for initializing and distributing the task fragments to the corresponding local task queues according to the number and running state of the available processing units in the system, and constructing task index information for each task fragment for subsequent scheduling management; A parallel task scheduling module that controls the parallel execution of the task fragments by each processing unit based on the nuclear emergency parallel task stealing algorithm, each processing unit preferentially executes tasks from the local task queue, and when the local task is completed, it acquires unfinished tasks from the task queues of other processing units through the stealing mechanism combined with the AI algorithm, achieving dynamic load balancing and collaborative work among processing units; A data integration module for formatting and integrating the intermediate result data obtained after all task fragments are completed, and reorganizing them into structured output files in the order of time sequence and spatial structure of the original data; The output file can be directly called by the environmental simulation model; Error monitoring and fault tolerance module, for real-time monitoring of task execution state and system running parameters, when the failure or task error will be failure task fragment rescheduled to idle processing unit for automatic retry, and record error log information for system operation analysis and optimization improvement.

8. The model for efficient parallel data extraction and pre-processing for nuclear emergencies according to claim 7, characterized in that The data fragmentation module adopts a multi-dimensional slicing task division mechanism, divides the GRIB data into several task fragments with independent processing boundaries by constructing a three-dimensional data mapping model containing time index, geographic grid coordinates and meteorological and ocean variable types, and can adjust the data size of each task fragment according to the number of currently available processing units, task historical average execution time, data density distribution and other dynamic parameters, to ensure reasonable task granularity and meet the load balancing requirements of subsequent scheduling; The data fragmentation module also supports compatible analysis of different versions of GRIB format data, and can automatically configure the corresponding decoding parameters according to the file format version, thereby improving the adaptability of the model to multi-source meteorological and ocean data.

9. The model for efficient parallel data extraction and pre-processing for nuclear emergencies as claimed in claim 7 wherein The parallel task scheduling module adopts a tail-first task stealing strategy, which allows any processing unit to obtain a task fragment that has not been processed from the tail of the task queue of another processing unit when its local task queue is empty; The algorithm preferentially schedules fragments related to wind direction, temperature, humidity, potential height and precipitation parameters near the nuclear accident point and at similar times before and after the nuclear accident, which have the greatest impact on the spread of the nuclear accident; And the scheduling module dynamically adjusts the granularity size of the stolen task fragments according to the real-time feedback information of task execution, avoids resource overload or processing duplication caused by improper task granularity, and improves the overall parallel processing efficiency and load balancing level of the system; The data integration module has result recombination and interface adaptation functions, can reorder and integrate the intermediate results obtained by preprocessing multiple task fragments according to the time sequence and spatial structure of the original data, and generate output data files that meet the preset format requirements; The output data file can be directly read and called by various meteorological, ocean or environmental simulation models including WRF, CALMET, PMTM, RAMS, CFD, etc., and the output format can be dynamically set to NetCDF, Json, HDF5 or CSV format through the configuration interface of the data integration module according to user requirements.

10. A computer readable storage medium characterized by, The storage medium stores computer program instructions executable by one or more processors; When the computer program instructions are executed, the processor executes the steps of the efficient parallel data extraction and preprocessing method for nuclear emergency in any one of claims 1-6.

Images