Distributed hydrological model parallel computing method, device, equipment and medium

Abstract

The invention discloses a distributed hydrological model parallel computing method, which comprises the following steps of: dividing a drainage basin into a plurality of grid units, taking the grid units as computing units of a model, and determining a hierarchical structure according to a topological relation of the computing units; wherein the hierarchical structure comprises the number of hierarchies and calculation units needing to be executed by each hierarchy; constructing a to-be-executed calculation task queue, and pushing the calculation unit and the unit information of the calculation unit into the to-be-executed calculation task queue based on the hierarchical structure; wherein one computing unit corresponds to one computing task; calculating tasks are obtained from the to-be-executed calculating task queue, and the calculating tasks are distributed to different calculating nodes, so that the calculating nodes execute the calculating tasks in parallel; according to the method, dynamic load adjustment can be realized when multiple tasks are parallel, and the utilization rate of computing resources and the computing efficiency of a distributed hydrological model are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of water resource scheduling, and in particular to a distributed hydrological model parallel computing method, device, equipment and medium. Background Art

[0002] A distributed hydrological model is a hydrological simulation model that breaks down the hydrological processes (such as rainfall, evaporation, runoff, etc.) in a basin into spatial distribution. Different from traditional runoff models or lumped hydrological models, it divides the basin into several small units or grids as calculation units, and analyzes the hydrological processes between these units, the interactions between various hydrological elements, and their temporal and spatial variations, so as to simulate the hydrological cycle in the basin in detail.

[0003] When calculating distributed hydrological models, existing technologies mostly rely on physical machine clusters or virtual machine clusters for resource allocation. It is usually necessary to reserve a large amount of hardware resources in advance for high-load scenarios and perform manual configuration to cope with peak demand. As a result, these computing resources are often wasted due to idleness when the overall load is low. This task scheduling based on static configuration is difficult to dynamically adjust according to the real-time load of the computing nodes, and it is impossible to dynamically allocate tasks in real time, resulting in load imbalance and performance degradation of the computing nodes, affecting the overall computing efficiency of the model. Summary of the invention

[0004] The embodiment of the present invention provides a distributed hydrological model parallel computing method, which can realize dynamic load adjustment when multiple tasks are in parallel, improve the utilization rate of computing resources and the computing efficiency of the distributed hydrological model.

[0005] In a first aspect, an embodiment of the present invention provides a distributed hydrological model parallel computing method, including:

[0006] Divide the watershed into a number of grid units, use the grid units as calculation units of the model, and determine a hierarchical structure according to the topological relationship of the calculation units; wherein the hierarchical structure includes the number of levels and the calculation units to be executed at each level;

[0007] Constructing a queue of computing tasks to be executed, and pushing the computing units and the unit information of the computing units into the queue of computing tasks to be executed based on the hierarchical structure; wherein one computing unit corresponds to one computing task;

[0008] Obtain computing tasks from the to-be-executed computing task queue, and assign the computing tasks to different computing nodes so that the computing nodes execute the computing tasks in parallel.

[0009] Further, determining the hierarchical structure according to the topological relationship of the computing units includes:

[0010] Obtaining a topological relationship table of the computing unit, and taking the computing unit whose upstream computing unit in the topological relationship table is empty as the first layer; wherein the topological relationship table records the upstream computing unit and the downstream computing unit of each computing unit;

[0011] Iterate the computing units of the latest layer. In each round of iteration, the downstream computing units of all computing units in the current layer are used as the next layer of the current layer until the iteration termination condition is met, and the layer that meets the iteration termination condition is used as the last layer; wherein, the iteration termination condition is that the downstream computing units of all computing units in the current layer are empty.

[0012] Furthermore, the unit information includes the level to which the current computing unit belongs, the current computing unit ID, and the upstream computing unit ID of the current computing unit.

[0013] Further, obtaining computing tasks from the to-be-executed computing task queue and assigning the computing tasks to different computing nodes so that the computing nodes execute the computing tasks in parallel includes:

[0014] Obtaining computing tasks of each level in sequence from the to-be-executed computing task queue, and assigning the computing tasks to different computing nodes;

[0015] Obtain the image according to the image address specified in the computing task, and schedule all the images to the corresponding computing nodes;

[0016] Creating a corresponding container instance for each of the computing tasks, and inputting the data files required by the computing tasks into the container instance;

[0017] Control multiple container instances to execute the computing task in parallel, and output the task execution result after the task is successfully executed.

[0018] Furthermore, the obtaining of computing tasks of each level in sequence from the queue of computing tasks to be executed includes:

[0019] According to the hierarchical structure, obtaining computing tasks from the to-be-executed computing task queue in units of hierarchies;

[0020] Constructing a queue of executed computing tasks, and pushing the successfully executed computing tasks into the queue of executed computing tasks;

[0021] When it is detected that all computing tasks of the same level are in the executed computing task queue, the computing tasks of the next level continue to be obtained from the to-be-executed computing task queue.

[0022] Furthermore, the method further comprises:

[0023] When it is detected that the computing task fails to execute or the container instance corresponding to the computing task is in an abnormal state, the computing task is reallocated to other computing nodes until the computing task is successfully executed.

[0024] Furthermore, the method further comprises:

[0025] The node load conditions of all the computing nodes are monitored in real time, and the allocation of the computing tasks is dynamically adjusted according to the node load conditions.

[0026] In a second aspect, an embodiment of the present invention provides a distributed hydrological model parallel computing device, including:

[0027] A hierarchical structure determination module, used to divide the watershed into a number of grid units, use the grid units as calculation units of the model, and determine the hierarchical structure according to the topological relationship of the calculation units; wherein the hierarchical structure includes the number of levels and the calculation units to be executed at each level;

[0028] A task queue construction module is used to construct a computing task queue to be executed, and push the computing unit and the unit information of the computing unit into the computing task queue to be executed based on the hierarchical structure; wherein one computing unit corresponds to one computing task;

[0029] The computing task execution module is used to obtain computing tasks from the computing task queue to be executed, and distribute the computing tasks to different computing nodes so that the computing nodes execute the computing tasks in parallel.

[0030] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0031] Memory for storing computer programs;

[0032] A processor, configured to execute the computer program;

[0033] Wherein, when the processor executes the computer program, the distributed hydrological model parallel computing method described in any one of the first aspects above is implemented.

[0034] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed, the distributed hydrological model parallel computing method described in any one of the first aspects above is implemented.

[0035] Compared with the prior art, a distributed hydrological model parallel computing method provided by an embodiment of the present invention has the following beneficial effects: by dividing the watershed into a number of grid units, taking the grid units as the computing units of the model, and determining the hierarchical structure according to the topological relationship of the computing units; wherein the hierarchical structure includes the number of levels and the computing units to be executed at each level; constructing a queue of computing tasks to be executed, and pushing the computing units and the unit information of the computing units into the queue of computing tasks to be executed based on the hierarchical structure; wherein one computing unit corresponds to one computing task; obtaining computing tasks from the queue of computing tasks to be executed, and assigning the computing tasks to different computing nodes, so that the computing nodes execute the computing tasks in parallel; the present invention can realize dynamic load adjustment when multiple tasks are in parallel, and effectively improve the utilization rate of computing resources and the computing efficiency of distributed hydrological models. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical features of the embodiments of the present invention, the drawings required for use in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 It is a flow chart of an embodiment of a distributed hydrological model parallel computing method provided by the present invention;

[0038] Figure 2 It is a hierarchical structure diagram of an embodiment of a distributed hydrological model parallel computing method provided by the present invention;

[0039] Figure 3 It is a schematic diagram of a computing task queue of an embodiment of a distributed hydrological model parallel computing method provided by the present invention;

[0040] Figure 4 It is a schematic diagram of a scheduling framework of an embodiment of a distributed hydrological model parallel computing device provided by the present invention;

[0041] Figure 5 It is a structural schematic diagram of an embodiment of a distributed hydrological model parallel computing device provided by the present invention;

[0042] Figure 6 It is a structural schematic diagram of an embodiment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0046] In a first aspect, an embodiment of the present invention provides a distributed hydrological model parallel computing method, see Figure 1 , which is a flow chart of an embodiment of a distributed hydrological model parallel computing method provided by the present invention.

[0047] like Figure 1 As shown, the method comprises the following steps:

[0048] S1: Divide the watershed into a number of grid units, use the grid units as calculation units of the model, and determine a hierarchical structure according to the topological relationship of the calculation units; wherein the hierarchical structure includes the number of levels and the calculation units to be executed at each level;

[0049] S2: constructing a queue of computing tasks to be executed, and pushing the computing units and the unit information of the computing units into the queue of computing tasks to be executed based on the hierarchical structure; wherein one computing unit corresponds to one computing task;

[0050] S3: Obtain computing tasks from the to-be-executed computing task queue, and assign the computing tasks to different computing nodes, so that the computing nodes execute the computing tasks in parallel.

[0051] In a specific implementation, the watershed is divided into a number of grid units, and the grid units are used as the calculation units of the model. The results of each calculation unit are calculated in turn. This process starts at the entrance of the watershed, and the result of the first calculation unit is calculated. The upstream and downstream topological relationship between the calculation units is utilized, and the output result of the upstream calculation unit is used as the basis for the input of the downstream calculation unit. The calculation task of each calculation unit is executed one by one. Each calculation unit will receive data from its upstream unit, and perform calculations based on these data and its own algorithm logic, and then output its results. These results will serve as the input of subsequent downstream calculation units, and this cycle will be repeated until all calculation units in the entire watershed have completed their calculation tasks. Finally, in the calculation unit where the watershed outlet is located, the model summarizes and processes the results of all upstream calculation units to obtain the overall calculation result of the model.

[0052] During the implementation of the present invention, the hierarchical structure of the computing units is determined according to the topological relationship between the computing units, including the number of levels and the computing units that need to be executed at each level, and a queue of computing tasks to be executed is constructed. The computing units and the unit information of the computing units are pushed into the queue of computing tasks to be executed according to the hierarchical relationship, wherein one computing unit corresponds to one computing task, and the computing tasks are obtained from the queue of computing tasks to be executed, and the computing tasks are assigned to different computing nodes so that the computing nodes execute the computing tasks in parallel. When executing computing tasks, container instances are created and containers are orchestrated through cloud native technologies such as Kubernetes and Docker, and distributed tasks are scheduled in parallel. The computing tasks are scheduled to each computing node according to the load of the computing resources for parallel execution, so as to make full use of the existing computing resources and improve the overall computing efficiency of the model.

[0053] In an optional implementation, determining the hierarchical structure according to the topological relationship of the computing units includes:

[0054] Obtaining a topological relationship table of the computing unit, and taking the computing unit whose upstream computing unit in the topological relationship table is empty as the first layer; wherein the topological relationship table records the upstream computing unit and the downstream computing unit of each computing unit;

[0055] Iterate the computing units of the latest layer. In each round of iteration, the downstream computing units of all computing units in the current layer are used as the next layer of the current layer until the iteration termination condition is met, and the layer that meets the iteration termination condition is used as the last layer; wherein, the iteration termination condition is that the downstream computing units of all computing units in the current layer are empty.

[0056] Specifically, a topological relationship table of the computing unit is obtained, and the topological relationship table records the upstream computing unit and the downstream computing unit of each computing unit. Exemplarily, the topological relationship table is shown in Table 1 below:

[0057] Table 1 Topological relationship table

[0058] Serial number Compute unit ID Upstream computing unit ID Downstream computing unit ID 1 a1 / b1 2 a2 / b2 3 b1 a1 c1 4 b2 a2 c1 5 c1 b1,b2 d1 … … … …

[0059] Traverse the topological relationship table, identify all computing units whose upstream computing units are empty, and use these computing units as the first layer. Taking the above table as an example, the upstream computing units of a1 and a2 are empty, and the computing units a1 and a2 are used as the first layer. Then enter the iteration process. In each round of iteration, process all computing units in the current layer. Assume that the current layer is the nth layer, that is, traverse all computing units in the nth layer, find its downstream computing units, and use these downstream computing units to form the next layer of the current layer, that is, the n+1th layer. For example, in the above table, traverse all computing units in the first layer, find its downstream computing units b1 and b2, and use computing units b1 and b2 as the second layer. After each iteration, check whether the n+1th layer is empty. If not, continue to the next round of iteration. If it is empty, it indicates that the traversal has reached the situation where there is no downstream computing unit. At this time, the iteration termination condition is met, the iteration is terminated, and the layer currently processed is the last layer. When the iteration is terminated, an ordered hierarchical structure is obtained. From the first layer to the last layer, each layer contains corresponding computing units.

[0060] For example, see Figure 2 As shown, it is a schematic diagram of the hierarchical structure obtained according to the topological relationship table, wherein each layer contains corresponding computing units, the computing units of the first layer have no upstream computing units, the computing units of the nth layer are all downstream computing units of the computing units of the n-1th layer, and the nth layer is the final layer.

[0061] In an optional implementation, the unit information includes the level to which the current computing unit belongs, the current computing unit ID, and the upstream computing unit ID of the current computing unit.

[0062] Specifically, the unit information of each computing unit includes the layer (Layer) to which the current computing unit belongs, the current computing unit ID (UnitID), and the upstream computing unit ID (DependUnitID) of the current computing unit.

[0063] Further, a queue of computing tasks to be executed is constructed, and the computing units and the unit information of the computing units are pushed into the queue of computing tasks to be executed in the order of low to high levels, with one computing unit corresponding to one computing task. For example, see Figure 3, is a schematic diagram of the computing task queue in the present invention, including a computing task queue to be executed on the left and a computing task queue that has been executed on the right. The computing units with a determined hierarchical structure and the unit information of the computing units are pushed into the computing task queue to be executed in order from low to high levels. The computing units with low levels are located at the head of the queue, and the computing units with high levels are located at the tail of the queue.

[0064] In an optional implementation, obtaining computing tasks from the to-be-executed computing task queue and assigning the computing tasks to different computing nodes so that the computing nodes execute the computing tasks in parallel includes:

[0065] Obtaining computing tasks of each level in sequence from the to-be-executed computing task queue, and assigning the computing tasks to different computing nodes;

[0066] Obtain the image according to the image address specified in the computing task, and schedule all the images to the corresponding computing nodes;

[0067] Creating a corresponding container instance for each of the computing tasks, and inputting the data files required by the computing tasks into the container instance;

[0068] Control multiple container instances to execute the computing task in parallel, and output the task execution result after the task is successfully executed.

[0069] For details, see Figure 4 , is a schematic diagram of the scheduling framework of the distributed hydrological model parallel computing method of the present invention, including a central scheduling node and a plurality of computing nodes. The central computing node is used to sequentially obtain computing tasks of each level from the queue of computing tasks to be executed, and reasonably allocate the computing tasks to different computing nodes. These computing nodes each have different processing capabilities and resource conditions for executing the allocated computing tasks. The number of computing tasks allocated to different computing nodes may be different, mainly depending on the specific requirements of each computing task and the load of the computing node itself. For example, Figure 4 The central scheduling node in takes out a computing task of a certain layer from the queue of computing tasks to be executed, including K computing tasks in total. Computing node 1 is assigned to 3 computing tasks, computing node 2 is assigned to 2 computing tasks, and computing node N is assigned to 1 computing task.

[0070] It should be noted that the present invention packages the core algorithm package of the distributed hydrological model and its dependencies into a Docker container image, executes the instructions of the Dockerfile file through the Docker build command and generates an image file, uploads the image file to the image warehouse, and the image warehouse uniformly manages and distributes the model calculation package. After the tasks are assigned, the image warehouse is used to obtain the corresponding image file according to the image address specified in each computing task, and the Kubernetes tool is used to schedule these images to N computing nodes. During the scheduling process, Kubernetes will create a corresponding Pod and container instance for each image, and each computing task is assigned a container instance. For example, Figure 4 The computing node 1 in the figure is assigned to three computing tasks. The image is obtained according to the image address specified in each computing task and scheduled to the computing node 1. Three pods and container instances are assigned to the three computing tasks respectively. The container instances are not drawn in the figure.

[0071] The Kubernetes Parallel Job controller controls the parallel execution of multiple container instances. The completions field in the YAML file specifies the total number of parallel tasks K, and the parallelism field specifies the number of computing nodes N. After the task is completed, the central scheduling node is notified in the form of an event, so that the corresponding Pod is automatically terminated on the computing node and the computing resources are released.

[0072] It is understandable that Kubernetes is an open source container orchestration and management platform for automatically deploying, scaling, and managing containerized applications. Pod is the smallest deployment unit in Kubernetes. A Pod can contain one or more containers, as well as some storage and network resources shared by these containers. Kubernetes manages the life cycle of Pods to achieve automated deployment, scaling, and management of containerized applications. It also provides rich functions and services to support the efficient operation of containerized applications.

[0073] During the execution of computing tasks, the central scheduling node is also used to coordinate the data transmission and sharing of each computing node and container instance. The present invention uses a file system as shared storage. For example, a distributed file system such as Hadoop HDFS and MooseFS can be selected. No specific limitation is made here. The input data required by the model, such as rainfall, terrain data, etc., are stored in the shared storage file system in the form of files. These data files can be text files, binary files or other formats, depending on the input requirements of the model. The output data generated by the model, such as control section flow, etc., are also stored in the shared storage in the form of files to facilitate subsequent data processing and analysis. The intermediate process data generated during the calculation of the computing task is also stored in the file system and is uniformly managed by the central scheduling node.

[0074] After the container instance is created, all data files required for the model computing task are written to the specified directory inside the container instance through the Kubernetes API or the kubectl exec command. It can be understood that the Kubernetes API and kubectl exec are two different ways of interacting with Pods in Kubernetes. The model algorithm package in the container instance reads the data files in the specified directory for calculation at runtime. After the calculation task is completed, the model algorithm package process writes the calculation results in the form of data files to the specified directory inside the container instance. After the central scheduling node receives the notification of the completion of the calculation task, it reads the output task execution results from the specified directory through the Kubernetes API or kubectl exec, and puts them into the file system for storage management.

[0075] Through the automatic scheduling and containerization technology of Kubernetes, the present invention can efficiently manage a large number of model computing tasks, ensuring that they can fully utilize the computing resources in the cluster while maintaining a high degree of scalability and fault tolerance.

[0076] In an optional implementation, sequentially acquiring computing tasks of each level from the to-be-executed computing task queue includes:

[0077] According to the hierarchical structure, obtaining computing tasks from the to-be-executed computing task queue in units of hierarchies;

[0078] Constructing a queue of executed computing tasks, and pushing the successfully executed computing tasks into the queue of executed computing tasks;

[0079] When it is detected that all computing tasks of the same level are in the executed computing task queue, the computing tasks of the next level continue to be obtained from the to-be-executed computing task queue.

[0080] Specifically, the computing tasks in the queue of computing tasks to be executed are executed in layers, for example, see Figure 3 As shown, a queue of executed computing tasks is constructed. Whenever a computing task is completed, the completed computing task is placed in the queue of executed computing tasks and traversed. When the queue of executed computing tasks already contains the computing tasks of the entire layer, the computing task information of the next layer is continued to be taken out from the queue of computing tasks to be executed until the queue of computing tasks to be executed is empty.

[0081] In an optional embodiment, the method further includes:

[0082] When it is detected that the computing task fails to execute or the container instance corresponding to the computing task is in an abnormal state, the computing task is reallocated to other computing nodes until the computing task is successfully executed.

[0083] Specifically, Kubernetes continuously tracks the status and resource usage of each container through its built-in monitoring components. When it detects that a container cannot continue to execute tasks or the computing task fails due to various reasons, such as application crash, insufficient resources, hardware failure, etc., it starts a new container instance and migrates the original computing task to a healthy computing node to ensure that the task can be successfully executed. If the task fails again, Kubernetes will repeat the above process until the task is finally successfully completed. This continuous parallel execution and fault recovery mechanism enables the Kubernetes cluster to cope with various complex scenarios, ensuring that all computing tasks can be completed smoothly while maximizing the use of cluster resources.

[0084] In an optional embodiment, the method further includes:

[0085] The node load conditions of all the computing nodes are monitored in real time, and the allocation of the computing tasks is dynamically adjusted according to the node load conditions.

[0086] Specifically, the Kubernetes tool monitors the CPU, memory, and disk I / O of each computing node in real time. If the load of a computing node is too high, the Kubernetes scheduler will assign new computing tasks to other computing nodes with lower loads. Dynamically adjusting task allocation can ensure that the resources of the entire cluster are used evenly, avoid overloading of some nodes while other nodes are idle, reduce waste of resources, and thus maintain the stability and reliability of the system.

[0087] In summary, the present invention makes full use of the characteristics of cloud native technology, uses technologies such as Kubernetes and Docker to implement parallel scheduling and fault-tolerant mechanisms, improves the stability of the system, and executes large-scale and diversified computing tasks through a distributed architecture, which is in line with the model mechanism of the distributed hydrological model. Based on the resource scheduling characteristics of Kubernetes, it can realize dynamic load adjustment when multiple tasks are running in parallel, and can automatically recover when a task fails, thereby ensuring the continuity and accuracy of the calculation, and improving the computing efficiency of the distributed hydrological model and the utilization of computing resources.

[0088] In a second aspect, an embodiment of the present invention provides a distributed hydrological model parallel computing device, see Figure 5 , which is a structural diagram of an embodiment of a distributed hydrological model parallel computing device provided by the present invention.

[0089] like Figure 5 As shown, the device comprises:

[0090] A hierarchical structure determination module 21 is used to divide the watershed into a number of grid units, use the grid units as calculation units of the model, and determine the hierarchical structure according to the topological relationship of the calculation units; wherein the hierarchical structure includes the number of levels and the calculation units to be executed at each level;

[0091] A task queue construction module 22 is used to construct a computing task queue to be executed, and push the computing unit and the unit information of the computing unit into the computing task queue to be executed based on the hierarchical structure; wherein one computing unit corresponds to one computing task;

[0092] The computing task execution module 23 is used to obtain computing tasks from the computing task queue to be executed, and distribute the computing tasks to different computing nodes so that the computing nodes execute the computing tasks in parallel.

[0093] In an optional implementation, the hierarchical structure determination module 21 is further configured to:

[0094] Obtaining a topological relationship table of the computing unit, and taking the computing unit whose upstream computing unit in the topological relationship table is empty as the first layer; wherein the topological relationship table records the upstream computing unit and the downstream computing unit of each computing unit;

[0095] Iterate the computing units of the latest layer. In each round of iteration, the downstream computing units of all computing units in the current layer are used as the next layer of the current layer until the iteration termination condition is met, and the layer that meets the iteration termination condition is used as the last layer; wherein, the iteration termination condition is that the downstream computing units of all computing units in the current layer are empty.

[0096] In an optional implementation, the unit information includes the level to which the current computing unit belongs, the current computing unit ID, and the upstream computing unit ID of the current computing unit.

[0097] In an optional implementation, the computing task execution module 23 is further used to:

[0098] Obtaining computing tasks of each level in sequence from the to-be-executed computing task queue, and assigning the computing tasks to different computing nodes;

[0099] Obtain the image according to the image address specified in the computing task, and schedule all the images to the corresponding computing nodes;

[0100] Creating a corresponding container instance for each of the computing tasks, and inputting the data files required by the computing tasks into the container instance;

[0101] Control multiple container instances to execute the computing task in parallel, and output the task execution result after the task is successfully executed.

[0102] In an optional implementation, the computing task execution module 23 is further used to:

[0103] According to the hierarchical structure, obtaining computing tasks from the to-be-executed computing task queue in units of hierarchies;

[0104] Constructing a queue of executed computing tasks, and pushing the successfully executed computing tasks into the queue of executed computing tasks;

[0105] When it is detected that all computing tasks of the same level are in the executed computing task queue, the computing tasks of the next level continue to be obtained from the to-be-executed computing task queue.

[0106] In an optional embodiment, the device is also used for:

[0107] When it is detected that the computing task fails to execute or the container instance corresponding to the computing task is in an abnormal state, the computing task is reallocated to other computing nodes until the computing task is successfully executed.

[0108] In an optional embodiment, the device is also used for:

[0109] The node load conditions of all the computing nodes are monitored in real time, and the allocation of the computing tasks is dynamically adjusted according to the node load conditions.

[0110] In a third aspect, an embodiment of the present invention provides an electronic device, see Figure 6 , which is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.

[0111] like Figure 6 As shown, the device includes:

[0112] A memory 31, used for storing computer programs;

[0113] A processor 32, configured to execute the computer program;

[0114] Wherein, when the processor 32 executes the computer program, the distributed hydrological model parallel computing method as described in any of the above embodiments is implemented.

[0115] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 31 and executed by the processor 32 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the electronic device.

[0116] The processor 32 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0117] The memory 31 can be used to store the computer program and / or module, and the processor 32 realizes various functions of the electronic device by running or executing the computer program and / or module stored in the memory 31, and calling the data stored in the memory 31. The memory 31 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 31 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0118] It should be noted that the above electronic device includes, but is not limited to, a processor and a memory. Those skilled in the art can understand that Figure 6 The structural diagram is merely an example of the electronic device described above and does not constitute a limitation on the electronic device, and may include more components than shown in the figure, or a combination of certain components, or different components.

[0119] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed, the distributed hydrological model parallel computing method described in any of the above embodiments is implemented.

[0120] It should be understood that the present invention implements all or part of the processes in the above-mentioned distributed hydrological model parallel computing method, and can also be completed by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned distributed hydrological model parallel computing method can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0121] The above description is only a preferred embodiment of the present invention, but the protection scope of the present invention is not limited thereto. It should be pointed out that for those skilled in the art, several equivalent obvious variations and / or equivalent substitutions can be made without departing from the technical principles of the present invention. These obvious variations and / or equivalent substitutions should also be regarded as the protection scope of the present invention.

Claims

1. A distributed hydrological model parallel computing method, characterized in that: include: Divide the watershed into a number of grid units, use the grid units as calculation units of the model, and determine a hierarchical structure according to the topological relationship of the calculation units; wherein the hierarchical structure includes the number of levels and the calculation units to be executed at each level; Constructing a queue of computing tasks to be executed, and pushing the computing units and the unit information of the computing units into the queue of computing tasks to be executed based on the hierarchical structure; wherein one computing unit corresponds to one computing task; Obtain computing tasks from the to-be-executed computing task queue, and assign the computing tasks to different computing nodes so that the computing nodes execute the computing tasks in parallel.

2. The distributed hydrological model parallel computing method according to claim 1, characterized in that: The determining of the hierarchical structure according to the topological relationship of the computing units includes: Obtaining a topological relationship table of the computing unit, and taking the computing unit whose upstream computing unit in the topological relationship table is empty as the first layer; wherein the topological relationship table records the upstream computing unit and the downstream computing unit of each computing unit; Iterate the computing units of the latest layer. In each round of iteration, the downstream computing units of all computing units in the current layer are used as the next layer of the current layer until the iteration termination condition is met, and the layer that meets the iteration termination condition is used as the last layer; wherein, the iteration termination condition is that the downstream computing units of all computing units in the current layer are empty.

3. The distributed hydrological model parallel computing method according to claim 1, characterized in that: The unit information includes the level to which the current computing unit belongs, the current computing unit ID, and the upstream computing unit ID of the current computing unit.

4. The distributed hydrological model parallel computing method according to claim 1, characterized in that: The obtaining of computing tasks from the to-be-executed computing task queue and allocating the computing tasks to different computing nodes so that the computing nodes execute the computing tasks in parallel includes: Obtaining computing tasks of each level in sequence from the to-be-executed computing task queue, and assigning the computing tasks to different computing nodes; Obtain the image according to the image address specified in the computing task, and schedule all the images to the corresponding computing nodes; Creating a corresponding container instance for each of the computing tasks, and inputting the data files required by the computing tasks into the container instance; Control multiple container instances to execute the computing task in parallel, and output the task execution result after the task is successfully executed.

5. The distributed hydrological model parallel computing method according to claim 4, characterized in that: The obtaining of computing tasks of each level in sequence from the queue of computing tasks to be executed includes: According to the hierarchical structure, obtaining computing tasks from the to-be-executed computing task queue in units of hierarchies; Constructing a queue of executed computing tasks, and pushing the successfully executed computing tasks into the queue of executed computing tasks; When it is detected that all computing tasks of the same level are in the executed computing task queue, the computing tasks of the next level continue to be obtained from the to-be-executed computing task queue.

6. The distributed hydrological model parallel computing method according to claim 4, characterized in that: The method further comprises: When it is detected that the computing task fails to execute or the container instance corresponding to the computing task is in an abnormal state, the computing task is reallocated to other computing nodes until the computing task is successfully executed.

7. The distributed hydrological model parallel computing method according to claim 1, characterized in that: The method further comprises: The node load conditions of all the computing nodes are monitored in real time, and the allocation of the computing tasks is dynamically adjusted according to the node load conditions.

8. A distributed hydrological model parallel computing device, characterized in that: include: A hierarchical structure determination module, used to divide the watershed into a number of grid units, use the grid units as calculation units of the model, and determine the hierarchical structure according to the topological relationship of the calculation units; wherein the hierarchical structure includes the number of levels and the calculation units to be executed at each level; A task queue construction module, used to construct a to-be-executed computing task queue, and push the computing unit and the unit information of the computing unit into the to-be-executed computing task queue based on the hierarchical structure; wherein one computing unit corresponds to one computing task; The computing task execution module is used to obtain computing tasks from the computing task queue to be executed, and distribute the computing tasks to different computing nodes so that the computing nodes execute the computing tasks in parallel.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program; Wherein, when the processor executes the computer program, the distributed hydrological model parallel computing method as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the distributed hydrological model parallel computing method according to any one of claims 1 to 7 is implemented.

Images