Process-adaptive geological environment data processing method

Abstract

The invention discloses a flow-adaptive geological environment data processing method, which comprises the following steps of: 1, selecting a required data processing model for an input data processing task; 2, constructing a directed acyclic graph (DAG) according to the steps of the data processing model; step 3, finding a path sequence from a starting node to an end point from a directed acyclic graph (DAG) by using a depth-first search algorithm, the sequence comprising a plurality of micro-services associated with each other; and 4, creating a data processing task by using a Spring Batch batch processing framework, and scheduling task execution by taking each node in the path sequence as a step. According to the invention, more efficient geological environment data processing management is realized, and the efficiency and flexibility of the whole data processing flow are improved.

Description

Technical Field

[0001] The present invention relates to the field of geological environment big data processing and analysis, and in particular to a process self-adaptive geological environment data processing method. Background Art

[0002] As for the methods, there are many ways to achieve self-adaptation of remote sensing data processing flow to improve its processing efficiency. In terms of container orchestration and scheduling, the tool Docker Compose can be used to run multiple container applications in a single machine environment; through Docker Compose, all containers or services required by the application can be configured using YML files to achieve container orchestration. However, the number and configuration of container instances defined in Docker Compose are static. Once the definition is completed, if an error occurs in a container, it will not be able to restart automatically, and the Docker Compose command needs to be re-executed. Another method is to use Apache Airflow, which is an open source task scheduling and workflow orchestration system that supports defining dependencies between tasks, timed scheduling, monitoring and other functions. However, the Apache Airflow framework is developed based on Python, and its adaptability to migration to the mainstream Spring Boot framework is poor, and it does not support batch processing tasks; therefore, the self-adaptation effect for the geological environment remote sensing data processing flow may not be good. Although both Docker Compose and Apache Airflow have certain advantages and functions, they have some limitations in processing the self-adaptation of geological environment remote sensing data flow; in order to better improve processing efficiency and adaptability, it may be necessary to explore other methods and tools that are more suitable for geological environment data processing.

[0003] Therefore, it is necessary to design a process-adaptive geological environment data processing method to overcome the above problems. Summary of the invention

[0004] In order to avoid the above problems, a process-adaptive geological environment data processing method is provided. According to the user's demand for geological environment remote sensing data processing, the algorithm model is deployed by using the Docker Swarm cluster to build microservices, and the container cloud resource scheduling strategy based on the Ant Kubernetes System (ACS) is implemented to replace the default scheduling strategy of the swarm cluster. By determining the dependencies between models and constructing a directed acyclic graph (DAG), the depth-first search algorithm is then used to determine the model combination required for a task as the task flow. At the same time, the Spring Batch task process processing mechanism is combined to select the appropriate processing strategy, and the tasks are carried out in order according to the process processing steps.

[0005] The present invention provides a process-adaptive geological environment data processing method, comprising the following steps:

[0006] Step 1: Select the required data processing model for the input data processing task;

[0007] Step 2: construct a directed acyclic graph DAG according to the steps of the data processing model;

[0008] Step 3: Use the depth-first search algorithm to find a path sequence from the start node to the end point from the directed acyclic graph DAG, which contains multiple interrelated microservices;

[0009] Step 4: Use the Spring Batch batch processing framework to create a data processing task, and take each node in the path sequence as a step to schedule the task execution.

[0010] Preferably, step 2 specifically includes:

[0011] 2.1 According to the steps of the model, split it into multiple step modules;

[0012] 2.2 Integrate multiple geological environment data processing models, and only retain one of the common processing steps of multiple models;

[0013] 2.3 Use Docker Swarm to deploy modules and form the required microservices;

[0014] 2.4 According to the steps of the geological environment data processing model, determine the dependencies between microservices and construct a directed acyclic graph DAG.

[0015] Preferably, step 2.3 is: first, build the required container on each node in the cluster; then, build a microservice on the built container, wherein the microservice includes one or more replicas, each replica represents a task; finally, schedule the tasks in the cluster through the resource scheduling strategy based on the ant colony system ACS to achieve the best task placement solution.

[0016] Preferably, in step 2.3: assuming that the size of the ant population is s, and the parameters α, β, and ρ are all given, the implementation steps of the scheduling strategy are as follows:

[0017] 1) Initialize the pheromone and iteration times of each node in the cluster;

[0018] First, calculate the resource R(j) of each node: Among them, R(j) represents the resources of node j, r′ m，j represents the available memory of node j, r m，j represents the total memory of node j, r′ p，jrepresents the available CPU of node j, r p，j represents the total CPU of node j, and Represent memory weight and CPU weight respectively, satisfying

[0019] Then, initialize the pheromone τ for each node 0，j :τ 0，j = R(j); At the initial time t = 0, the pheromone τ of node j 0，j The initial value of is equal to the resource R(j) of node j; this initial pheromone value is used to start the operation of the ant colony algorithm and is updated and adjusted according to the rules of the algorithm in subsequent iterations; the value of the initialized pheromone plays an important role in the convergence and performance of the algorithm.

[0020] 2) Randomly select a node for all ants to place the first task at the same time;

[0021] 3) Calculate the transition probability p(t, j) of each ant at each node: Among them, p(t, j) represents the probability of selecting node j at time t, τ t，j represents the pheromone at node j, η j represents the heuristic value at node j, α and β represent the importance of pheromone and heuristic value respectively;

[0022] 4) Each ant selects the next node to place the task;

[0023]

[0024] Among them, j represents the selected node, N represents the optional node set, q represents a random number, and q0 represents the exploration rate;

[0025] 5) After all ants have selected nodes, evaporate the pheromone;

[0026]

[0027] Among them, Δ represents the change of pheromone, ρ represents the evaporation rate of pheromone, 0<ρ<1, and P(k) represents the solution generated by ant k;

[0028] For each ant, when it places a task on a node, it immediately updates the pheromone of this node. Δ is generally less than 0, and the pheromone τ is calculated using the above formula, which significantly reduces the pheromone level of the selected node and makes it less important to the next ant; the remaining tasks will be distributed on the resources of each node, helping to optimize the overall resource utilization efficiency and avoid over-concentration of some nodes while ignoring other nodes.

[0029] In the ant colony algorithm, the process of ants choosing a path is based on the trade-off between pheromone concentration and heuristic value. Although pheromone evaporates and decreases in each iteration, ants can still make choices based on the current pheromone concentration and heuristic value.

[0030] Specifically, when choosing a path, the ant will decide the next move based on the probability. The formula in step 3) describes the probability transfer process of the ant's node selection, in which the pheromone concentration and the heuristic value will affect the probability calculation. The ant will decide the next node based on this probability, thereby realizing the path search and task allocation.

[0031] Although the pheromone will decrease in each iteration, the ants can still make appropriate choices based on the pheromone and heuristic value through probabilistic transfer. This trade-off and random selection process enables ants to explore in the search space and eventually find a better solution.

[0032] 6) Repeat steps 3) to 5) until all tasks are placed;

[0033] 7) Select the best solution at present;

[0034] 8) Repeat steps 2) to 7) until the iteration is completed;

[0035] 9) Select the best global solution:

[0036] The above formula means finding the solution P(k) with the highest frequency equal to the value of x. The meaning of this expression is to find the solution P(k) with the highest frequency among all the solutions generated by ants, and then determine what this solution with the highest frequency is.

[0037] Specifically, represents the sum of all solutions generated by ants, where [P(k) = x] is an indicator function, which takes the value of 1 when P(k) equals x, and 0 otherwise. By summing the solutions generated by all ants, the number of times each solution appears can be obtained. Then, It means finding the x value that maximizes the sum, that is, finding the x value corresponding to the solution P(k) with the highest frequency. Therefore, the meaning of this part is to find the x value corresponding to the solution P(k) with the highest frequency by counting all the solutions generated by ants, so as to determine the best task allocation solution.

[0038] This method fully considers the complexity and needs of geological environment remote sensing data processing. By using the DockerSwarm cluster and microservice architecture, combined with the container cloud resource scheduling strategy of the ant colony system ACS, the efficient deployment and management of the algorithm model is achieved. By constructing DAG and depth-first search algorithms, the model combination of task flows can be effectively determined, thereby improving the efficiency and accuracy of data processing. At the same time, combined with the Spring Batch task process processing mechanism, the task process and scheduling strategy can be better managed to ensure that tasks are executed in an orderly manner according to the predetermined process, further improving processing efficiency and quality. The comprehensive application of these methods and technologies can effectively meet users' needs for geological environment remote sensing data processing, provide efficient and intelligent data processing solutions, and provide strong support for data analysis and decision-making in the field of geological environment.

[0039] There are great differences in geological environment data processing models in different operating environments and different programming languages, so unified management is required. To achieve this goal, each model needs to be encapsulated into a portable container. By Dockerizing the model, a general model reasoning service can be provided to the outside world, which effectively solves the problem of heavy dependencies and difficult migration during the deployment of deep learning models, making model deployment more flexible and convenient. However, simply deploying the entire model does not ensure the correct execution of the intermediate processing process, so a better way is to disassemble the model into multiple modules and deploy each module independently. For multiple models, if there are common processing steps, the same module can be shared.

[0040] By using Docker Swarm to deploy modules as microservices in the cluster, the ACS algorithm is used to schedule the tasks that constitute the microservices, and the tasks are placed on each node in the best solution; these microservices are used as DAG nodes, and according to the process of model execution, the relationship between each node is determined and directed edges are added to build a directed acyclic graph (DAG). This microservice-based DAG construction method can better reflect the dependencies and processes between geological environment data processing models, and provide a more flexible and efficient way for the management and optimization of data processing processes.

[0041] For a given geological environment data processing task, the required processing model must be determined first. Then, based on the constructed DAG, the depth-first search algorithm is used to find the microservice sequence for data processing execution and combine it into a Spring Batch scheduling task Job. Each microservice will be executed in sequence in the sequence found by topological sorting as a step. During the task execution process, if an error occurs in a step, the task can be retried at a breakpoint. Finally, when the task is executed without error, a geological environment data product will be generated.

[0042] By utilizing the container resource scheduling strategy based on the ant colony system, load balancing and more efficient container orchestration of the Docker Swarm cluster are achieved. Combined with the constructed DAG, the process of geological environment data processing tasks is adaptively determined. At the same time, Spring Batch is used to monitor the execution of tasks to ensure that the tasks can be completed reliably. This comprehensive approach realizes efficient geological environment data processing services, improves processing efficiency and reliability, and provides a better solution for geological environment data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flowchart of a method for processing geological environment data with process self-adaptation according to a preferred embodiment of the present invention;

[0044] Figure 2 A detailed flow of the Docker Swarm cluster deployment module steps of a preferred embodiment of the present invention;

[0045] Figure 3 This is an example diagram of a directed acyclic graph built based on microservices according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0046] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

[0047] A process-adaptive geological environment data processing method, Figure 1 The middle right part is the process of geological environment data processing, which includes the following steps:

[0048] Step 1: For the data processing task input by the user, first select the required model.

[0049] Step 2: Use the depth-first search algorithm in the DAG according to the steps of the data processing model to find a path sequence from the starting node to the end point (for example, the result output), which contains multiple interrelated microservices.

[0050] The steps to construct DAG are as follows: Figure 1 As shown in the left half, the detailed construction process is as follows:

[0051] 2.1 For a given geological environment data processing model, first split it into multiple step modules according to the steps of the model;

[0052] 2.2 Integrate multiple geological environment data processing models, and only retain one of the common processing steps of multiple models;

[0053] 2.3 Use Docker Swarm to deploy modules and form the required microservices; the detailed process of Docker Swarm cluster deployment of microservices and resource scheduling is as follows Figure 2 As shown:

[0054] First, build the required containers on each node in the cluster, and then build microservices on the built containers. Microservices can contain multiple replicas, each of which represents a task. The tasks in the cluster are scheduled through the resource scheduling strategy based on the ant colony system ACS to achieve the best task placement solution. Assuming that the size of the ant population is s, and the parameters α, β, and ρ are all given, the implementation steps of the scheduling strategy are as follows:

[0055] 1) Initialize the pheromone and iteration times of each node in the cluster;

[0056] First, calculate the resource R(j) of each node:

[0057]

[0058] Among them, R(j) represents the resources of node j,

[0059] r′ m，j represents the available memory of node j,

[0060] r m，j represents the total memory of node j,

[0061] r′ p，j represents the available CPU of node j,

[0062] r p，j represents the total CPU of node j,

[0063] and Represent memory weight and CPU weight respectively, satisfying

[0064] Then, initialize the pheromone τ for each node 0，j :

[0065] τ 0，j = R(j);

[0066] Indicates that at the initial time t = 0, the pheromone τ of node j 0，j The initial value of is equal to the resource R(j) of node j; this initial pheromone value is used to start the operation of the ant colony algorithm and is updated and adjusted according to the rules of the algorithm in subsequent iterations; the value of the initialized pheromone plays an important role in the convergence and performance of the algorithm.

[0067] 2) Randomly select a node for all ants to place the first task at the same time;

[0068] 3) Calculate the transition probability p(t, j) of each ant at each node:

[0069]

[0070] Among them, p(t, j) represents the probability of selecting node j at time t,

[0071] τ t，j represents the pheromone at node j,

[0072] η j represents the heuristic value at node j,

[0073] α and β represent the importance of pheromone and heuristic value, respectively;

[0074] 4) Each ant selects the next node to place the task;

[0075]

[0076] Among them, j represents the selected node,

[0077] N represents an optional node set.

[0078] q represents a random number,

[0079] q0 represents the exploration rate;

[0080] 5) After all ants have selected nodes, evaporate the pheromone;

[0081]

[0082] Among them, Δ represents the change of pheromone,

[0083] ρ represents the evaporation rate of pheromone, 0<ρ<1,

[0084] P(k) represents the solution generated by ant k;

[0085] For each ant, when it places a task on a node, it immediately updates the pheromone of this node. Δ is generally less than 0, and the pheromone τ is calculated using the above formula, which significantly reduces the pheromone level of the selected node and makes it less important to the next ant; the remaining tasks will be distributed on the resources of each node, helping to optimize the overall resource utilization efficiency and avoid over-concentration of some nodes while ignoring other nodes.

[0086] In the ant colony algorithm, the process of ants choosing a path is based on the trade-off between pheromone concentration and heuristic value. Although pheromone evaporates and decreases in each iteration, ants can still make choices based on the current pheromone concentration and heuristic value.

[0087] Specifically, when choosing a path, the ant will decide the next move based on probability. The formula in step 3) describes the probability transfer process of the ant's node selection, in which the pheromone concentration and the heuristic value will affect the probability calculation. The ant will decide the next node based on this probability, thereby realizing the path search and task allocation.

[0088] Although the pheromone will decrease in each iteration, the ants can still make appropriate choices based on the pheromone and heuristic value through probabilistic transfer. This trade-off and random selection process enables ants to explore in the search space and eventually find a better solution.

[0089] 6) Repeat steps 3) to 5) until all tasks are placed;

[0090] 7) Select the best solution at present;

[0091] 8) Repeat steps 2) to 7) until the iteration is completed;

[0092] 9) Select the best global solution:

[0093]

[0094] The above formula means finding the solution P(k) with the highest frequency equal to the value of x. The meaning of this expression is to find the solution P(k) with the highest frequency among all the solutions generated by ants, and then determine what this solution with the highest frequency is.

[0095] Specifically, represents the sum of all solutions generated by ants, where [P(k) = x] is an indicator function, which takes the value of 1 when P(k) equals x, and 0 otherwise. By summing the solutions generated by all ants, the number of times each solution appears can be obtained. Then, It means finding the x value that maximizes the sum, that is, finding the x value corresponding to the solution P(k) with the highest frequency. Therefore, the meaning of this part is to find the x value corresponding to the solution P(k) with the highest frequency by counting all the solutions generated by ants, so as to determine the best task allocation solution.

[0096] 2.4 According to the steps of the geological environment data processing model, determine the dependencies between microservices and construct a directed acyclic graph DAG.

[0097] Step 3: Use the Spring Batch batch processing framework to create a data processing task, and use each node in the path sequence as a step to schedule the task execution. In the process of scheduling task execution, Spring Batch provides a fault-tolerant processing mechanism and a breakpoint retransmission mechanism to ensure the reliability and accuracy of the processing flow.

[0098] Figure 3 This is a simple example of a directed acyclic graph of multiple geological environment data processing models built based on microservices, including a wetland mapping model, an open space product production model, and an urban village product production model. Each node represents a data processing step, and each subsequent step can only be executed after the previous step is completed. Since the microservices deployed by Docker Swarm are not one-time tasks, each microservice needs to be closed after execution before continuing to the next step.

[0099] Step 1 is a common step for multiple models, which is to read the required remote sensing image according to the provided input parameters.

[0100] Step 2 is a common step for multiple models, which is to merge the input remote sensing image size required and crop the initial input image.

[0101] Step 3: For wetland mapping tasks, the SegForm model is needed to predict the cropped images and generate the required products; for open space product production tasks, the MANet model is needed to predict the cropped images and generate the required products; for urban village product production tasks, the ResNet model is needed to predict the cropped images and generate the required products.

[0102] Step 4 is a common step for multiple models, which is to stitch the predicted image products to generate a product of the same size as the initial image.

[0103] Step 5 is a common step for multiple models, which outputs the results to the specified file system.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A process-adaptive geological environment data processing method, characterized in that: The steps include: Step 1: Select the required data processing model for the input data processing task; Step 2: construct a directed acyclic graph DAG according to the steps of the data processing model; Step 3: Use the depth-first search algorithm to find a path sequence from the start node to the end point from the directed acyclic graph DAG, which contains multiple interrelated microservices; Step 4: Use the Spring Batch batch processing framework to create a data processing task, and take each node in the path sequence as a step to schedule the task execution.

2. The process-adaptive geological environment data processing method as claimed in claim 1, characterized in that: Step 2 specifically includes: 2.1 According to the steps of the model, split it into multiple step modules; 2.2 Integrate multiple geological environment data processing models, and only retain one of the common processing steps of multiple models; 2.3 Use Docker Swarm to deploy modules and form the required microservices; 2.4 According to the steps of the geological environment data processing model, determine the dependencies between microservices and construct a directed acyclic graph DAG.

3. The process-adaptive geological environment data processing method as claimed in claim 2, characterized in that: Step 2.3 is: first, build the required containers on each node in the cluster; then, build microservices on the built containers, where the microservices contain one or more replicas, each replica represents a task; finally, schedule the tasks in the cluster through the resource scheduling strategy based on the ant colony system ACS to achieve the best task placement solution.

4. The process-adaptive geological environment data processing method as claimed in claim 3, characterized in that: In step 2.3: Assuming that the size of the ant population is s, and the parameters α, β, and ρ are given, the implementation steps of the scheduling strategy are as follows: 1) Initialize the pheromone and iteration times of each node in the cluster; First, calculate the resource R(j) of each node: Among them, R(j) represents the resources of node j, r ′ m,j represents the available memory of node j, r m,j represents the total memory of node j, r p ′ ,j represents the available CPU of node j, r p,j represents the total CPU of node j, and Represent memory weight and CPU weight respectively, satisfying Then, initialize the pheromone τ for each node 0,j :τ 0,j = R(j); At the initial time t = 0, the pheromone τ of node j 0,j The initial value of is equal to the resource R(j) of node j; 2) Randomly select a node for all ants to place the first task at the same time; 3) Calculate the transition probability p(t,j) of each ant at each node: Among them, p(t,j) represents the probability of selecting node j at time t, τ t,j represents the pheromone at node j, η j represents the heuristic value at node j, α and β represent the importance of pheromone and heuristic value respectively; 4) Each ant selects the next node to place the task; Among them, j represents the selected node, N represents the optional node set, q represents a random number, and q0 represents the exploration rate; 5) After all ants have selected nodes, evaporate the pheromone; Where Δ represents the change in pheromone, ρ represents the evaporation rate of pheromone, 0<ρ<1, and P(k) represents the solution generated by ant k; 6) Repeat steps 3) to 5) until all tasks are placed; 7) Select the best solution at present; 8) Repeat steps 2) to 7) until the iteration is completed; 9) Select the best global solution:

Images