A data migration and upgrade method based on multi-node parallelism

Abstract

The present invention discloses a data migration and upgrade method based on multi-node parallelism, including the steps S1. data acquisition; S2. generating a task table; S3. node scheduling; S4. task scheduling; S5. data storage task execution; the present invention separates the coupling relationship between multi-disciplinary data types, and uses multi-node parallel execution of data upgrade and migration operations, thereby realizing the rapid upgrade and migration of new and old GeoEast system exploration data, and improving the security and health of large-scale seismic exploration system data. The present invention can quickly migrate and upgrade heterogeneous data, and improve the security and health of large-scale seismic exploration system data; the present invention is applicable to the seismic exploration industry, and is used for rapid upgrade and migration of new and old GeoEast system exploration data.

Description

Technical Field

[0001] The invention belongs to the field of geophysical exploration, and specifically is a data migration and upgrading method based on multi-node parallelism. Background Art

[0002] As open source database technology matures, some industrial software uses PostgreSQL database. However, how to quickly migrate the historical data of old versions of software in commercial databases to PostgreSQL database becomes the key to reuse the software's historical data.

[0003] Seismic exploration data consists of two categories: database and file data. During the data migration and upgrade process of the new and old versions of software, due to the large volume of historical data, usually hundreds of TB to PB, the migration and upgrade rate of single node data is slow, affecting the user experience of data use. Therefore, the migration and upgrade rate of seismic exploration historical data has become an urgent problem to be solved in the iteration of new and old software. Summary of the invention

[0004] In order to solve the above deficiencies in the prior art, the present invention aims to provide a data migration and upgrade method based on multi-node parallelism, so as to achieve the purpose of rapid migration and upgrade of heterogeneous data.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0006] A data migration and upgrade method based on multi-node parallelism, the method comprising the following steps:

[0007] S1. Data collection: used to reorganize and segment existing data in low-version databases and file storage systems; this step includes the following:

[0008] S11. Connect to the lower version data environment;

[0009] Create a database access connection, configure the ORACLE database address and port number, and create data configuration information and data storage path for the exploration database;

[0010] S12. Create a minimum unit for data collection;

[0011] Taking the project data storage structure as the minimum data collection unit, define the three-level data storage management nodes of project, work area and survey line. Correspondingly, earthquake, well, layer, fault, log, attribute and speed in the data type node are mounted under the work area and survey line data storage management nodes respectively, and the corresponding data entity is mounted under the data type node;

[0012] S13. Obtaining data storage information;

[0013] According to the data configuration information of step S11 and the created database access connection, connect to the database and obtain data authority authentication;

[0014] Traverse the data storage management nodes in step S12 in order, obtain the data model characteristics of the smallest model unit of the storage mode SCHEMA according to the names and data storage characteristics of all projects in the current database, and obtain the table name, column name, index, trigger, foreign key, and primary key under the current SCHEMA in turn to form the smallest unit aggregate of the project data migration task;

[0015] S2. Generate a task list;

[0016] According to the minimum unit aggregate obtained in step S13, start to split the task, generate the task tree, and formulate the task execution list; this step includes the following contents:

[0017] S21. Task path generation;

[0018] Traverse the minimum unit aggregate of the project data migration task in step S13 in order, obtain data model features, and specify the version number of the current SCHEMA based on the data model features; determine the starting and ending paths and the upgrade program within the path range according to the version number of the current SCHEMA and the version number of the upgrade target, perform tree encapsulation on the tasks in the generated SCHEMA, and determine the upgrade path and the order of the upgrade execution code;

[0019] S22. Task segmentation;

[0020] Based on the task path determined in step S21, the task contents involved in the current task path are sequentially segmented according to the project data storage structure in step S12;

[0021] S3. Node scheduling; this step includes the following:

[0022] S31. Define the hostname table of the cluster, add node configuration information, and specify the master control node and data migration task execution node;

[0023] S32. The master control node obtains the load information of the cluster nodes, the real-time network bandwidth, and the node read and write rate according to the node configuration information in step S31, and sends a task execution request to the master control node based on the load of each data migration task execution node;

[0024] S33. The main control node pushes out the execution tasks according to the task list divided in step S22; after the multiple execution nodes in the idle state obtain the tasks, they perform a task migration check and synchronization verification on the database. When the data verification is qualified, it means that the baseline of data migration is consistent, and the dispatch upgrade task starts to be executed at this time;

[0025] S4. Task scheduling; this step specifically includes the following:

[0026] S41. Data extraction task distribution;

[0027] The main control node obtains the extraction task in the task according to the task list divided in step S22, and starts distributing the data extraction task to the execution node in the order of database-project-work area-survey line-data entity type;

[0028] S42. Data conversion task distribution;

[0029] After the data extraction task in step S41 is completed, the task execution status is pushed to the data conversion task, and the data conversion node starts to execute table conversion; at the same time, a project with the same name is created on the high-version database server, and the high-version database INSERT SQL script is written, and the result value of the database table extraction and filtering in step S41 is bound to the INSERT script and written into the high-version database;

[0030] S43. Data verification task execution;

[0031] This task is automatically executed by each task execution node after completing step S42;

[0032] After the conversion of the library table data in step S41 is completed, the conversion of files and unstructured data begins. During the conversion process, each execution node continues to send data verification packets to the main control node, and the data conversion execution node is scheduled according to the verification logic of step S33;

[0033] S5. Data storage task execution; this step includes the following:

[0034] S51. After step S43 is completed, the data of the entire project is completed and large blocks of data are merged;

[0035] S52. Configuration parameter merging;

[0036] Write the database, file data, and configuration parameters into the configuration information of the target system to complete data migration and upgrade.

[0037] As a limitation of the present invention: the data configuration information of the database in step S11 includes hostname, dbname, ip, port and user.

[0038] As a further limitation of the present invention, the sequential traversal in step S13 refers to traversing the data under each survey line according to the overall route of project-work area-survey line, returning to the previous level after the traversal is completed, traversing the data under each work area, and returning to the previous level after the traversal is completed, traversing the data under each project.

[0039] As another limitation of the present invention, step S21 specifically includes the following contents:

[0040] S211. Define the data entity upgrade version number entity set E={e1,e2,…en}, where ei represents the version number of the current upgraded data entity model;

[0041] S212. Define the association and dependency characteristics of data between different versions D={(ei, ej)}, where (ei, ej) represents the order of data upgrade;

[0042] S213. Define the data resource consumption vector R=[r1,r2…,rm], where ri represents the nodes, bandwidth and storage required for different volumes of data;

[0043] S214. Define a cost matrix C=[Cij], where Cij represents the cost of data model entity ei-ej;

[0044] S215. Construct a directed graph;

[0045] Construct a directed graph G=(V,A) based on E and D, where the vertex set V=E, the arc set A=D, and the direction of the arc indicates the direction of the dependency relationship, from the dependent data entity to the dependent data entity. Each arc is the resource consumption C during the upgrade process;

[0046] S216. Based on the topological sorting algorithm of the in-degree table, sort the directed graph G constructed in step S215 to find a linear order that satisfies the dependency relationship.

[0047] As a further limitation of the present invention: each data type task in step S22 is divided into data verification and data execution, wherein the data execution task is mapped from a low version of project-work area-survey line-data type-data entity to a high version of project-work area-survey line-data type-data entity according to the data configuration information in step S11.

[0048] As a further limitation of the present invention: in the synchronous verification process in step S33, the main control node periodically sends a data verification packet to the execution node, and after the execution node obtains the verification packet, it returns the current task execution parameters to the main node, where the task execution parameters include execution time, execution task, execution real-time bandwidth and other parameters;

[0049] The main control node waits for the verification packet to return data. If it times out, it considers that the execution node task has failed. At this time, it continues to connect with the execution node for tasks. If the connection is successful, the failed task will be pushed again. If the continuous connection is unsuccessful, the database and file writing status of the data storage center will be verified. If the task is determined to have failed, the failed task will be assigned to the next execution node in the cluster hostname table.

[0050] As another limitation of the present invention: Step S41 is specifically that the execution node accesses and filters the low-version database; after connecting to the database, the database table structure is obtained, the column data of each entity data type is filtered, and the filter value rule=operator(And / Or / Not)value, Rules={rulei(And / Or)rulej} is matched in sequence; at the same time, the values ​​of the data filter column are frozen, modification and update operations are prohibited, and the values ​​are checked for null values, character abnormal values, and out-of-limit values; the formulated screening rules and the storage path of the filtering result data are pushed to the task execution node; if the push fails, the node scheduling of the main control node is sent to obtain a new execution list.

[0051] Due to the adoption of the above technical solution, the present invention has the following beneficial effects compared with the prior art:

[0052] The present invention starts with the seismic exploration database model and node status, and provides a data migration and upgrade method based on multi-node parallelism, which includes S1. data acquisition; S2. generating a task table; S3. node scheduling; S4. task scheduling; S5. data storage task execution. The present invention separates the coupling relationship between multi-disciplinary data types, uses multi-node parallel execution of data upgrade and migration operations, realizes the rapid upgrade and migration of new and old GeoEast system exploration data, and improves the security and health of large-scale seismic exploration system data.

[0053] In summary, the present invention can quickly migrate and upgrade heterogeneous data, thereby improving the security and robustness of data in large-scale seismic exploration systems; the present invention is suitable for the seismic exploration industry and is used to quickly upgrade and migrate exploration data of new and old GeoEast systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] Figure 1 It is a flowchart of the overall process of an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of new and old system configuration information in an embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram of the structure of the project data storage in an embodiment of the present invention;

[0058] Figure 4 This is a schematic diagram of the minimum model unit structure in an embodiment of the present invention;

[0059] Figure 5 It is a schematic diagram of the encapsulation of the task tree in an embodiment of the present invention;

[0060] Figure 6 A flowchart of data extraction task distribution in an embodiment of the present invention;

[0061] Figure 7 A statistical chart of multi-node parallel upgrade of processing explanation data generated by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The preferred embodiment of the present invention is described below in conjunction with the accompanying drawings. It should be understood that the multi-node parallel data migration and upgrade method described herein is a preferred embodiment, which is only used to illustrate and explain the present invention and does not constitute a limitation of the present invention.

[0063] like Figure 1 As shown, this embodiment includes the following steps:

[0064] S1. Data collection;

[0065] This step is used to reorganize and split the existing data in the low-version database and file storage system. It should be noted that before and after the reorganization and splitting, the seismic exploration data feature information and project deployment information are stored in the database, and the fragmented files and large files are stored in the disk path.

[0066] Step S1 specifically includes the following contents:

[0067] S11. Connect to the lower version data environment;

[0068] Create a database access connection, configure the ORACLE database address and port number, and create the data configuration information and data storage path for the exploration database. Figure 2 , where the database data configuration information includes hostname, dbname, ip, port, user, etc.

[0069] S12. Create a minimum unit for data collection;

[0070] Taking the project data storage structure as the minimum data collection unit, define the three-level data storage management nodes of project, work area and survey line. Correspondingly, earthquake, well, layer, fault, log, attribute and speed in the data type node are mounted under the work area and survey line data storage management nodes respectively, and the corresponding data entity is mounted under the data type node. Figure 3 .

[0071] S13. Obtaining data storage information;

[0072] According to the data configuration information of step S11 and the created database access connection, the database is connected to obtain data authority authentication. The database in the connected database here refers to the low-version data used by the low-version exploration software during operation, including the low-version database and the existing data of the file storage system.

[0073] Traverse the data storage management nodes in step S12 in order, and obtain the data model characteristics of the smallest model unit of the storage mode SCHEMA according to the names and data storage characteristics of all items in the current database, such as Figure 4 As shown, the table name, column name, index, trigger, foreign key, and primary key under the current SCHEMA are obtained in turn to form the smallest unit collection of the project data migration task.

[0074] like Figure 3 As shown in the figure, there are multiple projects under one database, multiple work areas under each project, multiple survey lines under each work area, and each work area and survey line have a corresponding complete data storage type structure. The above-mentioned sequential traversal means that according to the overall route of project-work area-survey line, the data under each survey line is traversed, and after the traversal is completed, it returns to the previous level to traverse the data under each work area, and after the traversal is completed, it returns to the previous level to traverse the data under each project.

[0075] S2. Generate a task list;

[0076] According to the minimum unit aggregate obtained in step S13, start to split the tasks, generate a task tree, and formulate an execution list of the tasks; this step includes the following contents:

[0077] S21. Task path generation;

[0078] Traverse the minimum unit collection of the project data migration task in step S13 in order, obtain the data model characteristics, and specify the version number of the current SCHEMA based on the data model characteristics; determine the starting and ending paths and the upgrade program within the path range according to the version number of the current SCHEMA and the version number of the upgrade target, encapsulate the tasks in the generated SCHEMA in a tree shape, and determine the upgrade path and the order of the upgrade execution code, see Figure 5 .

[0079] Specifically, step S21 includes the following contents:

[0080] S211. Define a data entity upgrade version number entity set E={e1,e2,…en}, where ei represents the version number of the current upgraded data entity model.

[0081] S212. Define the association and dependency characteristics of data between different versions D={(ei, ej)}, where (ei, ej) represents the order of data upgrade. There is an order constraint relationship between the high and low versions of the data entity model. If you upgrade to version j first and then execute version i, the data upgrade script and optimized SQL script will fail to execute.

[0082] S213. Define the data resource consumption vector R=[r1,r2…,rm], where ri represents the nodes, bandwidth and storage required for different volumes of data; this step is established because different types of data have large differences in resource consumption during the upgrade process.

[0083] S214. Define the cost matrix C=[Cij], where Cij represents the cost of the data model entity ei-ej.

[0084] S215. Construct a directed graph;

[0085] A directed graph G=(V,A) is constructed based on E and D, where the vertex set V=E, that is, the vertices of the directed graph are the data model entities, the arc set A=D, and the direction of the arc represents the direction of the dependency relationship, from the dependent data entity to the dependent data entity; each arc represents the resource consumption C of the upgrade process, and the directed graph contains the nodes and the relationships between the nodes, and the relationships between the nodes are represented by the values ​​on the arcs. Since the directed graph is a prior art, it will not be described in detail here.

[0086] S216. Based on the topological sorting algorithm of the in-degree table, sort the directed graph G constructed in step S215 to find a linear order that satisfies the dependency relationship.

[0087] S22. Task segmentation;

[0088] Based on the task path determined in step S21, the task contents involved in the current task path are divided in turn according to the project data storage structure in step S12. Specifically, each data type task is divided into data verification and data execution, wherein the data execution task is mapped from the low version of project-work area-survey line-data type-data entity to the high version of project-work area-survey line-data type-data entity according to the data configuration information in step S11.

[0089] S3. Node scheduling; this step includes the following:

[0090] S31. Define the hostname table of the cluster, add node configuration information, and specify the main control node and the data migration task execution node. The node in the node configuration information here refers to the entire process of the control node and storage node of the data execution task.

[0091] S32. The master control node periodically obtains the load information, network real-time bandwidth, and node read and write rates of the cluster nodes according to the node configuration information in step S31. The periodic acquisition here can be set to be once every 5 minutes, and based on the load conditions of each data migration task execution node, sends a task execution request to the master control node.

[0092] It should be noted that a cluster node is a combination of multiple computer nodes, where each computer can be understood as a node. When a migration task is executed, one of the nodes is a control node, also known as a master node, which is used to schedule tasks for other nodes, and the node that executes the migration is an execution node.

[0093] S33. The main control node pushes out the execution tasks according to the task list divided in step S22. After the idle execution nodes obtain the tasks, they perform a task migration check and synchronization verification on the database. If the data verification is qualified, it means that the baseline of data migration is consistent, and then the assigned upgrade task is executed.

[0094] In addition, during the synchronization verification process, the main control node periodically sends data verification packets to the execution node. After the execution node obtains the verification packet, it returns the current task execution parameters to the main node. The task execution parameters here include execution time, execution task, execution real-time bandwidth and other parameters.

[0095] The main control node waits for the verification packet to return data. If it times out, it considers that the execution node task has failed. At this time, it continues to connect with the execution node for tasks. If the connection is successful, the failed task will be pushed again. If the continuous connection is unsuccessful, the database and file writing status of the data storage center will be verified. If the task is determined to have failed, the failed task will be assigned to the next execution node in the cluster hostname table.

[0096] S4. Task scheduling; this step specifically includes the following:

[0097] S41. Data extraction task distribution;

[0098] The main control node obtains the extraction task in the task according to the task list divided in step S22, and starts distributing the data extraction task to the execution node in the order of database-project-work area-survey line-data entity type.

[0099] Specifically, the execution node accesses and filters the lower version database, such as Figure 6 As shown, after connecting to the database, the database table structure is obtained, the column data of each entity data type is filtered, and the filter value rule=operator(And / Or / Not)value, Rules={rulei(And / Or)rulej} is matched in sequence; at the same time, the values ​​of the data filter column are frozen, modification and update operations are prohibited, and the values ​​are checked for null values, character abnormal values, and out-of-limit values; the formulated filtering rules and the storage path of the filtering result data are pushed to the task execution node; if the push fails, the node scheduling of the main control node is sent to obtain a new execution list.

[0100] S42. Data conversion task distribution;

[0101] After the data extraction task is completed in step S41, the task execution status is pushed to the data conversion task, and the data conversion node starts to execute table conversion. It should be noted that there is a sequence here, and the extraction must be completed and there is a temporary path for storing data results in the task.

[0102] At the same time, on the high-version database server, create a project with the same name, write the high-version database INSERT SQL script, bind the result value of the database table extraction and filtering in step S41 to the INSERT script, and write it into the high-version database.

[0103] S43. Data verification task execution;

[0104] This task is automatically executed by each task execution node after completing step S42;

[0105] After the conversion of the library table data in step S41 is completed, the conversion of files and unstructured data begins. During the conversion process, each execution node continues to send data verification packets to the main control node, and the data conversion execution node is scheduled according to the verification logic of step S33.

[0106] S5. Data storage task execution; this step includes the following:

[0107] S51. After step S43 is completed, the data of the entire project is completed and large blocks of data are merged.

[0108] S52. Configuration parameter merging;

[0109] Write the database, file data, and configuration parameters into the configuration information of the target system to complete data migration and upgrade.

[0110] For example, a user's low-version seismic exploration data project contains a data volume of 10TB, which is divided into multiple small data and stored in multiple paths. Then, the storage name and path of all small data in the data volume are obtained from the database, converted into high-version data through migration, and stored in the path specified by the user.

[0111] The present invention separates the coupling relationship between multidisciplinary data types, performs data upgrade and migration operations based on database model characteristics and multi-node parallelism, and completes the rapid upgrade and migration of exploration data of the new and old GeoEast systems; it can maximize the user's real-time bandwidth, and the upgrade speed is proportional to the number of nodes within a relative range. For the first time, cross-storage media direct connection upgrade and migration of data has been realized, such as Figure 7 As shown in the figure, the migration speed is directly related to the real-time bandwidth of the cluster, and the overall range of the migration speed is 1-2.92T / h.

[0112] It should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention is described in detail with reference to the above embodiment, those skilled in the art can still modify the technical solutions described in the above embodiment or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data migration and upgrade method based on multi-node parallelism, characterized in that: The method comprises the following steps: S1. Data collection: used to reorganize and segment existing data in low-version databases and file storage systems; this step includes the following: S11. Connect to the lower version data environment; Create a database access connection, configure the ORACLE database address and port number, and create data configuration information and data storage path for the exploration database; S12. Create a minimum unit for data collection; Taking the project data storage structure as the minimum data collection unit, define the three-level data storage management nodes of project, work area and survey line. Correspondingly, earthquake, well, layer, fault, log, attribute and speed in the data type node are mounted under the work area and survey line data storage management nodes respectively, and the corresponding data entity is mounted under the data type node; S13. Obtaining data storage information; According to the data configuration information of step S11 and the created database access connection, connect to the database and obtain data authority authentication; Traverse the data storage management nodes in step S12 in order, obtain the data model characteristics of the smallest model unit of the storage mode SCHEMA according to the names and data storage characteristics of all projects in the current database, and obtain the table name, column name, index, trigger, foreign key, and primary key under the current SCHEMA in turn to form the smallest unit aggregate of the project data migration task; S2. Generate a task list; According to the minimum unit aggregate obtained in step S13, start to split the task, generate the task tree, and formulate the task execution list; this step includes the following contents: S21. Task path generation; Traverse the minimum unit aggregate of the project data migration task in step S13 in order, obtain data model features, and specify the version number of the current SCHEMA based on the data model features; determine the starting and ending paths and the upgrade program within the path range according to the version number of the current SCHEMA and the version number of the upgrade target, perform tree encapsulation on the tasks in the generated SCHEMA, and determine the upgrade path and the order of the upgrade execution code; S22. Task segmentation; Based on the task path determined in step S21, the task contents involved in the current task path are sequentially segmented according to the project data storage structure in step S12; S3. Node scheduling; this step includes the following: S31. Define the hostname table of the cluster, add node configuration information, and specify the master control node and data migration task execution node; S32. The master control node obtains the load information of the cluster nodes, the real-time network bandwidth, and the node read and write rate according to the node configuration information in step S31, and sends a task execution request to the master control node based on the load of each data migration task execution node; S33. The main control node pushes out the execution tasks according to the task list divided in step S22; after the multiple execution nodes in the idle state obtain the tasks, they perform a task migration check and synchronization verification on the database. When the data verification is qualified, it means that the baseline of data migration is consistent, and the dispatch upgrade task starts to be executed at this time; S4. Task scheduling; this step specifically includes the following: S41. Data extraction task distribution; The main control node obtains the extraction task in the task according to the task list divided in step S22, and starts distributing the data extraction task to the execution node in the order of database-project-work area-survey line-data entity type; S42. Data conversion task distribution; After the data extraction task in step S41 is completed, the task execution status is pushed to the data conversion task, and the data conversion node starts to execute table conversion; at the same time, a project with the same name is created on the high-version database server, and the high-version database INSERT SQL script is written, and the result value of the database table extraction and filtering in step S41 is bound to the INSERT script and written into the high-version database; S43. Data verification task execution; This task is automatically executed by each task execution node after completing step S42; After the conversion of the library table data in step S41 is completed, the conversion of files and unstructured data begins. During the conversion process, each execution node continues to send data verification packets to the main control node, and the data conversion execution node is scheduled according to the verification logic of step S33; S5. Data storage task execution; this step includes the following: S51. After step S43 is completed, the data of the entire project is completed and large blocks of data are merged; S52. Configuration parameter merging; Write the database, file data, and configuration parameters into the configuration information of the target system to complete data migration and upgrade.

2. According to claim 1, a multi-node parallel data migration and upgrade method is characterized in that: The data configuration information of the database in step S11 includes hostname, dbname, ip, port and user.

3. The method for data migration and upgrading based on multi-node parallelism according to claim 2 is characterized in that: The sequential traversal in step S13 means traversing the data under each survey line according to the overall route of project-work area-survey line, returning to the previous level after the traversal is completed, traversing the data under each work area, and returning to the previous level after the traversal is completed, traversing the data under each project.

4. A multi-node parallel data migration and upgrade method according to any one of claims 1 to 3, characterized in that: Step S21 specifically includes the following contents: S211. Define the data entity upgrade version number entity set E={e1,e2,…en}, where ei represents the version number of the current upgraded data entity model; S212. Define the association and dependency characteristics of data between different versions D={(ei, ej)}, where (ei, ej) represents the order of data upgrade; S213. Define the data resource consumption vector R=[r1,r2…,rm], where ri represents the nodes, bandwidth and storage required for different volumes of data; S214. Define a cost matrix C=[Cij], where Cij represents the cost of data model entity ei-ej; S215. Construct a directed graph; Construct a directed graph G=(V,A) based on E and D, where the vertex set V=E, the arc set A=D, and the direction of the arc indicates the direction of the dependency relationship, from the dependent data entity to the dependent data entity. Each arc is the resource consumption C during the upgrade process; S216. Based on the topological sorting algorithm of the in-degree table, sort the directed graph G constructed in step S215 to find a linear order that satisfies the dependency relationship.

5. The method for data migration and upgrading based on multi-node parallelism according to claim 4 is characterized in that: In step S22, each data type task is divided into data verification and data execution, wherein the data execution task is mapped from the low version project-work area-survey line-data type-data entity to the high version project-work area-survey line-data type-data entity according to the data configuration information in step S11.

6. The method for data migration and upgrading based on multi-node parallelism according to claim 5 is characterized in that: During the synchronization verification process in step S33, the main control node periodically sends a data verification packet to the execution node. After the execution node obtains the verification packet, it returns the current task execution parameters to the main node. Here, the task execution parameters include execution time, execution task, and execution real-time bandwidth parameters; The main control node waits for the verification packet to return data. If it times out, it considers that the execution node task has failed. At this time, it continues to connect with the execution node for tasks. If the connection is successful, the failed task will be pushed again. If the continuous connection is unsuccessful, the database and file writing status of the data storage center will be verified. If the task is determined to have failed, the failed task will be assigned to the next execution node in the cluster hostname table.

7. A multi-node parallel data migration and upgrade method according to any one of claims 1 to 3, 5 and 6, characterized in that: Step S41 specifically includes: the execution node accesses and filters the database of the lower version; after connecting to the database, obtains the database table structure, filters the column data of each entity data type, and matches the filter value rule=operator(And / Or / Not)value, Rules={rulei(And / Or)rulej} in sequence; at the same time, freezes the value of the data filter column, prohibits modification and update operations, and checks the value for null values, character abnormal values, and over-limit values; pushes the formulated screening rules and the storage path of the filtering result data to the task execution node; If the push fails, the node scheduling of the main control node obtains a new execution list.

Images