A rapid migration method and system based on customized data packages, and readable storage media.

Abstract

This application provides a rapid migration method and system based on customized data packages, as well as a readable storage medium. The rapid migration method based on customized data packages includes: extracting metadata from oil and gas exploration data using a general data acquisition module, employing the system's native API and file parsing algorithms; based on the metadata and the target operating system, formulating a migration strategy and monitoring the migration process through a control module, and generating a task execution list; dynamically adjusting the execution plan of the task execution list through a scheduling module, and generating a task execution priority list; and according to the task execution priority list, encapsulating and transmitting the data to the target system through a migration module, and unpacking the data according to the mapping index file and the unpacking priority list. Through the technical solution of this application, seamless data synchronization across operating systems can be achieved through customized data packaging and index mapping strategies, ensuring that the migrated data is immediately available and improving the execution efficiency of seismic exploration projects in multi-platform collaborative scenarios.

Description

Technical Field

[0001] This application relates to the field of data management technology, and more specifically, to a fast migration method and system based on a data customization package, and a readable storage medium. Background Technology

[0002] With the widespread application of seismic exploration technology, project collaboration and data sharing are increasingly crossing different operating system platforms. Traditional data migration methods involving copying are inefficient when dealing with massive and complex seismic exploration data. On the one hand, due to a lack of refined data customization, the migration process is often haphazard and disorganized, failing to accurately adapt to different software environments and business needs. On the other hand, the failure to fully utilize efficient indexing and package management mechanisms leads to slow and error-prone data transmission. After migrating project data from Linux systems, it is difficult to quickly integrate it into the seismic exploration software ecosystem under Windows systems, greatly hindering the collaborative advancement and efficient operation of projects.

[0003] Currently, mainstream seismic exploration processing, interpretation, and reservoir software organizes and structures project data in ways closely related to their specific business operations. Oil and gas exploration project data is characterized by its diversity, multi-dimensionality, and multiple versions. Overall project data types include dozens of data types such as pre-stack and post-stack seismic data, well data, oil, gas, and water data, stratigraphic data, fault data, and layer data. The data management structure includes multi-level and node-based data organization methods such as project, work area, survey line, system management, permissions, standard codes, and various data type extensions.

[0004] In particular, some oil and gas exploration data are deeply bound to databases, and the database table structure cannot be deployed and put online in a timely manner due to differences in the installation, deployment and data types of database servers on different operating systems, which prevents rapid productivity. Summary of the Invention

[0005] This application aims to solve or improve the aforementioned technical problems.

[0006] Therefore, the primary objective of this application is to provide a rapid migration method based on a data customization package.

[0007] The second objective of this application is to provide a rapid migration system based on a data customization package.

[0008] The third objective of this application is to provide a rapid migration system based on a data customization package.

[0009] The fourth objective of this application is to provide a readable storage medium.

[0010] To achieve the primary objective of this application, the technical solution of the first aspect of this application provides a rapid migration method based on customized data packages, comprising: extracting metadata of oil and gas exploration data using a general data acquisition module based on oil and gas exploration data configuration information, employing the system's native API and file parsing algorithm, and generating a mapping index file; the metadata of the oil and gas exploration data includes file path, data size, and hash value; based on the metadata and the target operating system, formulating a migration strategy and monitoring the migration process through a control module, and generating a task execution list; based on the cluster's system resources, dynamically adjusting the execution scheme of the task execution list through a scheduling module, and sorting the migration tasks according to preset priorities in descending order, generating a task execution priority list; according to the task execution priority list, encapsulating the data into customized packages through a migration module, transmitting the customized packages to the target system through multi-node parallel or asynchronous I / O technology, and unpacking the data according to the mapping index file and the unpacking priority list.

[0011] According to the rapid migration method based on customized data packages provided in this application, firstly, the general data acquisition module extracts the metadata of the oil and gas exploration data based on the configuration information of the oil and gas exploration data, using the system's native API (Application Programming Interface) and file parsing algorithms, and generates a mapping index file. The metadata of the oil and gas exploration data includes file path, data size, and hash value. Then, based on the metadata and the target operating system, the control module formulates a migration strategy and monitors the migration process, generating a task execution list. Next, based on the cluster's system resources, the scheduling module dynamically adjusts the execution plan of the task execution list and sorts the migration tasks according to preset priorities, generating a task execution priority list. Finally, according to the task execution priority list, the migration module encapsulates the data into customized packages, transmits the customized packages to the target system through multi-node parallel or asynchronous I / O technology, and unpacks the data according to the mapping index file and the unpacking priority list, restoring the data to the correct location. This ensures that the data can be successfully recognized and used by the target seismic exploration software under the Windows system, achieving seamless integration. Through customized data packaging and index mapping strategies, various types of data closely associated with seismic exploration software under Linux systems can be quickly, accurately, and reliably migrated to Windows systems, achieving seamless data synchronization across operating systems. This ensures that the migrated data is immediately available, significantly improving the execution efficiency of seismic exploration projects in multi-platform collaborative scenarios, and significantly enhancing the efficiency and quality of seismic exploration data processing and management, thereby promoting the progress and development of seismic exploration technology.

[0012] In some technical solutions, the fast migration method based on data customization packages may optionally include: project initialization, data preprocessing, and verification checks through task assistance modules.

[0013] In this technical solution, the rapid migration method based on data customization packages also includes project initialization, data preprocessing, and verification checks through a task assistance module, thereby improving data quality and migration accuracy.

[0014] In some technical solutions, optionally, project initialization, data preprocessing, and verification checks can be performed through a task assistance module, including: preliminary cleaning of the raw data from the data source to remove outliers; preparation of auxiliary materials according to data type and task requirements, including permission configuration files and software configuration parameter files; and comparison of data capacity before and after migration through hash algorithm verification and total data volume statistics to obtain verification results.

[0015] In this technical solution, a task-assistance module is used for project initialization, data preprocessing, and verification checks. Specifically, in the data preprocessing stage, the task-assistance module performs preliminary cleaning of the raw data from the data source, removing outliers to ensure the data meets basic migration quality standards. In the packaging assistance stage, the task-assistance module works with the migration module to prepare various auxiliary materials, such as permission configuration files and software configuration parameter files, based on data type and task requirements, ensuring the integrity of the packaged content. In the verification check stage, methods such as hash algorithm verification and total data volume statistics are used to compare the data volume before and after migration, thereby improving data quality and migration accuracy. In some technical solutions, optionally, based on metadata and the target operating system, a migration strategy is formulated and the migration process is monitored through a control module to generate a task execution list. This includes: receiving migration instructions through the control module, formulating a migration strategy based on the target and source project information and the target operating system in the migration instructions, refining the sub-tasks of the current project according to preset rules, and formulating a task execution list.

[0016] In this technical solution, based on metadata and the target operating system, a control module formulates migration strategies and monitors the migration process, generating a task execution list. Specifically, the control module receives migration instructions, formulates migration strategies based on the target and source project information and the target operating system in the migration instructions, refines the sub-tasks of the current project according to preset rules, and creates a task execution list. Specifically, the control module is responsible for coordinating the calling order and UI interaction instructions between various modules. It receives migration instructions from the master controller and quickly formulates detailed migration strategies based on the target and source project information (project name, version number, etc.) and the target operating system in the instructions. For example, for a large and complex oil and gas exploration and interpretation project, if it needs to be migrated to a newly deployed interpretation software environment under a Windows system, it will refine the sub-tasks of the current project, create a task execution list, assign tasks to modules, allocate resources to the scheduling module, and determine the execution order of migration modules according to pre-set rules (considering factors such as data volume, network bandwidth, and target software compatibility).

[0017] In some technical solutions, optionally, a migration strategy and migration process are formulated through a control module, including: real-time monitoring of the status of the entire migration process through the control module; collecting data acquisition progress, packaging completion rate, and data transmission rate through a network TCP communication link established with the general data acquisition module, scheduling module, migration module, and task assistance module; and generating a fault-tolerant task list when abnormal situations occur, including data acquisition lag, packaging verification errors, or transmission interruptions.

[0018] In this technical solution, a control module is used to formulate migration strategies and monitor the migration process. Specifically, the control module monitors the status of the entire migration process in real time. Through a network TCP (Transmission Control Protocol) communication link established with the general data acquisition module, scheduling module, migration module, and task assistance module, it collects data acquisition progress, packaging completion rate, and data transmission rate. In case of abnormal situations, it generates a fault-tolerant task list, which enables breakpoint resumption of the migration work and ensures the integrity and availability of project data.

[0019] In some technical solutions, optionally, based on the system resources of the cluster, the execution scheme of the task execution list is dynamically adjusted through the scheduling module, and the migration tasks are sorted in descending order according to preset priorities to generate a task execution priority list. This includes: dynamically adjusting the execution scheme of the task execution list through SSH commands based on the current hardware resource status of the system and the load of each module. The hardware resource status includes one or a combination of the following: number of CPU cores, memory capacity, disk read / write speed, and network bandwidth; managing the task queue through the scheduling module, first initializing the database of the target cluster, then initializing the software project, continuing to inject project, work area, and survey line management node information, and then writing various types of data through multiple processes.

[0020] In this technical solution, based on the system resources of the cluster, the execution plan of the task execution list is dynamically adjusted through the scheduling module. Migration tasks are sorted according to preset priorities in descending order to generate a task execution priority list. Specifically, the execution plan of the task execution list is dynamically adjusted through SSH (Secure Shell) commands based on the current hardware resource status and the load of each module. Specifically, during the data packaging stage, if a computing node (responsible for parallel packaging tasks) is found to be stuck in memory, the task is withdrawn and reassigned to other computing nodes. Simultaneously, the task queue is managed by the scheduling module. First, the database of the target cluster is initialized, then the software project is initialized, followed by the injection of project, work area, and survey line management node information, and finally, multi-process writing of various types of data.

[0021] In some technical solutions, optionally, the data is packaged into a custom package through a migration module, including: for database table-related data, the migration module generates an SQL package containing complete database table structure creation statements and data insertion statements.

[0022] In this technical solution, the migration module encapsulates the data into a customized package, including database table-related data. The migration module generates a package containing complete database table structure creation statements and data insertion statements in SQL (Structured Query Language), ensuring that the database can be accurately restored under the target Windows system.

[0023] To achieve the second objective of this application, the technical solution of the second aspect of this application provides a rapid migration system based on customized data packages, comprising: an extraction module, used to extract metadata of oil and gas exploration data through a general data acquisition module, based on oil and gas exploration data configuration information, using the system's native API and file parsing algorithm, and generate a mapping index file, wherein the metadata of the oil and gas exploration data includes file path, data size, and hash value; a first generation module, used to generate a task execution list based on metadata and the target operating system, through a control module to formulate a migration strategy and monitor the migration process; a second generation module, used to dynamically adjust the execution scheme of the task execution list through a scheduling module based on the system resources of the cluster, and sort the migration tasks according to preset priorities in descending order, generating a task execution priority list; and a migration module, used to encapsulate the data into customized packages according to the task execution priority list, transmit the customized packages to the target system through multi-node parallel or asynchronous I / O technology, and unpack the data according to the mapping index file and the unpacking priority list.

[0024] The rapid migration system based on customized data packages provided in this application includes an extraction module, a first generation module, a second generation module, and a migration module. The extraction module uses a general data acquisition module, based on oil and gas exploration data configuration information, and employs the system's native API and file parsing algorithms to extract metadata from the oil and gas exploration data and generate a mapping index file. The metadata of the oil and gas exploration data includes file paths, data sizes, and hash values. The first generation module, based on the metadata and the target operating system, uses a control module to formulate migration strategies and monitor the migration process, generating a task execution list. The second generation module, based on the cluster's system resources, uses a scheduling module to dynamically adjust the execution scheme of the task execution list and sorts the migration tasks according to preset priorities, generating a task execution priority list. The migration module, based on the task execution priority list, encapsulates the data into customized packages, transmits the customized packages to the target system using multi-node parallel or asynchronous I / O technology, and unpacks the data according to the mapping index file and the unpacking priority list. Through customized data packaging and index mapping strategies, various types of data closely associated with seismic exploration software under Linux systems can be quickly, accurately, and reliably migrated to Windows systems, achieving seamless data synchronization across operating systems. This ensures that the migrated data is immediately available, significantly improving the execution efficiency of seismic exploration projects in multi-platform collaborative scenarios, and significantly enhancing the efficiency and quality of seismic exploration data processing and management, thereby promoting the progress and development of seismic exploration technology.

[0025] To achieve the third objective of this application, the technical solution of the third aspect of this application provides a fast migration system based on a data customization package, including: a memory and a processor, wherein the memory stores a program or instructions that can be run on the processor, and when the processor executes the program or instructions, it implements the fast migration method based on a data customization package of any one of the technical solutions of the first aspect, and thus has the technical effects of any one of the technical solutions of the first aspect, which will not be repeated here.

[0026] To achieve the fourth objective of this application, the technical solution of the fourth aspect of this application provides a readable storage medium storing a program or instructions thereon. When the program or instructions are executed by a processor, they implement the steps of the fast migration method based on data customization packages of any one of the technical solutions of the first aspect, and thus have the technical effects of any one of the technical solutions of the first aspect, which will not be repeated here.

[0027] Additional aspects and advantages of this application will become apparent in the following description or may be learned by practice of this application. Attached Figure Description

[0028] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart illustrating the steps of a fast migration method based on a data customization package according to an embodiment of this application; Figure 2 This is a flowchart illustrating the steps of a fast migration method based on a data customization package according to an embodiment of this application; Figure 3 This is a flowchart illustrating the steps of a fast migration method based on a data customization package according to an embodiment of this application; Figure 4 This is a flowchart illustrating the steps of a fast migration method based on a data customization package according to an embodiment of this application; Figure 5 This is a flowchart illustrating the steps of a fast migration method based on a data customization package according to an embodiment of this application; Figure 6 This is a flowchart illustrating the steps of a fast migration method based on a data customization package according to an embodiment of this application; Figure 7 This is a flowchart illustrating the steps of a fast migration method based on a data customization package according to an embodiment of this application; Figure 8 This is a schematic block diagram of a fast migration system based on a data customization package according to an embodiment of this application; Figure 9 This is a schematic block diagram of a rapid migration system based on a data customization package, according to another embodiment of this application. Figure 10 This is an overall structural diagram of a rapid migration system based on a data customization package according to an embodiment of this application; Figure 11 This is a sequence diagram showing the overall process execution of a fast migration method based on a data customization package according to an embodiment of this application. Figure 12 This is a schematic diagram of a simulated basin-level data migration for a fast migration method based on a data customization package, according to an embodiment of this application.

[0029] in, Figure 8 and Figure 9 The correspondence between the reference numerals and component names in the attached drawings is as follows: 10: Rapid migration system based on data customization package; 110: Extraction module; 120: First generation module; 130: Second generation module; 140: Migration module; 20: Rapid migration system based on data customization package; 300: Memory; 400: Processor. Detailed Implementation

[0030] To better understand the above-mentioned objectives, features, and advantages of this application, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0031] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Therefore, the scope of protection of this application is not limited to the specific embodiments disclosed below.

[0032] The following reference Figures 1 to 12 This application describes a fast migration method and system based on a data customization package, as well as a readable storage medium, according to some embodiments of this application.

[0033] like Figure 1 As shown, an embodiment of the first aspect of this application provides a rapid migration method based on a customized data package, comprising the following steps: Step S102: Using the general data acquisition module, based on the oil and gas exploration data configuration information, the system's native API and file parsing algorithm are used to extract the metadata of the oil and gas exploration data and generate a mapping index file. The metadata of the oil and gas exploration data includes file path, data size, and hash value. Step S104: Based on metadata and the target operating system, formulate migration strategies and monitor the migration process through the control module, and generate a task execution list; Step S106: Based on the system resources of the cluster, the execution plan of the task execution list is dynamically adjusted through the scheduling module, and the migration tasks are sorted in descending order according to the preset priority to generate a task execution priority list; Step S108: Based on the task execution priority list, the data is encapsulated into a custom package by the migration module, and the custom package is transmitted to the target system through multi-node parallel or asynchronous I / O technology. The data is then unpacked according to the mapping index file and the unpacking priority list.

[0034] According to the rapid migration method based on customized data packages provided in this embodiment, firstly, the general data acquisition module extracts the metadata of the oil and gas exploration data based on the configuration information of the oil and gas exploration data, using the system's native API and file parsing algorithm, and generates a mapping index file. The metadata of the oil and gas exploration data includes file path, data size, and hash value. Then, based on the metadata and the target operating system, the control module formulates a migration strategy and monitors the migration process, generating a task execution list. Next, based on the system resources of the cluster, the scheduling module dynamically adjusts the execution plan of the task execution list and sorts the migration tasks according to preset priorities, generating a task execution priority list. Finally, according to the task execution priority list, the migration module encapsulates the data into customized packages, transmits the customized packages to the target system through multi-node parallel or asynchronous I / O technology, and unpacks the data according to the mapping index file and the unpacking priority list, restoring the data to the correct location. This ensures that the data can be successfully recognized and used by the target seismic exploration software under the Windows system, achieving seamless integration. Through customized data packaging and index mapping strategies, various types of data closely associated with seismic exploration software under Linux systems can be quickly, accurately, and reliably migrated to Windows systems, achieving seamless data synchronization across operating systems. This ensures that the migrated data is immediately available, significantly improving the execution efficiency of seismic exploration projects in multi-platform collaborative scenarios, and significantly enhancing the efficiency and quality of seismic exploration data processing and management, thereby promoting the progress and development of seismic exploration technology.

[0035] like Figure 2 As shown, the fast migration method based on a data customization package according to an embodiment of this application further includes the following steps: Step S202: Perform project initialization, data preprocessing, and verification checks through the task assistance module.

[0036] In this embodiment, the rapid migration method based on the data customization package also includes project initialization, data preprocessing, and verification checks through a task assistance module, thereby improving data quality and migration accuracy.

[0037] like Figure 3As shown, according to an embodiment of the fast migration method based on a data customization package proposed in this application, the project initialization, data preprocessing, and verification checks are performed through a task assistance module, specifically including the following steps: Step S302: Perform preliminary cleaning on the raw data from the data source to remove outliers; Step S304: Prepare auxiliary materials according to the data type and task requirements. Auxiliary materials include permission configuration files and software configuration parameter files. Step S305: Compare the data capacity before and after the migration by verifying with a hash algorithm and by statistically analyzing the total amount of data to obtain the verification results.

[0038] In this embodiment, a task assistance module is used for project initialization, data preprocessing, and verification checks. Specifically, in the data preprocessing stage, the task assistance module performs preliminary cleaning of the raw data from the data source, removing outliers to ensure the data meets basic migration quality standards. In the packaging assistance stage, the task assistance module works with the migration module to prepare various auxiliary materials, such as permission configuration files and software configuration parameter files, based on data type and task requirements, ensuring the integrity of the packaged content. In the verification check stage, hash algorithm verification and total data volume statistics are used to compare the data volume before and after migration.

[0039] like Figure 4 As shown, according to an embodiment of the fast migration method based on a data customization package proposed in this application, based on metadata and the target operating system, a migration strategy is formulated and the migration process is monitored through a control module, generating a task execution list, specifically including the following steps: Step S402: Receive migration instructions through the control module, formulate migration strategies based on the target and source project information and target operating system in the migration instructions, refine the sub-tasks of the current project according to preset rules, and formulate a task execution list.

[0040] In this embodiment, based on metadata and the target operating system, a migration strategy is formulated and the migration process is monitored by a control module, generating a task execution list. Specifically, the control module receives migration instructions, formulates a migration strategy based on the target and source project information and the target operating system in the migration instructions, refines the sub-tasks of the current project according to preset rules, and creates a task execution list. Specifically, the control module is responsible for coordinating the calling order and UI interaction instructions between various modules. It receives migration instructions from the master controller and quickly formulates a detailed migration strategy based on the target and source project information (project name, version number, etc.) and the target operating system in the instructions. For example, for a large and complex oil and gas exploration and interpretation project, if it needs to be migrated to a newly deployed interpretation software environment under a Windows system, it will refine the sub-tasks of the current project according to preset rules (considering factors such as data volume, network bandwidth, and target software compatibility), create a task execution list, assign tasks to modules, allocate resources to the scheduling module, and determine the execution order of migration modules.

[0041] like Figure 5 As shown, according to an embodiment of the fast migration method based on a customized data package proposed in this application, a migration strategy is formulated and the migration process is monitored through a control module, specifically including the following steps: Step S502: The control module monitors the status of the entire migration process in real time. Through the network TCP communication link established with the general data acquisition module, scheduling module, migration module, and task assistance module, it collects data acquisition progress, packaging completion rate, and data transmission rate. In case of abnormal situations, it generates a fault-tolerant task list. Abnormal situations include data acquisition lag, packaging verification errors, or transmission interruption.

[0042] In this embodiment, the control module formulates migration strategies and monitors the migration process. Specifically, the control module monitors the status of the entire migration process in real time. Through the network TCP communication link established with the general data acquisition module, scheduling module, migration module, and task assistance module, it collects data acquisition progress, packaging completion rate, and data transmission rate. In case of abnormal situations, it generates a fault-tolerant task list, which can resume the migration work from breakpoints and ensure the integrity and availability of project data.

[0043] like Figure 6 As shown, according to an embodiment of the fast migration method based on a customized data package proposed in this application, based on the system resources of the cluster, the execution scheme of the task execution list is dynamically adjusted by the scheduling module, and the migration tasks are sorted in descending order according to preset priorities to generate a task execution priority list. Specifically, the method includes the following steps: Step S602: Based on the current hardware resource status of the system and the load of each module, dynamically adjust the execution plan of the task execution list using SSH commands. The hardware resource status includes one or a combination of the following: number of CPU cores, memory capacity, disk read / write speed, and network bandwidth. Step S604: Manage the task queue through the scheduling module. First, initialize the database of the target cluster, then initialize the software project, continue to inject project, work area, and survey line management node information, and then write various types of data through multiple processes.

[0044] In this embodiment, based on the system resources of the cluster, the execution scheme of the task execution list is dynamically adjusted by the scheduling module, and the migration tasks are sorted in descending order according to preset priorities to generate a task execution priority list. Specifically, the execution scheme of the task execution list is dynamically adjusted through SSH commands based on the current hardware resource status of the system and the load of each module. Specifically, during the data packaging stage, if a memory deadlock is found on a computing node (responsible for parallel packaging tasks), the task is withdrawn and reassigned to other computing nodes. At the same time, the task queue is managed by the scheduling module. First, the database of the target cluster is initialized, then the software project is initialized, and then the project, work area, and survey line management node information is injected. Finally, various types of data are written in multiple processes.

[0045] like Figure 7 As shown, according to an embodiment of the fast migration method based on a customized data package proposed in this application, the migration module encapsulates the data into a customized package, specifically including the following steps: Step S702: For the database table-related data, generate an SQL package containing complete database table structure creation statements and data insertion statements through the migration module.

[0046] In this embodiment, the migration module encapsulates the data into a customized package, including database table-related data. The migration module generates an SQL package containing complete database table structure creation statements and data insertion statements to ensure accurate database restoration under the target Windows system.

[0047] like Figure 8As shown, an embodiment of the second aspect of this application provides a fast migration system 10 based on a customized data package, comprising: an extraction module 110, used to extract metadata of oil and gas exploration data through a general data acquisition module, based on oil and gas exploration data configuration information, using the system's native API and file parsing algorithm, and generate a mapping index file, wherein the metadata of the oil and gas exploration data includes file path, data size, and hash value; a first generation module 120, used to generate a task execution list based on metadata and the target operating system, through a control module to formulate a migration strategy and monitor the migration process; a second generation module 130, used to dynamically adjust the execution scheme of the task execution list through a scheduling module based on the system resources of the cluster, and sort the migration tasks according to a preset priority in descending order, generating a task execution priority list; and a migration module 140, used to encapsulate the data into a customized package through the migration module according to the task execution priority list, transmit the customized package to the target system through multi-node parallel or asynchronous I / O technology, and unpack the data according to the mapping index file and the unpacking priority list.

[0048] The rapid migration system 10 based on customized data packages provided in this embodiment includes an extraction module 110, a first generation module 120, a second generation module 130, and a migration module 140. The extraction module 110 uses a general data acquisition module, based on oil and gas exploration data configuration information, and employs the system's native API and file parsing algorithms to extract metadata from the oil and gas exploration data and generate a mapping index file. The metadata includes file path, data size, and hash value. The first generation module 120, based on the metadata and the target operating system, uses a control module to formulate a migration strategy and monitor the migration process, generating a task execution list. The second generation module 130, based on the cluster's system resources, uses a scheduling module to dynamically adjust the execution scheme of the task execution list and sorts the migration tasks according to preset priorities, generating a task execution priority list. The migration module 140, based on the task execution priority list, encapsulates the data into customized packages using the migration module, transmits the customized packages to the target system via multi-node parallel or asynchronous I / O technology, and unpacks the data according to the mapping index file and the unpacking priority list. Through customized data packaging and index mapping strategies, various types of data closely associated with seismic exploration software under Linux systems can be quickly, accurately, and reliably migrated to Windows systems, achieving seamless data synchronization across operating systems. This ensures that the migrated data is immediately available, significantly improving the execution efficiency of seismic exploration projects in multi-platform collaborative scenarios, and significantly enhancing the efficiency and quality of seismic exploration data processing and management, thereby promoting the progress and development of seismic exploration technology.

[0049] like Figure 9As shown, an embodiment of the third aspect of this application provides a fast migration system 20 based on a data customization package, including a memory 300 and a processor 400. The memory 300 stores a program or instructions that can be executed on the processor 400. When the processor 400 executes the program or instructions, it implements the steps of the fast migration method based on a data customization package according to any one of the embodiments of the first aspect, and thus has the technical effects of any embodiment of the first aspect, which will not be repeated here.

[0050] An embodiment of the fourth aspect of this application provides a readable storage medium having a program or instructions stored thereon. When the program or instructions are executed by a processor, they implement the steps of the fast migration method based on a data customization package according to any one of the embodiments of the first aspect, and thus have the technical effects of any of the embodiments of the first aspect described above, which will not be repeated here.

[0051] like Figure 10 , Figure 11 and Figure 12 As shown, a rapid migration method based on a customized data package, according to a specific embodiment of this application, aims to solve the problem of seamless data synchronization across operating systems, ensuring that the migrated data is immediately available. It quickly and efficiently migrates various types of data closely associated with seismic exploration software from a Linux system to a Windows system, achieving seamless data synchronization across operating systems through customized data packaging and index mapping strategies. This ensures that the migrated data is immediately available, significantly improving the execution efficiency of seismic exploration projects in multi-platform collaborative scenarios.

[0052] Its core modules consist of four execution control modules: a control module, a scheduling module, a migration module, and a task module, as well as a cross-operating system universal data acquisition module, totaling five parts. The overall system architecture diagram is shown below. Figure 10 As shown. The functions of each module are as follows: (a) General data acquisition module: The general-purpose data acquisition module is a universal module compatible with data access APIs for both Windows and Linux systems, adaptable to both Linux and Windows environments. It employs native system APIs and a customized file parsing algorithm for oil and gas exploration data. Based on oil and gas exploration data configuration information, it quickly searches file paths and locates data size, hash, and other information. It includes built-in data format parsing and metadata extraction functions, responsible for reading and writing the underlying data.

[0053] (II) Control Module: The control module is responsible for coordinating the calling order and UI interaction commands between various modules. It receives migration commands from the master controller and, based on the target and source project information (project name, version number, etc.) and the target operating system, quickly formulates a detailed migration strategy. For example, for a large and complex oil and gas exploration and interpretation project, if it needs to be migrated to a newly deployed interpretation software environment under a Windows system, it will, according to pre-set rules (considering factors such as data volume, network bandwidth, and target software compatibility), refine the sub-tasks of the current project, create a task execution list, assign tasks to modules, allocate resources to the scheduling module, and determine the execution order of the migration modules.

[0054] Simultaneously, it monitors the status of the entire migration process in real time, collecting metrics such as data acquisition progress, packaging completion rate, and data transmission rate through network TCP communication links established with each module. Upon detecting any anomalies (such as data acquisition lag, packaging verification errors, or transmission interruptions), it provides error logs, generates a fault-tolerant task list, and enables breakpoint resumption of the migration process, ensuring the integrity and availability of project data.

[0055] (III) Scheduling Module: The scheduling module optimizes the allocation of migration task lists based on the cluster's system resources, ensuring efficient execution of migration tasks. It dynamically adjusts the execution plan of the task list via SSH commands based on the current system hardware resources (including CPU core count, memory capacity, disk read / write speed, network bandwidth, etc.) and the load of each task module. For example, during the data packaging phase, if a compute node (responsible for parallel packaging tasks) experiences a memory freeze, the task is cancelled and reassigned to another compute node. Meanwhile, the scheduling module manages the task queue, sorting migration tasks according to priority and processing high-priority tasks first. For example, the target migration task first initializes the target cluster's database, then initializes the software project, injects project, work area, and survey line management node information, and then performs multi-process writing of various types of data. Priority must be followed in descending order; otherwise, subsequent steps cannot proceed.

[0056] (iv) Migration Module: The migration module has a built-in migration baseline, responsible for deploying and executing data migration tasks. Based on the nodes and tasks specified by the scheduling module, it specifically performs data packaging, transmission, and unpacking / recovery operations. During packaging, it calls a general data acquisition tool and, combined with index information from the project database, encapsulates oil and gas exploration data into efficient data packets according to type. For database table-related data, it generates SQL packages containing complete table structure creation statements and data insertion statements, ensuring accurate database restoration on the target Windows system. During transmission, the migration module employs multi-node parallel or multi-threaded, asynchronous I / O technology to improve cross-system data read / write efficiency. Upon arrival at the target end, based on the pre-built index mapping relationship, it quickly unpacks and restores the data to the correct location, ensuring that the data can be successfully recognized and used by the target seismic exploration software on the Windows system, achieving seamless integration.

[0057] (v) Task Assistance Module: The task assistance module has a built-in task tree that assists with specific tasks on the terminal, including project initialization, data preprocessing, and verification checks. During the data preprocessing stage, the raw data from the data source is initially cleaned to remove outliers, ensuring the data meets basic transferable quality standards.

[0058] During the packaging assistance phase, the task assistance module works with the migration module to prepare various auxiliary materials, such as permission configuration files and software configuration parameter files, based on data type and task requirements, to ensure the integrity of the packaged content. In the verification and checking phase, methods such as hash algorithm verification and total data volume statistics are used to compare the data volume before and after the migration.

[0059] The specific implementation steps are shown in the timing diagram below. Figure 11 The detailed execution steps are as follows: 1. Start the migration master process. The master control module implements specific functions and is responsible for controlling the overall data migration process. Obtain the project information of the data migration source specified by the UI (project data version number, project path, project coordinate projection ellipsoid parameters, customized data migration type), the node list of the migration cluster, the system of the migration target, and the storage path of the data migration package (select cross-system shared storage).

[0060] 1.1 Based on the cluster node list obtained in section 1, begin pre-start checks on the project data storage and computing hardware environment. Ping checks network connectivity and bandwidth, while SSH checks node trust. Ping is a network diagnostic tool primarily used to test the connection between a device and another device on the network.

[0061] 1.2 Based on the project information obtained in section 1, access the database table contents of the source database, formulate and refine the sub-tasks of the current project, formulate the task execution list, and issue the command to start the migration task execution.

[0062] 2. Start the scheduling process. According to the instructions of the main control process in step 1, the scheduling node obtains the execution order of the subtasks.

[0063] 2.1 The data acquisition process is initiated to extract metadata from the storage paths and database schema of the oil and gas exploration project. File paths are quickly searched to locate the number and size of data items, and a hash algorithm is applied to the located files. Relevant metadata and located file information are recorded in a configuration file. A source-target mapping index is created based on the metadata, hash values, and file path storage relationships. The `hash` attribute stores the file hash value; the hash algorithm is selected as needed, such as MD5, SHA-1, SHA-256, etc., to ensure consistent file content identification. The `c` attribute describes the database column structure, constraints, and correspondences, distinguished by the `name` field. For example, the column structure is as follows: <metadata name="column_structure"> {"name": "id", "type": "integer", "nullable": false, "primary_key": true}, {"name": "name", "type": "varchar", "nullable": false}, {"name": "age", "type": "integer", "nullable": true}< / metadata> Embedded in JSON format text <metadata>Elements record column structure information of related database tables, including key attributes such as column name, data type, nullability, and primary key, providing in-depth support for the data table logic involved in exploration data files. External constraints, such as... <metadata name="foreign_key_constraints"> [{"source_column": "department_id", "target_table":"departments", "target_column": "id"}]< / metadata> Also using JSON format, this records the external constraints in the related tables, specifying the relationships between source and target columns, and between target columns, thus aiding in database relationship maintenance and data integrity assurance. Permission-related content includes: <metadata name="permissions"> {"read": "user1,user2", "write": "admin"}< / metadata> This uses JSON format to indicate the permissions that different users or user groups have for the database resource corresponding to that file, clearly distinguishing between read and write permissions. Management information includes: <metadata name="management_info"> {"last_backup_time": "2023 - 01 - 01T12:00:00", "backup_frequency": "weekly", "responsible_person": "John Doe"}< / metadata> Record database management details related to files in JSON format, including the last backup time, backup frequency, and responsible person, to ensure the traceability of database operation and maintenance.

[0064] 2.2 Based on the configuration file, prioritize the located data. Using a rapid migration scheme based on the migration baseline of multiple projects, work areas, survey lines, and data objects, traverse the management node information of the metadata. Prioritize the data according to the following order: database, project, work area, survey line, data entity object, data entity sub-object, data entity file object, data entity file sub-object, permission object, and configuration object, forming a priority list for packaged tasks. Write this list to the index mapping file in section 2.1.

[0065] 2.3 Start the packaging process, access the task list file in 2.2, and package the data according to the tasks from highest to lowest priority. Simultaneously, feed back information such as the start, completion, progress, and pause status of the packaging tasks to the main control process in 1. Write the customized data package to the storage path specified in 1.

[0066] 3. Start the main control process for data migration on the Windows side, and obtain the project information of the target data migration specified by the UI (project data version number, project path, project projection ellipsoid parameters, type of customized data migration), the system of the target migration, and the storage path of the customized data package.

[0067] 3.1 Start the deployment process Based on the project information in section 3, create an empty project database under the Windows system, set the project's projection ellipsoid parameters, access the index mapping file in section 2.1, obtain the paths of relevant data packages in the customized package, database table structures, and other information, and then map the source-target index... <metadata>The element records the database table structure, constraints, etc., and the initial project database table structure. Based on the UI storage path, configure the project's file storage path.

[0068] 3.2 Start the unpacking process Based on the task execution priority list in section 2.2, the data unpacking process begins. For database table-related data, a SQL package containing complete table structure creation statements and data insertion statements is generated to ensure accurate database restoration on the target Windows system. For file data, the data acquisition module is invoked, using the native Windows file API to write the data to the project storage path on the Windows system. Simultaneously, information regarding the unpacking task's start, completion, progress, and pause status is fed back to the main control process in section 3.

[0069] 3.3 Start the verification process The verification process is initiated based on the mapping index file. It checks the quantity, size, and hash value of data in the database. If inconsistencies are found, the relevant mapping information and task information of the inconsistent data records are fed back to the master controller of 3. 3 then submits the information to the master controller of 1.

[0070] 4. If the master controller detects migration tasks that failed validation, it checks the read / write permissions of the current data and the management permission values ​​in the database, modifies the status of the failed migration tasks in the task list to "not executed," and submits them to section 2.3 for repackaging. Migration data visualization is as follows: Figure 12 .

[0071] The GeoEast system, through the seismic exploration database construction method and apparatus provided in this application, can achieve efficient, flexible and reliable database construction.

[0072] In summary, the beneficial effects of the embodiments of this application are as follows: 1. Improve database building efficiency: By rapidly migrating and building data projects in batches, build time is significantly reduced, and work efficiency is improved.

[0073] 2. Simplify and optimize migration steps: By monitoring the data metrics of the task, data consistency can be confirmed.

[0074] 3. Through customized data packaging and index mapping strategies, various types of data closely associated with seismic exploration software under the Linux system can be quickly, accurately, and reliably migrated to the Windows system, achieving seamless data synchronization across operating systems. This ensures that the migrated data is immediately available, significantly improving the execution efficiency of seismic exploration projects in multi-platform collaborative scenarios, and significantly enhancing the efficiency and quality of seismic exploration data processing and management, thereby promoting the progress and development of seismic exploration technology.

[0075] In this application, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance; the term "multiple" refers to two or more unless otherwise expressly defined. The terms "install," "connect," "link," and "fix" should be interpreted broadly. For example, "connect" can be a fixed connection, a detachable connection, or an integral connection; "link" can be a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0076] In the description of this application, it should be understood that the terms "upper," "lower," "front," "rear," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or module referred to must have a specific orientation or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0077] In the description of this specification, the terms "one embodiment," "some embodiments," "specific embodiment," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0078] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.< / metadata> < / metadata>

Claims

1. A rapid migration method based on a customized data package, characterized in that, include: Using the general data acquisition module, based on the configuration information of oil and gas exploration data, and employing the system's native API and file parsing algorithm, the metadata of the oil and gas exploration data is extracted and a mapping index file is generated. The metadata of the oil and gas exploration data includes file path, data size, and hash value. Based on the metadata and the target operating system, the control module formulates migration strategies and monitors the migration process, generating a task execution list. Based on the system resources of the cluster, the execution scheme of the task execution list is dynamically adjusted by the scheduling module, and the migration tasks are sorted in descending order according to the preset priority to generate a task execution priority list. According to the task execution priority list, the data is encapsulated into a custom package by the migration module, and the custom package is transmitted to the target system through multi-node parallel or asynchronous I / O technology. The data is then unpacked according to the mapping index file and the unpacking priority list.

2. The rapid migration method based on data customization packages according to claim 1, characterized in that, Also includes: The task assistance module performs project initialization, data preprocessing, and verification checks.

3. The rapid migration method based on data customization packages according to claim 2, characterized in that, The process of initializing the project, preprocessing the data, and performing verification checks through the task assistance module includes: Perform preliminary cleaning on the raw data source to remove outliers; Prepare auxiliary materials according to the data type and task requirements. The auxiliary materials include permission configuration files and software configuration parameter files. The data capacity before and after the migration was compared by verifying the hash algorithm and statistically analyzing the total data volume to obtain the verification results.

4. The rapid migration method based on data customization packages according to claim 1, characterized in that, Based on the metadata and the target operating system, the control module formulates a migration strategy and monitors the migration process, generating a task execution list, including: The control module receives migration instructions, formulates migration strategies based on the target and source project information and the target operating system in the migration instructions, refines the sub-tasks of the current project according to preset rules, and formulates a task execution list.

5. The rapid migration method based on data customization packages according to claim 2, characterized in that, The process of formulating migration strategies and monitoring the migration process through the control module includes: The control module monitors the status of the entire migration process in real time. Through the network TCP communication link established with the general data acquisition module, the scheduling module, the migration module, and the task assistance module, it collects data acquisition progress, packaging completion rate, and data transmission rate. In case of abnormal situations, it generates a fault-tolerant task list, including data acquisition lag, packaging verification errors, or transmission interruptions.

6. The rapid migration method based on data customization packages according to any one of claims 1 to 5, characterized in that, The cluster-based system resources dynamically adjust the execution scheme of the task execution list through the scheduling module, and sort the migration tasks according to preset priorities in descending order to generate a task execution priority list, including: Based on the current hardware resource status of the system and the load of each module, the execution scheme of the task execution list is dynamically adjusted through SSH commands. The hardware resource status includes one or a combination of the following: number of CPU cores, memory capacity, disk read / write speed, and network bandwidth. The task queue is managed by the scheduling module. First, the database of the target cluster is initialized, then the project of the software is initialized, then the project, work area and survey line management node information is injected, and then the data of various types is written in multiple processes.

7. The rapid migration method based on data customization packages according to any one of claims 1 to 5, characterized in that, The process of encapsulating data into a customized package via the migration module includes: For database table-related data, the migration module generates an SQL package containing complete table structure creation statements and data insertion statements.

8. A rapid migration system based on a customized data package, characterized in that, include: The extraction module (110) is used to extract the metadata of oil and gas exploration data through the general data acquisition module, based on the configuration information of oil and gas exploration data, using the system's native API and file parsing algorithm, and generate a mapping index file. The metadata of the oil and gas exploration data includes file path, data size, and hash value. The first generation module (120) is used to generate a task execution list based on the metadata and the target operating system, by formulating a migration strategy and monitoring the migration process through the control module. The second generation module (130) is used to dynamically adjust the execution scheme of the task execution list based on the system resources of the cluster through the scheduling module, and sort the migration tasks in order of high and low priority according to the preset priority to generate a task execution priority list. The migration module (140) is used to encapsulate data into a customized package according to the task execution priority list, transmit the customized package to the target system through multi-node parallel or asynchronous I / O technology, and unpack the data according to the mapping index file and the unpacking priority list.

9. A rapid migration system based on a customized data package, characterized in that, include: A memory (300) and a processor (400), wherein the memory (300) stores a program or instructions executable on the processor (400), and the processor (400) executes the program or instructions to implement the steps of the fast migration method based on a data customization package as described in any one of claims 1 to 7.

10. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the steps of the fast migration method based on data customization packages as described in any one of claims 1 to 7.

Images