Data processing method and device and electronic equipment

Abstract

The invention discloses a data processing method and device and electronic equipment. The method comprises the steps that data information in a source database is obtained, and the data information comprises a table structure file, an index file and a constraint file; exporting the data information into a target database, and determining a data table list in a source database; grouping the data table list to obtain multiple groups of export files; exporting the plurality of groups of exported files to a target database in parallel, and determining a plurality of data scripts according to the data information; a plurality of data scripts are executed in a target database to determine a plurality of sets of exported file indices and constraints in the target database. According to the method and the device, the technical problems of high workload and low efficiency due to the fact that the content of the source database is transferred to the new database in a manual processing mode in the related technology are solved.

Description

Technical Field

[0001] This application relates to the field of databases, and more specifically, to a data processing method, apparatus, and electronic device. Background Technology

[0002] In the context of traditional database migration and synchronization, the manual transfer of content from a source database to a new database presents significant challenges and limitations. Manual processing is not only extremely labor-intensive but also inefficient, especially when dealing with massive amounts of data and complex structures. Technicians must meticulously inspect and modify the structure, indexes, and constraints of each table to adapt to the new database's environment and resource constraints—a time-consuming and error-prone process that often requires substantial effort from database administrators or experienced developers. Therefore, the manual transfer of content from a source database to a new database in these technologies suffers from both high workload and low efficiency.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a data processing method, apparatus, and electronic device to at least solve the technical problems of large workload and low efficiency in the related art of transferring the content of a source database to a new database by manual processing.

[0005] According to one aspect of the embodiments of this application, a data processing method is provided, comprising: acquiring data information from a source database, wherein the data information includes table structure files, index files, and constraint files; exporting the data information to a target database and determining a list of data tables in the source database; grouping the list of data tables to obtain multiple sets of exported files; exporting the multiple sets of exported files to the target database in parallel and determining multiple data scripts based on the data information; and executing the multiple data scripts in the target database to determine the indexes and constraints of the multiple sets of exported files in the target database.

[0006] Optionally, obtaining data information from the source database includes: determining a first statement to query the table structure file in the source database, a second statement to query the index file in the source database, and a third statement to query the constraint file in the source database; executing the first statement to determine the table structure file; executing the second statement to determine the index file; executing the third statement to determine the constraint file; and determining the data information based on the table structure file, index file, and constraint file.

[0007] Optionally, exporting data information to a target database includes: obtaining the environment parameters of the target database and determining the data conversion script based on the environment parameters; and executing the data conversion script in the target database.

[0008] Optionally, the data table list is grouped to obtain multiple sets of export files, including: determining the total amount of data in the data table list; determining the number of groups corresponding to the data table list based on the total amount of data and the preset amount of data for each group; and grouping the data table list according to the number of groups to obtain multiple sets of export files.

[0009] Optionally, the method further includes: identifying whether the source database and the target database are on the same physical machine, and obtaining the identification result; if the identification result indicates that the source database and the target database are on the same physical machine, performing data processing using the first method; if the identification result indicates that the source database and the target database are not on the same physical machine, performing data processing using the second method.

[0010] Optionally, data processing is performed using a first method, including: obtaining the maximum number of processes supported by disk input / output; processing multiple sets of exported files in parallel based on the maximum number of processes, and monitoring the number of running processes through a control script; if the number of running processes is less than the maximum number of processes, starting a new process to process the remaining unprocessed files in the multiple sets of exported files, until all multiple sets of exported files have been processed.

[0011] Optionally, a second method is used for data processing, including: obtaining the maximum number of export processes supported by the disk input / output of the physical machine where the source database is located, and obtaining the maximum number of import processes supported by the database storage input / output of the physical machine where the target database is located; exporting multiple sets of export files based on the maximum number of export processes, and importing multiple sets of export files based on the maximum number of import processes; monitoring the number of first processes performing export processing, and monitoring the number of second processes performing import processing; if the number of first processes is less than the maximum number of export processes, starting a new process to process the remaining unexported files in the multiple sets of export files until all multiple sets of export files are exported; if the number of second processes is less than the maximum number of import processes, starting a new process to process the remaining unimported files in the multiple sets of export files until all multiple sets of export files are imported.

[0012] Optionally, the method further includes: obtaining log files for performing file export operations in the source database and file import operations in the target database, wherein the log files are used to record the start time, end time, and success or failure of the operations; checking the log files to obtain the check results; and re-executing the corresponding file operations if the check results indicate that the log files do not contain a success flag.

[0013] According to another aspect of the embodiments of this application, a data processing apparatus is also provided, comprising: an acquisition module for acquiring data information from a source database, wherein the data information includes table structure files, index files, and constraint files; a determination module for exporting the data information to a target database and determining a list of data tables in the source database; a grouping module for grouping the list of data tables to obtain multiple sets of export files; an export module for exporting the multiple sets of export files to the target database in parallel and determining multiple data scripts based on the data information; and an execution module for executing the multiple data scripts in the target database to determine the indexes and constraints of the multiple sets of export files in the target database.

[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory for storing program instructions; and a processor connected to the memory for executing program instructions to perform the following functions: acquiring data information from a source database, wherein the data information includes table structure files, index files, and constraint files; exporting the data information to a target database and determining a list of data tables in the source database; grouping the list of data tables to obtain multiple sets of exported files; exporting the multiple sets of exported files to the target database in parallel and determining multiple data scripts based on the data information; and executing the multiple data scripts in the target database to determine the multiple sets of exported file indexes and constraints in the target database.

[0015] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-described data processing method by running the computer program.

[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, including computer instructions that, when executed by a processor, implement the above-described data processing method.

[0017] In this embodiment, by acquiring data information from the source database, including table structure files, index files, and constraint files; exporting the data information to the target database and determining a list of data tables in the source database; grouping the list of data tables to obtain multiple sets of exported files; exporting the multiple sets of exported files to the target database in parallel and determining multiple data scripts based on the data information; and executing the multiple data scripts in the target database to determine the indexes and constraints of the multiple sets of exported files in the target database, the efficiency of data migration is improved, thereby achieving the technical effect of reducing labor costs. This solves the technical problem in related technologies where transferring the content of the source database to a new database through manual processing involves a large workload and low efficiency. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a hardware structure block diagram of a computer terminal for implementing a data processing method according to an embodiment of this application;

[0020] Figure 2 This is a flowchart of a data processing method according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of a data segmentation process according to an embodiment of this application;

[0022] Figure 4 This is a structural diagram of a data processing apparatus according to an embodiment of this application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] The information collected in this application embodiment is information and data authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant regions, and necessary confidentiality measures have been taken. It does not violate public order and good morals, and provides corresponding operation entry points for users to choose to authorize or reject the automated decision results. If the user chooses to reject, the process will proceed to the expert decision-making process.

[0026] First, some nouns or terms that appear in the explanation of the embodiments of this application shall be interpreted as follows:

[0027] Logical database data: For many databases, block-based and log-based synchronization is physical synchronization, while logical data export and import is conditional, constrained by the current table's transaction and limited by rollback segments.

[0028] Maximum IO (Maximum Input / Output): Measured by the storage IO throughput generated per unit time. It can be easily measured by cpio concurrency testing. Generally, with a single disk throughput of 40MB / s, some arrays can reach 60-80MB / s (because the array is shared).

[0029] To address the problems existing in related technologies, embodiments of this application provide a data processing method that can be run on... Figure 1 The computer terminal shown is described below.

[0030] The data processing method embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal for implementing a data processing method is shown. Figure 1 As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions connected via wired and / or wireless networks. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1The different configurations shown.

[0031] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be implemented wholly or partially as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be wholly or partially integrated into any other element in the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0032] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the aforementioned data processing method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0033] The transmission module 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission module 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0034] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0035] It should be noted here that, in some optional embodiments, the above... Figure 1 The computer terminal shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computer terminal.

[0036] In the above operating environment, this application provides a method embodiment for data processing. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than that shown here.

[0037] Figure 2 This is a flowchart of a data processing method according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0038] Step S202: Obtain data information from the source database, including table structure files, index files, and constraint files.

[0039] In step S202 above, the table structure file contains detailed definitions of each table in the source database, including table name, field name, field type, field length, whether null values ​​are allowed, default values ​​for fields, and table storage options (such as tablespace and storage type). During the migration process, the table structure file is used to recreate the same table structure in the target database as in the source database. This ensures that after data migration, the tables in the target database have the same columns and attributes as the tables in the source database, providing the necessary framework for data import. The index file describes all index structures on the tables in the source database, including index name, index type (such as B-tree, bitmap), index fields, tablespace, and storage parameters, used for quickly locating data records. In the target database, the indexes in the source database can be rebuilt using the index file. The constraint file records all constraints on the tables in the source database in detail, including primary keys, foreign keys, uniqueness, and check constraints. The constraint file is used to reapply the same constraints in the target database as in the source database, ensuring the logical structure and integrity of the migrated data, preventing non-compliant data from entering the target database, thereby maintaining the accuracy and reliability of the data.

[0040] In some embodiments of this application, for example in an Oracle database, the expdp tool can be used to export table structure, index, and constraint information. expdp generates a file containing all necessary definitions, including DDL statements for the table structure, index creation statements, and constraint application statements. In other database systems, such as MySQL, tools like mysqldump can be used to achieve a similar purpose.

[0041] It should be noted that in the three sets of files mentioned above, for the table structure file, since the tablespace structure in the target database is different, it is unlikely that a default tablespace exists under the object owner user in practical applications, allowing tables to be freely created in the default tablespace. In fact, the tablespace of the target database is not 1:1 with that of the source database. Therefore, it is necessary to swap in the target tablespace according to the size when creating the table. This is of great significance during the initial synchronization. The embodiments of this application allow the tablespace to be changed and also support the change of the owner, which makes it very convenient to create the corresponding table structure in the target database so that subsequent data import will not report a series of errors such as table not found, tablespace not found, tablespace insufficient, etc. For the index file, it is used to create indexes on the table in batches and in a controllable parallel manner after the table data is imported. Here, it is also very convenient to change the new index creation script according to the existing tablespace in the target database. For the constraint file, some constraints depend on the index, such as the primary key. Therefore, it is necessary to execute the index file in batches and in parallel. After these three sets of files are executed, there will be tables in the target database that are the same size as those in the source database but have completely different storage space.

[0042] Step S204: Export the data information to the target database and determine the list of data tables in the source database.

[0043] In step S204 above, after exporting the data information to the target database, the corresponding structure is created in the target database using the data information. For example, tools such as SQL*Plus or SQL Developer are used to execute DDL statements to create tables, and to execute index and constraint creation statements to establish a data integrity guarantee mechanism.

[0044] Automated scripts (such as shell scripts) are commonly used for exporting and applying data. They can execute a series of SQL statements sequentially or in parallel, greatly improving efficiency and reducing human error. When exporting data, the environmental differences of the target database must be considered, including version, character set, tablespace, etc., to ensure that the data can be correctly parsed and applied in the target database. After exporting the data to the target database, a list of data tables needs to be determined. This list represents the scope of the data to be migrated.

[0045] Step S206: Group the data table list to obtain multiple sets of export files.

[0046] In step S206 above, the data table list typically contains detailed information about all tables to be migrated, such as table names and sizes. Grouping the data table list into multiple export files enables parallel data processing, avoiding performance bottlenecks when a single thread handles large amounts of data. The grouping strategy can be customized based on various factors, including table size, database I / O capabilities, and network bandwidth. In this embodiment, the focus is on the principle of parallel controllable IO maximization. This means that grouping needs to consider the I / O throughput of the storage device, allocating tables in the data table list to different groups according to size or other predefined rules, with each group having roughly the same amount of data. This allows for parallel export under simultaneous control, maximizing the utilization of storage I / O capabilities and ensuring that each export task can be completed within a similar timeframe.

[0047] Step S208: Export multiple sets of export files to the target database in parallel, and determine multiple data scripts based on the data information.

[0048] In step S208 above, a parallel export tool for the database, such as Oracle's expdp tool, can be used to start an independent export thread for each group of export files. Each thread is responsible for processing all table data in a group. Through parallel operations, storage and network resources can be fully utilized to quickly export data to the target database. After the data is migrated to the target database, the next step is to determine and execute multiple data scripts based on the previously acquired data information, such as table structure, index, and constraint information. These scripts are used to reconstruct the data architecture and logical relationships in the target database. For example, based on the table structure files, index files, and constraint files in the source database, corresponding SQL scripts are generated. These scripts include operations for creating tables, indexes, constraints, and possible tablespace conversions in the target database.

[0049] Step S210: Execute multiple data scripts in the target database to determine multiple sets of exported file indexes and constraints in the target database.

[0050] In step S210 above, the data script contains a series of SQL commands used to create indexes and constraints in the target database that match those in the source database. Specifically:

[0051] 1. Index creation script: Contains SQL statements for creating various indexes (such as B-tree indexes, bitmap indexes, reverse key indexes, etc.). The purpose of these indexes is to speed up data retrieval, especially in large databases, where indexes can significantly improve query performance.

[0052] 2. Constraint Application Scripts: These include SQL statements for setting primary keys, foreign keys, uniqueness constraints, and checking constraints. Constraints are used to maintain database integrity, ensure data consistency, and prevent the insertion or updating of illegal data.

[0053] Multiple data scripts are generated from the data information. Each script corresponds to a set of tables in the data table list and contains commands for creating indexes and applying constraints for that set of tables. Typically, indexes are created first, followed by constraints, to maintain data processing consistency and avoid potential deadlocks. Furthermore, the number of scripts to be executed in parallel is determined based on the target database's resource capacity (such as CPU, memory, I / O, etc.) to fully utilize resources without overloading them.

[0054] When creating indexes and constraints, indexes can be created in batches and in a controlled manner for imported tables before constraints are created. For example, with 5000 tables, some with more indexes and some with fewer, let's say we want to create 30000 indexes. The keyword would be `cr_idx_META`. Then, within a shell loop, we can control the number of concurrent index creations, say up to 100. As long as `ps -ef|grepcr_idx_META < 100`, we continue creating the next index. Because the tablespace has already been configured, there won't be any issues with missing or full tablespaces, until all 30000 indexes are created. Constraint creation doesn't take too long; with existing indexes, a single script can also handle this.

[0055] Through steps S202 to S210, the efficiency of data migration is improved, thereby reducing labor costs. This solves the problem of high workload and low efficiency in related technologies that involve manually transferring content from the source database to the new database. The following is a further explanation.

[0056] In step S202 of the above data processing method, obtaining data information from the source database includes: determining a first statement to query the table structure file in the source database, a second statement to query the index file in the source database, and a third statement to query the constraint file in the source database; executing the first statement to determine the table structure file; executing the second statement to determine the index file; executing the third statement to determine the constraint file; and determining the data information based on the table structure file, index file, and constraint file.

[0057] In some embodiments of this application, the first statement is a SQL statement specifically used to query and retrieve the structure definitions of all tables in the source database. Executing the first statement extracts the structure information of all tables from the source database, including table names, field names, data types, field lengths, whether NULL values ​​are allowed, and other DDL (Data Definition Language) information. The extracted structure information is then organized into table structure files, which contain all the DDL statements required to create the tables, facilitating the subsequent reconstruction of these tables in the target database. The second statement is an SQL statement used to query the structure of all indexes in the source database. Executing the second statement retrieves detailed information about all indexes from the source database, including index names, index types (such as B-trees, bitmaps), the tables and fields on which the indexes are based, and any additional index parameters. The index information is then organized into index files, which contain the SQL statements required to create the indexes, facilitating the addition of indexes to the tables in the target database after the table structure reconstruction is completed. The third statement is an SQL statement used to query all table constraints (such as primary keys, foreign keys, unique constraints, etc.) in the source database. Executing the third statement retrieves the constraint definitions of all tables in the source database, including constraint types, affected tables and fields, and dependencies between constraints. This constraint information is then compiled into constraint files, which contain the SQL statements for setting constraints, ensuring that constraint relationships can be correctly established in the target database, maintaining data consistency and integrity. Based on the obtained table structure files, index files, and constraint files, the data information is determined.

[0058] In step S204 of the above data processing method, exporting data information to the target database includes: obtaining the environmental parameters of the target database and determining the data conversion script based on the environmental parameters; and executing the data conversion script in the target database.

[0059] In some embodiments of this application, environmental parameters refer to configuration information specific to the target database. This information may differ from that of the source database, including but not limited to database version, character set, tablespace settings, storage capacity limits, and potential security policies. The purpose of obtaining these environmental parameters is to ensure that data information can be appropriately converted during migration, avoiding data incompatibility or performance issues caused by environmental differences. Based on the obtained target database environmental parameters, data conversion scripts can be generated. These scripts are used to adjust the data information exported from the source database to meet the environmental requirements of the target database. For example, they may include the following operations:

[0060] 1. Adjust data types: If the target database does not support certain data types in the source database, the conversion script will convert the data types to a format supported by the target database;

[0061] 2. Encoding Conversion: If the source and target databases use different character sets, the conversion script will perform encoding conversion to ensure correct data display and operation;

[0062] 3. Tablespace and storage attribute adjustment: The conversion script will adjust the table's storage parameters, such as tablespace allocation, data file size, segment space management, etc., according to the target database's tablespace and storage configuration, to ensure that the data table can be created and used normally in the target environment;

[0063] 4. Application of permissions and security policies: The conversion script checks the user permissions and security policies of the target database to ensure that the data migration operation is carried out within the legal permission scope. At the same time, it assigns appropriate permissions to newly created objects (such as tables, indexes, constraints, etc.) as needed.

[0064] After determining the data transformation script, execute the script in the target database environment to transform and reconstruct the data information.

[0065] In step S206 of the above data processing method, the data table list is grouped to obtain multiple sets of export files, including: determining the total amount of data in the data table list; determining the number of groups corresponding to the data table list based on the total amount of data and the preset amount of data for each group; and grouping the data table list according to the number of groups to obtain multiple sets of export files.

[0066] In some embodiments of this application, the total data volume of all tables in the data table list is first calculated. This can be done by querying the database metadata or by directly performing data volume statistics within the database. In an Oracle database, the size of each table is first obtained, and the sizes of all tables are summed to obtain the total data volume of the entire database. The determination of the number of groups depends on two key parameters: the total data volume and the preset data volume for each group (that is, the maximum data volume of each exported file). The selection of the preset data volume should be based on the I / O performance of the storage device to ensure that a certain amount of data can be processed effectively per unit time. For example, if the tested storage I / O performance is 40GB per hour, and the total data volume of the entire database is 500GB, the preset data volume cutsize can be set to 12000, i.e., 12GB, which will generate approximately 500 / 12 = 41 exported files.

[0067] Determining the table data included in the exported file can be automated by writing a script containing the following logic:

[0068] 1. Sorting and Selection: First, sort the data tables according to their size, prioritizing larger tables for grouping. This is because processing larger tables is often more time-consuming, and distributing them evenly across groups can prevent excessive I / O load on certain tasks.

[0069] 2. Dynamic Grouping: Create multiple empty filegroups. Each time, retrieve one or more tables from the sorted list of data tables until a preset data volume threshold is reached. Then move to the next filegroup and continue until all tables have been allocated. The purpose of this is to ensure that the amount of data within each group is approximately equal, thus completing the export of each group within a similar timeframe.

[0070] 3. Recording and Marking: When generating export files, it is necessary to record the table information contained in each group of files, as well as any special processing instructions related to them, such as the tablespace, index space, data transformation rules, etc., to ensure that the original structure and attributes of the data can be correctly restored in the subsequent import process.

[0071] The above data processing method further includes: identifying whether the source database and the target database are on the same physical machine and obtaining the identification result; if the identification result indicates that the source database and the target database are on the same physical machine, data processing is performed using the first method; if the identification result indicates that the source database and the target database are not on the same physical machine, data processing is performed using the second method.

[0072] In some embodiments of this application, the locations of the source and target databases can be identified by querying the database's system information or by detecting their network connection status at the application level. For example, for an Oracle database, the hostname can be queried to determine if they are located on the same physical machine. If the source and target databases have the same hostname, they are located on the same physical machine; otherwise, they are located on different physical machines. Determining the data processing method based on different identification results can significantly improve the efficiency of data migration or synchronization while reducing potential network latency and data security risks.

[0073] In the above steps, the first method is used for data processing, including: obtaining the maximum number of processes supported by disk input / output; processing multiple sets of exported files in parallel according to the maximum number of processes, and monitoring the number of running processes through a control script; if the number of running processes is less than the maximum number of processes, starting a new process to process the remaining unprocessed files in the multiple sets of exported files, until all multiple sets of exported files are processed.

[0074] In some embodiments of this application, it is necessary to determine the maximum number of concurrent processes that disk input / output (i.e., disk I / O) can effectively support, i.e., the aforementioned maximum number of processes. After determining the maximum number of concurrent processes that disk I / O can support, parallel processing strategies can be designed to accelerate data migration. Specific steps include:

[0075] 1. Task Scheduling and Distribution: Multiple sets of exported files are assigned as independent tasks to multiple processes, with each process responsible for processing one set of files. Ideally, each process should handle a number of files commensurate with its capacity to ensure all processes remain active.

[0076] 2. Parallel Execution: Use concurrency control scripts (such as shell scripts) to start parallel processing processes, ensuring that the number of running processes never exceeds the maximum number of processes. Parallel processing scripts can be dynamic, automatically adjusting the timing of starting new processes based on the existing process states.

[0077] 3. Process Monitoring: Monitor the number of running processes in real time using scripts to ensure it does not exceed the maximum value supported by the disk system. This can be achieved by checking the number of data processing threads in system processes or using the database's built-in monitoring tools.

[0078] When the number of currently running processes falls below the maximum concurrency supported by disk I / O, new processes need to be started immediately to handle the remaining unfinished export file groups. Specific implementation strategies include:

[0079] 1. Automatic process replenishment: The parallel control script contains logic that continuously checks the number of currently running processes. Once it finds that the number is below the maximum value, it automatically starts additional processes to process the next file group.

[0080] 2. Work Queue Management: A work queue is created for the group of export files to be processed. New processes will take tasks from this queue for processing, and completed tasks will be removed. The queue is always dynamically updated.

[0081] The entire parallel processing flow will continue to run until all exported file groups allocated to the processes have been processed. This process includes continuous monitoring of the process execution status and error recovery mechanisms in case of exceptions or process termination.

[0082] In the above steps, a second method is used for data processing, including: obtaining the maximum number of export processes supported by the disk input / output of the physical machine where the source database is located, and obtaining the maximum number of import processes supported by the database storage input / output of the physical machine where the target database is located; exporting multiple sets of export files based on the maximum number of export processes, and importing multiple sets of export files based on the maximum number of import processes; monitoring the number of first processes performing export processing, and monitoring the number of second processes performing import processing; if the number of first processes is less than the maximum number of export processes, starting a new process to process the remaining unexported files in the multiple sets of export files until all multiple sets of export files are exported; if the number of second processes is less than the maximum number of import processes, starting a new process to process the remaining unimported files in the multiple sets of export files until all multiple sets of export files are imported.

[0083] In some embodiments of this application, when the source database and the target database are not on the same physical machine, data processing (including data export and import) needs to take into account the impact of network transmission and evaluate the disk I / O (input / output) capability of the physical machine where the source database is located and the database storage I / O capability of the physical machine where the target database is located, respectively.

[0084] The following are the detailed steps for data processing using the second method:

[0085] 1. Disk I / O evaluation of the source database physical machine: Determine the maximum number of concurrent export processes that the source physical machine's disk system can effectively support through benchmark testing or performance monitoring tools, i.e., the maximum number of export processes mentioned above;

[0086] 2. Database storage I / O evaluation of the target physical machine: Similarly, evaluate the maximum number of concurrent import processes that the database storage system of the target physical machine can support, i.e., the maximum number of import processes mentioned above;

[0087] 3. Export processing based on the maximum number of export processes: Design parallel control scripts to ensure that the number of export processes running simultaneously does not exceed the maximum number of export processes. Each process is responsible for processing a group of export files, utilizing the disk I / O capabilities of the physical machine where the source database resides to read data blocks and generate files;

[0088] 4. Import processing based on the maximum number of import processes: On the target physical machine, the import processes are managed using parallel control scripts based on the assessed maximum number of import processes. Each import process is responsible for a set of files exported from the source database, using the target database's storage I / O capabilities to write data blocks into the database table;

[0089] 5. Monitor the number of first processes performing export processing: During the export process, continuously monitor the number of currently running first processes (i.e., export processes) to ensure that it does not exceed the maximum number of export processes. This can be achieved, for example, through system monitoring tools, database management commands, or custom scripts;

[0090] 6. Monitor the number of second processes performing import processing: Similar to monitoring the export process, monitor the number of second processes (i.e., import processes) in real time during the import process to ensure that the storage I / O of the target database is not overloaded by too many concurrent processes;

[0091] 7. Dynamically start new processes: When the number of first processes (export processes) is lower than the maximum number of export processes, a new export process is automatically started to process the remaining unexported files until all export file groups are completed. Similarly, when the number of second processes (import processes) is lower than the maximum number of import processes, a new import process is started to process the remaining unimported files until all export file groups are imported.

[0092] The entire data processing flow described above will continue to run until all specified export file groups are exported and all exported file groups are imported into the target database.

[0093] The process of grouping, exporting, and importing data table lists is as follows: Figure 3 As shown, the data table list is divided into three export file groups: export file group 1, export file group 2, and export file group 3. These export file groups are control files. Different processing processes store the corresponding export file groups into the target database. For example, processing process 1 stores export file group 1 into the first position of the target database, processing process 2 stores export file group 2 into the second position of the target database, and processing process 3 stores export file group 3 into the third position of the target database.

[0094] The above data processing method further includes: obtaining log files for performing file export operations in the source database and file import operations in the target database, wherein the log files are used to record the start time, end time, and success or failure of the operations; checking the log files to obtain the check results; and re-executing the corresponding file operations if the check results indicate that the log files do not contain a success flag.

[0095] In some embodiments of this application, log files are automatically generated record files during database operations, typically containing the start time, end time, and success or failure information of the operation. In these embodiments, both file export and file import operations generate corresponding log files, which aid in subsequent error detection and recovery processes. After obtaining the log files corresponding to different operations, the success of the operation is determined by checking these log files. Specifically, the success of the operation can be determined by searching for keywords in the log file indicating success, such as "success." By automating this checking process through scripts, log files can be monitored periodically or in real-time, allowing for the timely detection of any failed export or import operations.

[0096] If a success flag is missing from the log file during inspection, it indicates that the corresponding operation failed to complete correctly. The error messages in the log file help pinpoint the cause of the failure and the specific stage of the operation. Based on the error location results, the failed file export or import operation is re-executed.

[0097] By acquiring and inspecting log files, combined with automated error recovery mechanisms, the reliability and efficiency of data migration or synchronization processes can be significantly enhanced, reducing delays and costs caused by operational failures.

[0098] Figure 4 This is a structural diagram of a data processing apparatus according to an embodiment of this application, such as... Figure 4 As shown, the device includes:

[0099] The acquisition module 40 is used to acquire data information from the source database, including table structure files, index files, and constraint files.

[0100] Module 42 is used to export data information to the target database and determine the list of data tables in the source database;

[0101] Grouping module 44 is used to group the data table list to obtain multiple export files;

[0102] Export module 46 is used to export multiple sets of export files to the target database in parallel, and to determine multiple data scripts based on the data information;

[0103] Execution module 48 is used to execute multiple data scripts in the target database to determine multiple sets of exported file indexes and constraints in the target database.

[0104] By utilizing the acquisition module, determination module, grouping module, export module, and execution module in the aforementioned data processing device, the efficiency of data migration is improved, thereby achieving the technical effect of reducing labor costs. This solves the technical problems of large workload and low efficiency in related technologies where the content of the source database is transferred to the new database manually.

[0105] In the acquisition module of the aforementioned data processing apparatus, the acquisition module is further configured to determine a first statement for querying the table structure file in the source database, a second statement for querying the index file in the source database, and a third statement for querying the constraint file in the source database; execute the first statement to determine the table structure file; execute the second statement to determine the index file; execute the third statement to determine the constraint file; and determine the data information based on the table structure file, the index file, and the constraint file.

[0106] In the determination module of the aforementioned data processing apparatus, the determination module is further configured to acquire environmental parameters of the target database, determine the data conversion script based on the environmental parameters, and execute the data conversion script in the target database.

[0107] In the grouping module of the aforementioned data processing device, the grouping module is further used to determine the total amount of data in the data table list; determine the number of groups corresponding to the data table list based on the total amount of data and the preset amount of data for each group; and group the data table list according to the number of groups to obtain multiple sets of export files.

[0108] The aforementioned data processing apparatus also includes a processing module 50, which is further used to identify whether the source database and the target database are on the same physical machine and obtain an identification result; if the identification result indicates that the source database and the target database are on the same physical machine, data processing is performed using a first method; if the identification result indicates that the source database and the target database are not on the same physical machine, data processing is performed using a second method.

[0109] In the processing module of the aforementioned data processing device, the processing module is further configured to obtain the maximum number of processes supported by disk input / output; process multiple sets of exported files in parallel based on the maximum number of processes, and monitor the number of running processes through a control script; if the number of running processes is less than the maximum number of processes, start a new process to process the remaining unprocessed files in the multiple sets of exported files until all multiple sets of exported files have been processed.

[0110] In the processing module of the aforementioned data processing device, the processing module is further configured to: obtain the maximum number of export processes supported by the disk input / output of the physical machine where the source database is located, and obtain the maximum number of import processes supported by the database storage input / output of the physical machine where the target database is located; export multiple sets of export files based on the maximum number of export processes, and import multiple sets of export files based on the maximum number of import processes; monitor the number of first processes performing export processing, and monitor the number of second processes performing import processing; if the number of first processes is less than the maximum number of export processes, start a new process to process the remaining unexported files in the multiple sets of export files until all multiple sets of export files are exported; if the number of second processes is less than the maximum number of import processes, start a new process to process the remaining unimported files in the multiple sets of export files until all multiple sets of export files are imported.

[0111] In the processing module of the aforementioned data processing device, the processing module is further configured to obtain log files for performing file export operations in the source database and file import operations in the target database. The log files are used to record the start time, end time, and success or failure of the operations. The log files are checked to obtain the check results. If the check results indicate that the log files do not contain a success flag, the corresponding file operations are re-executed.

[0112] It should be noted that, Figure 4 The data processing apparatus shown is used to perform Figure 2 The data processing method shown above also applies to the data processing apparatus, and will not be repeated here.

[0113] This application also provides an electronic device, which includes a memory and a processor. The memory stores program instructions, and the processor is connected to the memory and executes program instructions to perform the following functions: acquiring data information from a source database, wherein the data information includes table structure files, index files, and constraint files; exporting the data information to a target database and determining a list of data tables in the source database; grouping the list of data tables to obtain multiple sets of exported files; exporting the multiple sets of exported files to the target database in parallel and determining multiple data scripts based on the data information; and executing the multiple data scripts in the target database to determine the indexes and constraints of the multiple sets of exported files in the target database.

[0114] It should be noted that the aforementioned electronic equipment is used to perform Figure 2 The data processing method shown above also applies to this electronic device, and will not be repeated here.

[0115] This application embodiment also provides a non-volatile storage medium, which includes a stored computer program. The device containing the non-volatile storage medium executes the following data processing method by running the computer program: acquiring data information from a source database, wherein the data information includes table structure files, index files, and constraint files; exporting the data information to a target database and determining a list of data tables in the source database; grouping the list of data tables to obtain multiple sets of exported files; exporting the multiple sets of exported files in parallel to the target database and determining multiple data scripts based on the data information; and executing the multiple data scripts in the target database to determine the indexes and constraints of the multiple sets of exported files in the target database.

[0116] It should be noted that the aforementioned non-volatile storage media is used for execution. Figure 2 The data processing method shown above also applies to this non-volatile storage medium, and will not be repeated here.

[0117] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the data processing methods in various embodiments of this application.

[0118] This application also provides a computer program that, when executed by a processor, implements the steps of the data processing methods in various embodiments of this application.

[0119] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0120] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0121] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0122] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0123] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0124] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0125] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, include: Obtain data information from the source database, wherein the data information includes table structure files, index files, and constraint files; Export the data information to the target database and determine the list of data tables in the source database; The data table list is grouped to obtain multiple sets of export files; The multiple sets of exported files are exported to the target database in parallel, and multiple data scripts are determined based on the data information; The plurality of data scripts are executed in the target database to determine multiple sets of exported file indexes and constraints in the target database.

2. The method according to claim 1, characterized in that, Retrieve data information from the source database, including: The system determines a first statement to query the table structure file in the source database, a second statement to query the index file in the source database, and a third statement to query the constraint file in the source database. Execute the first statement to determine the table structure file; Execute the second statement to determine the index file; Execute the third statement to determine the constraint file; The data information is determined based on the table structure file, the index file, and the constraint file.

3. The method according to claim 2, characterized in that, Exporting the data information to the target database includes: Obtain the environmental parameters of the target database, and determine the data conversion script based on the environmental parameters; The data transformation script is executed in the target database.

4. The method according to claim 1, characterized in that, The data table list is grouped to obtain multiple sets of exported files, including: Determine the total amount of data in the data table list; Based on the total data volume and the preset data volume for each group, determine the number of groups corresponding to the data table list; The data table list is grouped according to the number of groups to obtain the multiple groups of exported files.

5. The method according to claim 1, characterized in that, The method further includes: Identify whether the source database and the target database are on the same physical machine to obtain the identification result; If the identification result indicates that the source database and the target database are on the same physical machine, the first method is used for data processing. If the identification result indicates that the source database and the target database are not on the same physical machine, a second method is used for data processing.

6. The method according to claim 5, characterized in that, The first method is used for data processing, including: Get the maximum number of processes supported by disk I / O; The multiple sets of exported files are processed in parallel based on the maximum number of processes, and the number of running processes is monitored by a control script. If the number of running processes is less than the maximum number of processes, a new process is started to process the remaining unprocessed files in the multiple sets of exported files until all the multiple sets of exported files have been processed.

7. The method according to claim 5, characterized in that, The second method of data processing includes: Obtain the maximum number of export processes supported by the disk input / output of the physical machine where the source database is located, and obtain the maximum number of import processes supported by the database storage input / output of the physical machine where the target database is located; The multiple sets of exported files are exported according to the maximum number of export processes, and the multiple sets of exported files are imported according to the maximum number of import processes. Monitor the number of first processes performing export processing and monitor the number of second processes performing import processing; If the number of the first process is less than the maximum number of export processes, a new process is started to process the remaining unexported files in the multiple sets of export files until all the multiple sets of export files are exported. If the number of the second process is less than the maximum number of import processes, a new process is started to process the remaining unimported files in the multiple sets of exported files until all the multiple sets of exported files are imported.

8. The method according to claim 1, characterized in that, The method further includes: Obtain log files for the file export operation performed in the source database and the file import operation performed in the target database, wherein the log files are used to record the start time, end time and whether the operation was successful; The log file was examined to obtain the examination results; If the check result indicates that the log file does not contain a success flag, the corresponding file operation is re-executed.

9. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire data information from the source database, wherein the data information includes table structure files, index files, and constraint files; The determination module is used to export the data information to the target database and determine the list of data tables in the source database; The grouping module is used to group the data table list to obtain multiple sets of export files; The export module is used to export the multiple sets of export files to the target database in parallel, and to determine multiple data scripts based on the data information; An execution module is used to execute the plurality of data scripts in the target database to determine multiple sets of exported file indexes and constraints in the target database.

10. An electronic device, characterized in that, include: Memory, used to store program instructions; A processor, connected to the memory, is configured to execute program instructions that perform the following functions: acquiring data information from a source database, wherein the data information includes table structure files, index files, and constraint files; exporting the data information to a target database and determining a list of data tables in the source database; grouping the list of data tables to obtain multiple sets of exported files; exporting the multiple sets of exported files in parallel to the target database and determining multiple data scripts based on the data information; and executing the multiple data scripts in the target database to determine the indexes and constraints of the multiple sets of exported files in the target database.

11. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored computer program, wherein the device containing the non-volatile storage medium executes the data processing method according to any one of claims 1 to 8 by running the computer program.

12. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the data processing method according to any one of claims 1 to 8.

Images