Methods and related products for converting payload files for heterogeneous databases

Abstract

This invention provides a method and related products for converting payload files in heterogeneous databases. The method includes: in response to a conversion command, obtaining a payload file from the heterogeneous database; parsing the payload file to obtain a parsing result; obtaining an ordered index list corresponding to a target file in a local database, the target file being used to store the converted payload file; determining the target file to be written based on the ordered index list; converting the parsing result to obtain a conversion result recognizable by the local database; and writing the conversion result to the target file. This method enables the conversion of payload files between heterogeneous databases, thereby supporting further processing of payload files by domestic databases.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a method for converting payload files in heterogeneous databases and related products. Background Technology

[0002] In current popular database technologies, many customers hope to replace their databases with domestic ones without modifying the application source code. Oracle Database provides a method and apparatus for capturing database system workloads and replaying those workloads within the database system, namely the RAT (Real Application Load) component. RAT has the capability to replay real application workloads and supports synchronization ranges of different granularities (TIME / SCN). However, it is powerless to replay real application workloads across heterogeneous databases. Therefore, a tool is needed that can capture workloads between heterogeneous databases, thereby promoting the technological development of domestic database replacement. Summary of the Invention

[0003] In view of the above problems, the present invention proposes a method and related products for converting load files of heterogeneous databases to overcome or at least partially solve the above problems.

[0004] One objective of this invention is to enable the conversion of payload files between heterogeneous databases.

[0005] A further objective of this invention is to accelerate the conversion efficiency of payload files;

[0006] Another further objective of this invention is to make the reading, sorting, and merging of payload file contents more efficient;

[0007] Specifically, the present invention provides a method for converting payload files of heterogeneous databases, comprising:

[0008] In response to the transformation command, retrieve the payload file from the heterogeneous database;

[0009] The payload file is parsed to obtain the parsing results;

[0010] Retrieve an ordered list of indexes corresponding to the target file from the local database. The target file is used to store the transformed payload file.

[0011] The target file to be written is determined based on the ordered index list;

[0012] The parsed results are transformed to obtain a transformed result that can be recognized by the local database;

[0013] Write the conversion results to the target file.

[0014] Optionally, the steps for obtaining the load file in the heterogeneous database include: calling a preset read thread pool to allocate read threads to obtain the load file in the file system of the heterogeneous database, wherein the preset read thread pool is used to read the load file in parallel;

[0015] The process of obtaining the load file from the heterogeneous database also includes: calling a pre-defined write thread pool to allocate write threads to parse the load file and obtain the parsing results. The pre-defined write thread pool is used to parse the load file and write the parsing results to the target file in parallel.

[0016] Optionally, the steps of obtaining the ordered index list corresponding to the target file in the local database include:

[0017] The local database retrieves the data information corresponding to the target file from its built-in database. The data information includes the timestamp, file handle identifier, data offset, and storable data length of the target file. The built-in database is used to record the data information corresponding to the target file.

[0018] Encapsulate the data information into an index corresponding to the target file;

[0019] Sort the indexes according to their timestamps to obtain an ordered list of indexes.

[0020] Optionally, the index includes a file handle identifier, a data offset, and the length of data that can be stored;

[0021] The steps to retrieve the data information corresponding to the target file from the built-in database of the local database include:

[0022] Determine if a file handle corresponding to the target file exists in the built-in database;

[0023] If so, obtain the file handle identifier corresponding to the file handle, the data offset in the target file corresponding to the file handle, and the length of data that can be stored.

[0024] If no file handle exists, obtain the preset output directory and check if the target file exists based on the preset output directory. The preset output directory is the directory where the target file is generated.

[0025] If the target file does not exist, create a new target file and generate the corresponding file handle.

[0026] Optionally, the step of writing the conversion result to the target file includes:

[0027] Determine the starting point for writing to the target file based on the data offset;

[0028] The amount of writable data remaining in the target file is determined based on the data offset and the length of storable data.

[0029] Determine if the amount of writable data is greater than the amount of data in the conversion result;

[0030] If the amount of writable data is greater than or equal to the amount of data in the conversion result, then all the conversion result will be written to the target file, and the data offset in the index will be updated.

[0031] Optionally, after determining whether the writable data volume is greater than the data volume of the conversion result, the following steps may also be included:

[0032] If the amount of writable data is less than the amount of data in the conversion result, the conversion result is split according to the amount of writable data. The portion of the conversion result that meets the writable data requirement is written to the target file, and the remaining portion of the conversion result is written to the next target file.

[0033] Optionally, the steps of transforming the parsed results to obtain a transformed result recognizable by the local database include:

[0034] The parsed results are transformed to convert the database operation statements in the parsed results into database operation statements that can be recognized by the local database.

[0035] Optionally, heterogeneous databases include: Oracle databases;

[0036] The load files include: database load information files and database operation instruction files.

[0037] According to another aspect of the present invention, a computer program product is also provided, comprising a computer program that, when executed by a processor, implements the steps of the above-described heterogeneous database load file conversion method.

[0038] According to another aspect of the present invention, a computer device is also provided, including a memory, a processor, and a machine-executable program stored in the memory and running on the processor, wherein the processor executes the machine-executable program to implement the steps of the above-described heterogeneous database load file conversion method.

[0039] The heterogeneous database payload file conversion method of the present invention first obtains the payload file from the heterogeneous database in response to a conversion command; then, it parses the payload file to obtain the parsing result; it obtains the ordered index list corresponding to the target file in the local database; next, it determines the target file to be written based on the ordered index list; it converts the parsing result to obtain a conversion result recognizable by the local database; and finally, it writes the conversion result to the target file. This method enables the conversion of payload files between heterogeneous databases, thereby supporting further processing of payload files by domestic databases.

[0040] Furthermore, the heterogeneous database load file conversion method of the present invention obtains the load file in the file system of the heterogeneous database by calling a preset read thread pool to allocate read threads, and the preset read thread pool is used to read the load file in parallel; subsequently, a preset write thread pool is called to allocate write threads to perform the step of parsing the load file to obtain the parsing result, and the preset write thread pool is used to parse the load file and write the parsing result to the target file in parallel. Through the thread pool mechanism, multi-threaded parallel processing of load file data is realized, thereby accelerating the conversion efficiency of load files.

[0041] Furthermore, the heterogeneous database load file conversion method of the present invention obtains the index and timestamp corresponding to the target file from the local database when determining the target file; then, the index is sorted according to the timestamp to obtain an ordered index list. By simulating the database indexing mechanism, an index is created and maintained for each target file during its generation. The introduction of this indexing mechanism makes the reading, sorting, and merging of file content more efficient.

[0042] The above and other objects, advantages and features of the present invention will become more apparent to those skilled in the art from the following detailed description of specific embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0043] The following sections will describe some specific embodiments of the invention in a detailed manner by way of example and not limitation, with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar parts or portions. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0044] Figure 1 This is a flowchart illustrating a method for converting payload files in a heterogeneous database according to an embodiment of the present invention.

[0045] Figure 2 This is a flowchart illustrating a heterogeneous database payload file conversion method according to another embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of a computer program product according to an embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention; and

[0048] Figure 5 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0049] Those skilled in the art should understand that the embodiments described below are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. These partial embodiments are intended to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention. Based on the embodiments provided by the present invention, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of the present invention.

[0050] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0051] This invention provides a method for converting payload files in heterogeneous databases. Figure 1 This is a flowchart illustrating a method for converting payload files in a heterogeneous database according to an embodiment of the present invention, as shown below. Figure 1 As shown, the heterogeneous database load file conversion method includes at least the following steps S101 to S106.

[0052] Step S101: In response to the conversion instruction, obtain the payload file from the heterogeneous database.

[0053] In some optional embodiments, the step of obtaining the payload file from the heterogeneous database generally includes: calling a preset read thread pool to allocate read threads to obtain the payload file in the file system of the heterogeneous database. The preset read thread pool is used for parallel reading of the payload file. Taking advantage of the thread pool's characteristics, when the number of threads reaches the set maximum number of threads, the task enters a blocked state until a new task becomes available. The preset read thread pool enables parallel reading of the payload file, thereby accelerating the reading speed.

[0054] Optionally, the following parameters can generally be configured in the default read thread pool:

[0055] a) Core Threads (corePoolSize): Defines the initial number of threads in the thread pool.

[0056] b) Maximum number of threads (maxPoolSize): Defines the maximum number of threads allowed in the thread pool.

[0057] c) Thread keepAliveTime: Defines the keep-alive time (in seconds) for an idle thread.

[0058] d) Time Unit: Specifies the unit of time a thread can live.

[0059] e) Task Queue (BlockingQueue): A queue used to store tasks to be processed.

[0060] f) ThreadFactory: A factory class used to create threads.

[0061] g) Rejection policy (RejectedExecutionHandler): Specifies the handling strategy for tasks when the thread pool is full.

[0062] Those skilled in the art can adjust the above parameters according to the performance of the actual database.

[0063] Step S102 involves parsing the payload file to obtain the parsing result. In this invention, two thread pools are used to handle the reading and writing of the payload file respectively. Therefore, after the preset read thread pool completes step S101, it calls the preset write thread pool to allocate a write thread to perform the step of parsing the payload file and obtaining the parsing result. The preset write thread pool is used to parse the payload file and write the parsing result to the target file in parallel. Subsequent steps S103 to S106 are completed by the write thread calling the corresponding output method. This avoids data writing chaos caused by thread switching.

[0064] Step S103: Obtain the ordered index list corresponding to the target file in the local database. In some optional embodiments, the step of obtaining the ordered index list corresponding to the target file in the local database generally includes: obtaining the data information corresponding to the target file in the built-in database of the local database. The data information includes the timestamp, file handle identifier, data offset, and storable data length corresponding to the target file; encapsulating the data information into an index corresponding to the target file; and sorting the index according to the timestamp to obtain an ordered index list. Sorting the index according to the timestamp can generate a globally ordered index list, so that in subsequent processing, the file content can be read one by one according to the index order and output to a new file, ensuring the orderliness of data writing. Among them, the built-in database is generally used to record the data information corresponding to the target file. It is mainly used to store the data information of the target file during the operation of the local database. This data information is usually only used by the local database itself. An optional example of the built-in database is the H2 lightweight database, which has the characteristics of low resource consumption, high performance, and easy embedding. Those skilled in the art can choose the specific type of built-in database according to the actual situation.

[0065] The aforementioned built-in database is only a preferred embodiment of the method of the present invention. Those skilled in the art may also choose to store the data information of the target file directly in a local database.

[0066] Optionally, the index typically includes a file handle identifier, a data offset, and the length of data that can be stored;

[0067] The steps for retrieving data information corresponding to a target file from the built-in database of a local database generally include: determining whether a file handle corresponding to the target file exists in the built-in database; if so, obtaining the file handle identifier, the data offset in the target file, and the length of data that can be stored; if the file handle does not exist, obtaining a preset output directory and querying whether the target file exists based on the preset output directory, which is the target file's generation directory; if the target file does not exist, creating a new target file and generating a file handle corresponding to the target file. By maintaining a set of file handles in the memory of the local database, it is possible to determine the target file to write to, thereby avoiding frequent opening and closing of files. At the operating system level, a file handle acts like a "locator." When a program requests to open a file, the operating system allocates a file handle for that file. This handle uniquely identifies the file's location and related resources within the operating system. For example, in a multi-user or multi-tasking operating system environment, multiple programs may run simultaneously and all need to access files in the file system. File handles ensure that each program can accurately access the file it needs, just as each key (file handle) corresponds to a specific room (file). Therefore, by maintaining file handles, the accurate writing of the payload file can be guaranteed.

[0068] During the creation of the target file, a directory path for generating the target file is passed in advance. If the file handle does not exist, the preset output directory can be obtained through the target path, and the existence of the target file can be checked based on the preset output directory. The preset output directory is the directory where the target file is generated. If the target file does not exist, a new target file is created and the corresponding file handle is generated.

[0069] In addition, to ensure database performance and avoid resource waste, a maximum number of file handles can be set. When the number of handles exceeds the limit, handles that have not been used for a long time will be closed.

[0070] An example of an optional index is as follows:

[0071] fileId: File handle identifier.

[0072] offset: The data offset within the file.

[0073] length: The length of data that can be stored.

[0074] A timestamp can be represented as call_time.

[0075] After calling the output method, the write thread retrieves four pieces of data corresponding to the target file from the local database: fileId, offset, length, and call_time. These data are then encapsulated into an object, which provides a Compare method for comparison.

[0076] An example of the encapsulated object is: Call_time:{field, offset, length}.

[0077] After aggregating the indexes of all target files and creating the corresponding objects, the indexes can be globally sorted by the call_time field to generate an ordered index list.

[0078] Step S104: Determine the target file to be written based on the ordered index list. The sorted index list sequentially represents the contents of different target files from beginning to end. For each index, open the file pointed to by fileId, locate the specific position based on offset, and read the data segment of length length. Furthermore, during the writing process, the read data can be written to the new target file sequentially. The existence of file handles avoids frequent opening and closing of files, and the unified management of the file handle pool efficiently handles target file read and write operations.

[0079] Step S105 involves converting the parsed result to obtain a converted result recognizable by the local database. In some optional embodiments, the step of converting the parsed result to obtain a converted result recognizable by the local database generally includes: converting the parsed result to convert the database operation statements in the parsed result into database operation statements recognizable by the local database. This conversion can transform the payload file into a file recognizable by the local database, thereby facilitating subsequent processing of the payload file. The conversion method can be by calling a dedicated conversion tool or running a specific conversion method. Those skilled in the art can choose according to the actual situation.

[0080] Step S106: Write the conversion result to the target file. In some optional embodiments, the step of writing the conversion result to the target file generally includes: determining the writing start point in the target file based on the data offset; determining the remaining writable data amount in the target file based on the data offset and the storable data length; determining whether the writable data amount is greater than the data amount of the conversion result; if the writable data amount is greater than or equal to the data amount of the conversion result, then writing the entire conversion result to the target file and updating the data offset in the index.

[0081] If the amount of writable data is less than the amount of data in the conversion result, the conversion result is split according to the amount of writable data. The portion of the conversion result that meets the writable data requirement is written to the target file, and the remaining portion of the conversion result is written to the next target file.

[0082] This method enables the conversion of payload files between heterogeneous databases, thereby supporting further processing of payload files by domestic databases.

[0083] Optionally, heterogeneous databases typically include: Oracle databases.

[0084] A workload file typically includes a database workload information file and a database operation command file. The database workload information file is generally a WMD file, which primarily records information obtained after executing commands such as Start_Capture, Finish_Capture, and Workload_filter. The database operation command file is generally a REC file, which records commands sent from the client to the database, including SQL statements for user login (User_login), user logoff (User_logoff), transactions, simple queries (Simple_query), and SQL statements based on the PBE protocol.

[0085] Because Oracle databases and local domestic databases differ in communication protocols and database structures, it is necessary to use the method of this invention to parse the WMD and REC files in the production workload and convert them into a format that meets the playback requirements of local domestic databases, thereby supporting subsequent workload playback.

[0086] In addition, the method of the present invention also supports the participation of transactions. Each thread is separated by transactions during execution. Even the read thread that reads the payload file and the write thread that writes data can be isolated and protected by a transaction.

[0087] In summary, the method of the present invention has the following functions and features:

[0088] (1) Flexible and controllable output format: By configuring different parameters, it can output files in various formats, including binary format files, for use by local domestic database servers.

[0089] (2) File format conversion: It can convert WMD files captured by heterogeneous databases (such as Oracle) into new WMD files that meet the requirements of local domestic databases. At the same time, it can convert REC files captured by heterogeneous databases into new REC files for use by KES database, thereby enabling the server to correctly parse specific attributes and contents in the file.

[0090] (3) File splitting and sorting: It supports sorting the load file captured by a heterogeneous database according to the time axis and outputting it to multiple new files; it also supports splitting according to file size to meet different application needs.

[0091] (4) Independence of transactions and session connections: The method of the present invention can output data in the same transaction or data in the same connection to the same file, ensuring that data from different transactions or connections are strictly separated and will not be mixed in the same file.

[0092] Figure 2 This is a flowchart illustrating a heterogeneous database payload file conversion method according to another embodiment of the present invention, as shown below. Figure 2 As shown, the heterogeneous database load file conversion method includes at least the following steps S201 to S215.

[0093] Step S201: In response to the conversion instruction, the preset read thread pool is invoked to allocate read threads to obtain the load file in the file system of the heterogeneous database.

[0094] Step S202: Call the preset write thread pool to allocate write threads to parse the load file and obtain the parsing result.

[0095] Step S203: Determine whether the file handle corresponding to the target file exists in the built-in database of the local database.

[0096] Step S204: If the determination in step S203 is negative, obtain the preset output directory and query whether the target file exists based on the preset output directory.

[0097] Step S205: If the query result in step S204 is yes, open the file handle corresponding to the target file.

[0098] Step S206: If the query result in step S204 is negative, create a new target file and generate the file handle corresponding to the target file.

[0099] Step S207: If step S203 is satisfied or either step S205 or step S206 is completed, obtain the file handle identifier corresponding to the file handle, the data offset in the target file corresponding to the file handle, and the length of data that can be stored.

[0100] Step S208: Encapsulate the data information into an index corresponding to the target file.

[0101] Step S209: Sort the indexes according to the timestamps to obtain an ordered index list.

[0102] Step S210: Determine the target file to be written based on the ordered index list.

[0103] Step S211: The parsing result is converted to convert the database operation statements in the parsing result into database operation statements that can be recognized by the local database.

[0104] Step S212: Determine the start point for writing in the target file based on the data offset.

[0105] Step S213: Determine the amount of writable data remaining in the target file based on the data offset and the length of storable data.

[0106] Step S214: Determine whether the amount of writable data is greater than the amount of data in the conversion result.

[0107] In step S215, if the determination in step S214 is yes, write all the conversion results to the target file and update the data offset in the index.

[0108] Step S216: If the determination in step S214 is negative, the conversion result is divided according to the amount of writable data. The part of the conversion result that meets the writable data requirement is written to the target file, and the remaining part of the conversion result is written to the next target file.

[0109] In addition, the method of the present invention also supports the participation of transactions. Each thread is separated by transactions during execution. Even the read thread that reads the payload file and the write thread that writes data can be isolated and protected by a transaction.

[0110] The method of this invention aims to convert payload files from heterogeneous databases into a format that can be parsed and replayed by a local domestic database. This method can parse and convert REC and WMD files captured from heterogeneous databases into payload files that meet the requirements of local domestic databases, supporting the reproduction and performance verification of real-world production scenarios and providing technical support for compatibility and performance testing during subsequent heterogeneous database migrations.

[0111] Specifically, the method of this invention first employs a thread pool mechanism to achieve multi-threaded parallel processing of the load file data. Through the blocking design of the thread pool, the system can orderly acquire the load file data and allocate parsing tasks when the number of files is large or the workload is heavy. This approach not only improves the concurrency performance of data processing but also effectively avoids the risks of resource contention and thread overload.

[0112] Secondly, to address the sorting and management challenges caused by an excessive number of output files, this invention introduces a simulated database indexing mechanism. Each target file has an index created and maintained during its generation, containing key information such as file handles, data offsets, and data lengths. This indexing mechanism makes reading, sorting, and merging file content more efficient. By sorting all index entries according to their timestamps (e.g., call_time), the system can generate a globally ordered index list, allowing subsequent processing to read file content sequentially according to the index order and output it to a new file.

[0113] Finally, using the method of this invention, the system can parse REC and WMD files captured in heterogeneous database (e.g., Oracle) production systems. These files are stored in binary format in the Oracle database and contain critical operational information such as user login, transaction processing, and SQL statements. Through the tool's parsing and format conversion functions, the original REC and WMD files are processed into new REC and WMD files that conform to the format requirements of the local domestic database, thereby supporting further load preprocessing and load replay processes.

[0114] In summary, the heterogeneous capture method of this invention provides a comprehensive and realistic verification approach for database replacement at low cost. By accurately capturing and reproducing operational loads in the production environment, it can realistically simulate the operational behavior and performance of the database in actual application scenarios. This method effectively reduces the high resource consumption and time costs of traditional testing, thereby providing reliable data support and assurance for change evaluation during database replacement or upgrades. This method not only reduces resource consumption and time costs during testing but also improves the accuracy and reliability of replacement verification. This invention not only provides an efficient and controllable solution for database replacement verification but also lays a solid foundation for the smooth transition of the system in the production environment, further promoting the technological development of domestic database replacement.

[0115] The flowchart provided in this embodiment is not intended to indicate that the operations of the method will be performed in any particular order, or that all operations of the method are included in every case. Furthermore, the method may include additional operations. Within the scope of the technical concept provided by the method in this embodiment, additional variations can be made to the above method.

[0116] It should be understood that in some embodiments, the components may be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods may be implemented using software or firmware stored in memory and executed by a suitable instruction execution system.

[0117] This embodiment also provides a computer program product 10, a computer-readable storage medium 20, and a computer device 30. Figure 3 This is a schematic diagram of a computer program product 10 according to an embodiment of the present invention. Figure 4 This is a schematic diagram of a computer-readable storage medium 20 according to an embodiment of the present invention. Figure 5 This is a schematic diagram of a computer device 30 according to an embodiment of the present invention. The computer program product 10 includes a computer program 11, which, when executed by the processor 32, implements the steps of the heterogeneous database payload file conversion method described above. A computer-readable storage medium 20 stores the computer program 11 thereon, which, when executed by the processor 32, implements the steps of the heterogeneous database payload file conversion method described above. The computer device 30 may include a memory 31, a processor 32, and the computer program 11 stored on the memory 31 and running on the processor 32.

[0118] The computer program 11 used to perform the operations of this invention may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​and procedural programming languages. The computer program 11 may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a Local Area Network (LAN) or Wide Area Network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform aspects of this invention, electronic circuits, including, for example, programmable logic circuits, Field-Programmable Gate Arrays (FPGAs), or Programmable Logic Arrays (PLAs), may execute computer-readable program instructions using status information from computer-readable program instructions to personalize the electronic circuits.

[0119] For the purposes of this embodiment, computer program product 10 is a related product containing computer program 11. For the purposes of this embodiment, computer-readable storage medium 20 is a tangible device capable of holding and storing computer program 11, and can be any device capable of containing, storing, communicating, propagating, or transmitting program 11 for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable storage medium 20 include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanical encoding device, and any suitable combination thereof.

[0120] Computer device 30 can be, for example, a server, desktop computer, laptop computer, tablet computer, or smartphone. In some examples, computer device 30 can be a cloud computing node. Computer device 30 can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. Computer device 30 can be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can reside on local or remote computing system storage media, including storage devices.

[0121] Computer device 30 may include a processor 32 adapted to execute stored instructions and a memory 31 that provides temporary storage space for the operation of said instructions during operation. The processor 32 may be a single-core processor, a multi-core processor, a computing cluster, or any other configuration. The memory 31 may include random access memory (RAM), read-only memory, flash memory, or any other suitable storage system.

[0122] Computer device 30 may also include a network adapter / interface and an input / output (I / O) interface. The I / O interface allows external devices that can be connected to the computer device to input and output data. The network adapter / interface provides communication between the computer device and a network, typically represented as a communication network.

[0123] Therefore, those skilled in the art should recognize that although numerous exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications conforming to the principles of the present invention can be directly determined or derived from the disclosure of the present invention without departing from the spirit and scope of the invention. Thus, the scope of the present invention should be understood and construed as covering all such other variations or modifications.

Claims

1. A method for converting payload files in heterogeneous databases, comprising: In response to the conversion command, the payload file in the heterogeneous database is retrieved; The load file is parsed to obtain the parsing result; Retrieve an ordered list of indexes corresponding to the target file from the local database. The target file is used to store the transformed payload file. The target file to be written is determined based on the ordered index list; The parsing results are transformed to obtain a transformation result that can be recognized by the local database; The conversion result is written to the target file.

2. The method for converting payload files in heterogeneous databases according to claim 1, wherein, The step of obtaining the load file in the heterogeneous database includes: calling a preset read thread pool to allocate read threads to obtain the load file in the file system of the heterogeneous database, wherein the preset read thread pool is used to read the load file in parallel; After the step of obtaining the load file in the heterogeneous database, the method further includes: calling a preset write thread pool to allocate write threads to perform the step of parsing the load file to obtain the parsing result, wherein the preset write thread pool is used to parse the load file and write the parsing result to the target file in parallel.

3. The method for converting payload files in heterogeneous databases according to claim 1, wherein, The steps for obtaining the ordered index list corresponding to the target file in the local database include: The data information corresponding to the target file is obtained from the built-in database of the local database. The data information includes the timestamp, file handle identifier, data offset and storable data length of the target file. The built-in database is used to record the data information corresponding to the target file. The data information is encapsulated into an index corresponding to the target file; The ordered index list is obtained by sorting the indexes according to the timestamps.

4. The method for converting payload files in heterogeneous databases according to claim 3, wherein, The index includes a file handle identifier, a data offset, and a length of data that can be stored. The step of obtaining the data information corresponding to the target file from the built-in database of the local database includes: Determine whether a file handle corresponding to the target file exists in the built-in database; If so, obtain the file handle identifier corresponding to the file handle, the data offset in the target file corresponding to the file handle, and the storable data length; If the file handle does not exist, obtain a preset output directory and query whether the target file exists based on the preset output directory, where the preset output directory is the generation directory of the target file; If the target file does not exist, create a new target file and generate the file handle corresponding to the target file.

5. The method for converting payload files in heterogeneous databases according to claim 4, wherein, The step of writing the conversion result to the target file includes: The starting point for writing to the target file is determined based on the data offset; The remaining writable data in the target file is determined based on the data offset and the storable data length. Determine whether the amount of writable data is greater than the amount of data in the conversion result; If the amount of writable data is greater than or equal to the amount of data in the conversion result, then all of the conversion result is written to the target file, and the data offset in the index is updated.

6. The method for converting payload files in heterogeneous databases according to claim 5, wherein, After the step of determining whether the writable data volume is greater than the data volume of the conversion result, the following is also included: If the writable data volume is less than the data volume of the conversion result, the conversion result is divided according to the writable data volume. The portion of the conversion result that matches the writable data volume is written to the target file, and the remaining portion of the conversion result is written to the next target file.

7. The method for converting payload files in heterogeneous databases according to claim 1, wherein, The step of converting the parsed result to obtain a conversion result recognizable by the local database includes: The parsing results are transformed to convert the database operation statements in the parsing results into database operation statements that can be recognized by the local database.

8. The method for converting payload files in heterogeneous databases according to claim 1, wherein, The heterogeneous database includes: an Oracle database; The load file includes: a database load information file and a database operation instruction file.

9. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the heterogeneous database load file conversion method according to any one of claims 1 to 8.

10. A computer device comprising a memory, a processor, and a machine-executable program stored on the memory and running on the processor, wherein the processor, when executing the machine-executable program, implements the steps of the heterogeneous database load file conversion method according to any one of claims 1 to 8.

Images