Database security migration method and device based on memory protection

Abstract

The invention relates to a database security migration method and device based on memory protection, and aims to solve the problems of memory overflow (OOM) and migration interruption caused by rate mismatching under a read-write thread separation architecture. The method comprises the steps that the memory occupation amount of data read by a read thread is counted in real time and accumulated to a cache accumulated value, and the value is deducted to dynamically update the memory occupation after write-in by a write thread; when the cache accumulated value exceeds a preset backlog threshold value, pausing a read thread and scheduling the read thread to assist a write task; and continuously checking the accumulated value in the assistance mode, and recovering the original task of the read thread if the accumulated value reaches the standard. According to the method, through dynamic memory management and control and thread scheduling, excessive accumulation of the memory is accurately inhibited, and OOM faults are avoided; the efficient migration advantage of read-write separation is reserved, migration stability is guaranteed through closed-loop regulation and control, collaborative improvement of efficiency, stability and data safety is achieved, and the method is suitable for high-concurrency and large-data-volume migration scenes of the database.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database migration, and in particular to a database security migration method and device based on memory protection, a computer readable storage medium and an electronic device. BACKGROUND

[0002] With the continuous improvement of the demand for independent controllability of domestic information technology, the domestic database technology and product system has achieved significant breakthroughs and continuous improvement. Under this background, various key information infrastructure operators and enterprises are gradually promoting the migration and replacement of the original Oracle database to domestic distributed databases (such as openGauss, Vastbase, etc.) based on data security, supply chain stability and compliance requirements. As the core link of system upgrade, the efficiency and stability of the cut operation of database migration directly affect business continuity, so it is necessary to rely on professional database migration tools to complete the efficient and reliable migration of data.

[0003] The current mainstream database migration tools generally adopt a multi-thread architecture design of read-write thread separation, which improves the migration efficiency by decoupling data reading and writing operations: the read thread is responsible for batch extracting data from the source database, and the write thread is responsible for writing the extracted data to the target database, and the two are executed in parallel to shorten the overall migration time. However, this architecture faces key technical bottlenecks in actual operation: due to the differences in hardware performance between the source and target, different data processing logic (such as index construction, transaction submission strategy), or network delay, etc. The data reading rate of the read thread and the data writing rate of the write thread are prone to dynamic imbalance, which is specifically manifested as: when the read thread rate is significantly higher than the write thread rate, the data that has not been written in time will continue to accumulate in the memory buffer, causing the memory occupancy to grow exponentially with the migration process; if the duration of this state is too long, the memory resources will be exhausted and an out-of-memory (OOM) fault will be triggered, ultimately forcing the migration process to be interrupted. Such problems not only seriously affect the success rate of the migration task, but also may cause business data loss or increase the cost of system recovery, greatly reducing the stability and reliability of database migration.

[0004] In summary, the existing migration tools based on read-write separation architecture have the technical defect of memory overflow risk caused by the mismatch of read-write rates when dealing with high-throughput and large-scale data migration scenarios. Therefore, it is urgent to design a database security migration method with a memory protection mechanism to ensure the efficiency of migration while effectively suppressing the abnormal growth of memory occupancy through dynamic regulation of read-write rates, optimization of memory resource allocation, and other means, thereby achieving the coordinated improvement of efficiency and stability of the migration process. SUMMARY

[0005] To address the aforementioned issues, this application proposes a novel memory-protected secure database migration method. This application aims to overcome the shortcomings of existing technologies in database migration processes (such as migrating from Oracle databases to relational databases like openGauss and Vastbase G100), which suffer from speed mismatches, excessive memory consumption, and susceptibility to OutOfMemory (OOM) errors and migration interruptions due to the separate architecture of read and write threads. It provides a memory protection method suitable for these database migration scenarios to suppress excessive memory consumption during data migration, avoid OOM errors and migration interruptions, and ultimately achieve a balance between migration efficiency, migration stability, and data security.

[0006] This application proposes a memory protection mechanism based on data volume statistics and threshold control for database migration scenarios. By calculating the data size in real time during the execution of read and write threads and using a preset "backlog threshold" as the control benchmark, the task allocation of read and write threads is dynamically scheduled to achieve memory protection. The overall technical architecture of this solution is as follows: Figure 1 As shown, its core functions and processes are described below: 1. Data size statistics and memory usage are dynamically updated. During the migration process, this solution monitors the task execution status of read and write threads in real time: when the read thread reads data from the source database (such as an Oracle database), the program synchronously calculates the size of the data read into the memory cache and accumulates this data size to form a "cumulative value of memory cache data"; when the write thread writes data to the target database (such as a Vastbase G100 database), the program immediately deducts the size of the written data from the "cumulative value of memory cache data", thus dynamically releasing the memory usage.

[0007] 2. Backlog threshold determination and thread scheduling triggering When there is a difference in execution rates between read and write threads (read thread rate is higher than write thread rate), which may lead to excessive memory usage, this solution allows users to flexibly configure the core parameters of read and write threads and the "backlog threshold" according to the actual migration scenario to adapt to different migration needs. When the "cumulative value of cached data in memory" exceeds the preset "backlog threshold", the migration program immediately triggers the scheduling mechanism: suspending the read thread's operation of reading data from the source database, and simultaneously scheduling the read thread to switch to write task assistance mode to assist the write thread in performing the task of writing data to the target database.

[0008] 3. Thread state restoration and loop determination After the read thread enters the write task assistance mode, after each assisted write operation is completed, the migration program recalculates the "cumulative value of memory cached data" and compares it with the "backlog threshold". If the "cumulative value of memory cached data" is still greater than the "backlog threshold", the read thread continues to maintain the assistance mode and continues to assist the write thread in processing the remaining write tasks. If the "cumulative value of memory cached data" is less than or equal to the "backlog threshold", the read thread exits the assistance mode and resumes executing the original task of reading data from the source database. At the same time, the write thread continues to execute the data write task submission until the current round of data migration is completed.

[0009] Specifically, this application provides the following technical solutions: The first aspect of this application provides a memory-protected database secure migration method, the method comprising: S1. Start the migration task, establish a separate concurrent channel for reading and writing threads from the source database to the target database, and initialize memory statistics variables and preset backlog thresholds. S2. The read thread reads the source data into the memory buffer in batches, accumulates the current batch byte count in memory statistics in real time, and submits the batch write task to the write thread pool. S3. The write thread executes the batch write task asynchronously. When the task is submitted and completed, the corresponding number of bytes for the batch write task is deducted from the memory statistics variable to realize the dynamic release of the memory statistics variable. S4. Continuously compare memory statistics with the backlog threshold: If the memory statistics are less than or equal to the backlog threshold, maintain the original concurrency of the read and write threads; if the memory statistics are greater than the backlog threshold, immediately pause the read operation of the read thread and switch the read thread to "write task assistance mode" to pull pending tasks from the write thread pool to accelerate memory consumption. S5. Each time a read thread in the assist mode completes a write task, it updates the memory statistics variable and executes the comparison logic of S4 again until the memory statistics variable is less than or equal to the backlog threshold, at which point the original read action of the read thread is resumed. S6. Repeat S2-S5 until all data migration is complete.

[0010] Furthermore, in the method of this application, the memory statistics variable is the cumulative value of memory cache data.

[0011] Furthermore, in the method of this application, the backlog threshold is in GB and is dynamically set by the user through a web visualization interface based on the difference between the source end read rate and the target end write rate.

[0012] Furthermore, in the method of this application, if the write thread pool is empty and the memory statistics variable is greater than the backlog threshold during the assist mode, the read thread enters a timed sleep mode. After the sleep mode ends, the task is retrieved again to prevent idle running.

[0013] Furthermore, in the method of this application, after the write thread completes the write task and releases the memory, if it detects that the current memory statistics variable is less than or equal to the backlog threshold, it actively sends a wake-up signal to the sleeping read thread to wake up the sleeping read thread and enable the read thread to immediately resume the data reading operation.

[0014] Furthermore, in the method of this application, the batch size is fixed at 10MB, and the batch data read operation of the read thread is immediately triggered when the accumulated byte amount reaches 10MB.

[0015] Furthermore, in the method of this application, the update of the memory statistical variable adopts atomic operation to ensure the measurement accuracy under multi-threaded concurrency.

[0016] A second aspect of this application provides a memory-protected database secure migration apparatus, the apparatus comprising: The system settings module is used to start the migration task, establish a separate concurrent channel for reading and writing threads from the source database to the target database, and initialize memory statistics variables and preset backlog thresholds. The data reading module is used by the read thread to read source data into the memory buffer in batches, accumulate the current batch size in bytes to the memory statistics variable in real time, and submit the batch write task to the write thread pool. The data writing module is used to asynchronously execute the batch write tasks by the write thread. When the task submission is completed, the corresponding number of bytes for the batch write task is deducted from the memory statistics variable to realize the dynamic release of the memory statistics variable. The thread scheduling module continuously compares memory statistics with the backlog threshold: if the memory statistics are less than or equal to the backlog threshold, the original concurrency of read and write threads is maintained; if the memory statistics are greater than the backlog threshold, the read thread's read operation is immediately paused, and the read thread is switched to "write task assistance mode" to pull pending tasks from the write thread pool to accelerate memory consumption. The thread recovery module is used to update the memory statistics variable after each write task is completed by the read thread in assisted mode, and then execute the comparison logic of the thread scheduling module again until the memory statistics variable is less than or equal to the backlog threshold, after which the original read action of the read thread is resumed.

[0017] The device implements the steps of the aforementioned memory-protected database secure migration method during operation.

[0018] Furthermore, the device of this application also includes: The wake-up module, located on the write thread side, is used to send a wake-up signal to the sleeping read thread when the memory statistics variable is less than or equal to the backlog threshold.

[0019] A third aspect of this application provides an electronic device, including: a memory and a processor; Memory: Used to store computer programs; Processor: Used to execute the computer program to implement the steps of the aforementioned memory-protected database secure migration method.

[0020] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned memory-protected database secure migration method.

[0021] In summary, compared with existing technologies, the memory-protected database security migration method proposed in this application, while retaining the efficiency of the read-write thread separation architecture, achieves the following technical effects by innovatively integrating memory management and dynamic scheduling mechanisms: (1) Precise dynamic control of memory usage This method establishes a full lifecycle tracking mechanism for cached data in memory—accumulating cached data in real time during the data reading phase and precisely deducting the corresponding cache count after data is successfully written to the target end. Combined with a preset "backlog threshold" judgment rule, it can dynamically perceive the accumulation trend of cached data in memory. When the amount of cached data is detected to be approaching or exceeding the threshold, inhibitory control logic is triggered to fundamentally avoid the problem of continuously rising memory usage caused by read / write rate imbalance, significantly reducing the probability of memory overflow (OOM) failures and achieving fine-grained management of memory resources during the migration process.

[0022] (2) A fundamental improvement in migration stability To address the backlog and blocking issues caused by the "fast read, slow write" nature of traditional read-write separation architectures, this solution employs a dynamic compensation mechanism that schedules read threads to assist write tasks when memory exceeds a threshold. When the memory cache approaches its critical value, the system proactively coordinates some read threads to take on auxiliary data writing tasks. By flexibly adjusting thread allocation, the system promptly resolves data backlog and effectively prevents migration interruptions caused by memory overflow. This mechanism overcomes the inherent contradiction of "efficiency first, prone to instability" in traditional architectures, ensuring the continuous and stable operation of the migration task throughout its entire lifecycle.

[0023] (3) Synergistic assurance of efficiency, stability and security This method ensures high migration efficiency by fully preserving the parallel processing advantages of a read / write thread-separated architecture, maintaining a high-throughput data migration rate. In terms of stability, memory management and dynamic scheduling mechanisms eliminate the risk of interruptions caused by OutOfMemoryError (OOM). Regarding data security, the stable migration process avoids data loss or inconsistency due to task interruptions, while the memory management mechanism reduces the potential impact of program anomalies on data integrity. These three aspects work synergistically to achieve the triple technical value of "efficient migration, stable operation, and data security."

[0024] Tests have shown that this solution maintains high-efficiency data transmission capabilities even in complex scenarios such as high-concurrency, large-volume database migration, while also providing reliable memory protection and process stability assurance, offering comprehensive technical support for database migration tasks of critical business systems.

[0025] Other features and advantages of this application will be set forth in detail in the following description, or will become apparent through the implementation of the relevant technical solutions of this application. The objectives and other advantages of this application can be achieved through the technical features and means explicitly pointed out in the description, claims, and drawings, and will be obtained through the implementation of these technical contents. Attached Figure Description

[0026] To more clearly illustrate the technical solution of this application, the accompanying drawings involved in the description of this solution will be briefly introduced below. It should be noted that the drawings only show some embodiments of this application. For those skilled in the art, other related drawings can be derived from these drawings without creative effort.

[0027] Figure 1 This is a diagram of the overall technical architecture of this application.

[0028] Figure 2 This is a flowchart illustrating the overall implementation process of the memory-protected database secure migration method proposed in this application.

[0029] Figure 3 This is a flowchart illustrating the implementation process of an embodiment of this application.

[0030] Figure 4 This is a structural diagram of the database secure migration device based on memory protection according to this application.

[0031] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0033] In this document, the term "comprising" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not exhaustive and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors. The term "one embodiment" should be understood as "at least one embodiment," meaning that the described embodiment is not the only possible implementation, and other similar embodiments may exist.

[0034] In this application, the terms "a" and "a plurality of" are used to modify related elements or features, and their expression is illustrative rather than restrictive. Unless otherwise expressly stated in the context, "a" should be understood as "at least one," and "a plurality of" should be understood as "at least two." Those skilled in the art should reasonably interpret these terms based on the semantic and logical relationships of the context to ensure that they cover the possibility of "one or more."

[0035] Figure 2 The diagram shows the overall implementation flow of the memory-protected database security migration method provided in this application, including the following steps: S1. Start the migration task, establish a separate concurrent channel for reading and writing threads from the source database to the target database, and initialize memory statistics variables and preset backlog thresholds. S2. The read thread reads the source data into the memory buffer in batches, accumulates the current batch byte count in memory statistics in real time, and submits the batch write task to the write thread pool. S3. The write thread executes the batch write task asynchronously. When the task is submitted and completed, the corresponding number of bytes for the batch write task is deducted from the memory statistics variable to realize the dynamic release of the memory statistics variable. S4. Continuously compare memory statistics with the backlog threshold: If the memory statistics are less than or equal to the backlog threshold, maintain the original concurrency of the read and write threads; if the memory statistics are greater than the backlog threshold, immediately pause the read operation of the read thread and switch the read thread to "write task assistance mode" to pull pending tasks from the write thread pool to accelerate memory consumption. S5. Each time a read thread in the assist mode completes a write task, it updates the memory statistics variable and executes the comparison logic of S4 again until the memory statistics variable is less than or equal to the backlog threshold, at which point the original read action of the read thread is resumed. S6. Repeat S2-S5 until all data migration is complete.

[0036] To more clearly illustrate the technical solution of this application, the following will provide further explanation through specific scenario embodiments.

[0037] This embodiment proposes a memory protection mechanism based on data volume statistics and threshold control for the migration scenario from an Oracle database to a Vastbase G100 database. Figure 3 The diagram shows the overall implementation process of the solution in this embodiment.

[0038] I. System Configuration Module (Web Interface Interaction) This solution provides a web configuration page that allows users to visually configure core migration parameters. Specific configurable parameters include: 1. Backlog threshold: The unit is GB. It is used to set the maximum amount of data allowed to accumulate in memory. When the actual amount of data accumulated in memory reaches this threshold, the system will automatically trigger the read thread pause and scheduling logic.

[0039] 2. Number of concurrent read threads: The number of read threads can be configured according to the read performance of the source database (Oracle) to achieve multi-threaded parallel reading.

[0040] 3. Write thread concurrency: The number of write threads can be configured according to the target library (Vastbase G100) to adapt to the processing capabilities of the target library.

[0041] II. Read Thread Data Processing and Memory Statistics Flow When a read thread performs data read and write operations, it follows these steps: 1. Single data row read and concatenation: After the read thread reads a single data row from the Oracle database, it concatenates the data of each column of the row into the CopyString of the copy command.

[0042] 2. Memory Calculation: After a single data row is read and transferred, calculate the memory size occupied by the current CopyString and add this size to the memory statistics variable bytesOccupied (i.e., the cumulative value of memory cached data).

[0043] 3. Batch submission trigger: The read thread continues to repeat steps 1-2 to read the next data row. When the accumulated data volume corresponding to bytesOccupied reaches the preset "submission batch size" (fixed to 10MB in this embodiment), the read thread submits the current batch of write tasks to the write thread pool, and the write thread executes the data writing operation asynchronously.

[0044] III. Memory Release Process for Write Threads After the write thread obtains a write task from the thread pool and performs the data write operation, it triggers the memory release logic: After the write thread completes the data write task to the target library (Vastbase G100), it immediately deducts the total data size corresponding to the write task from the memory statistics variable bytesOccupied, thereby realizing the real-time release of memory usage and ensuring that bytesOccupied is consistent with the actual accumulated data size in memory.

[0045] IV. Thread Dynamic Scheduling Mechanism (Read Thread Scheduling Logic) The system monitors the relationship between the memory statistics variable bytesOccupied and the preset backlog threshold in real time. When bytesOccupied exceeds the backlog threshold, the read thread's source database data reading operation is paused, and the read thread scheduling process is initiated. The read thread obtains the write tasks to be processed from the write thread pool and executes them in the following three scenarios: 1. Scenario A: There are pending write tasks. If there are pending write tasks in the write thread pool, the read thread switches to "write task assistance mode" to assist the write thread in processing the pending write tasks. By increasing the concurrency of write task processing, the writing and releasing speed of memory data is accelerated, and bytesOccupied is reduced. When bytesOccupied drops below the backlog threshold, the read thread exits the assistance mode and resumes the source database data reading operation.

[0046] 2. Scenario B: No pending write tasks and bytesOccupied exceeds the threshold. If there are no pending write tasks in the write thread pool, but bytesOccupied is still greater than the backlog threshold, the read thread enters a sleep state (in this embodiment, the sleep duration is set to 5 seconds). After the sleep ends, the read thread recycles and pulls pending tasks from the write thread pool to avoid memory overflow (OOM) failures caused by the inability to release memory data.

[0047] 3. Scenario C: No pending write tasks and bytesOccupied is below the threshold. If there are no pending write tasks in the write thread pool and bytesOccupied has dropped below the backlog threshold, the read thread will exit the scheduling loop directly and resume reading data from the source database.

[0048] V. Write Thread Wake-up Mechanism After the write thread completes the data writing task and releases memory, it triggers the wake-up logic: The write thread checks whether the current bytesOccupied is less than the backlog threshold. If the condition is met, the write thread actively sends a wake-up signal to the system to wake up the read thread that is in a dormant state (dormant in scenario B), so that the read thread can immediately resume the data reading operation and avoid migration efficiency loss caused by dormancy.

[0049] In summary, this solution achieves dynamic control over memory usage through a full-process design of "visualized configuration, precise data processing, scenario-based thread scheduling, and proactive wake-up." It retains the high-efficiency migration advantages of separating read and write threads while ensuring migration stability and data security through a closed-loop scheduling mechanism, adapting to migration needs in different database environments.

[0050] Figure 4 The image shows a memory-protected database security migration device proposed in this application. The device includes: The system settings module is used to start the migration task, establish a separate concurrent channel for reading and writing threads from the source database to the target database, and initialize memory statistics variables and preset backlog thresholds. The data reading module is used by the read thread to read source data into the memory buffer in batches, accumulate the current batch size in bytes to the memory statistics variable in real time, and submit the batch write task to the write thread pool. The data writing module is used to asynchronously execute the batch write tasks by the write thread. When the task submission is completed, the corresponding number of bytes for the batch write task is deducted from the memory statistics variable to realize the dynamic release of the memory statistics variable. The thread scheduling module continuously compares memory statistics with the backlog threshold: if the memory statistics are less than or equal to the backlog threshold, the original concurrency of read and write threads is maintained; if the memory statistics are greater than the backlog threshold, the read thread's read operation is immediately paused, and the read thread is switched to "write task assistance mode" to pull pending tasks from the write thread pool to accelerate memory consumption. The thread recovery module is used to update the memory statistics variable after each write task is completed by the read thread in assisted mode, and then execute the comparison logic of the thread scheduling module again until the memory statistics variable is less than or equal to the backlog threshold, after which the original read action of the read thread is resumed.

[0051] When the above-mentioned device is in operation, it implements the steps of the memory-protected database secure migration method disclosed in this application.

[0052] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of apparatus, methods, and computer program products according to various embodiments of this application, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should be noted that each block in the block diagrams and / or flowcharts, and combinations thereof, can be implemented using either a dedicated hardware-based system or a combination of dedicated hardware and computer instructions to achieve the specified function or operation.

[0053] like Figure 5As shown in the illustration, an embodiment of this application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a processor-executable computer program, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the aforementioned memory-protected database secure migration method.

[0054] It is understood that, in addition to memory and a processor, this electronic device may also include input devices (such as a keyboard), output devices (such as a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor through I / O interfaces (i.e., input / output interfaces).

[0055] The operations described in this application can be implemented by writing computer program code using one or more programming languages ​​or a combination thereof. The programming languages ​​include, but are not limited to, the following types: Object-oriented programming languages, such as Java, Smalltalk, C++, etc. Conventional procedural programming languages, such as "C" or similar programming languages.

[0056] The execution methods of program code include, but are not limited to: It runs entirely on the user's computer; Part of it executes on the user's computer, and part of it executes on a remote computer; Execute as a standalone software package; It is executed entirely on a remote computer or server.

[0057] In scenarios involving remote computers, the remote computer can connect to the user's computer via any type of network, including but not limited to local area networks (LANs) or wide area networks (WANs). Furthermore, the remote computer can also connect to external computers via an internet service provider, for example, by utilizing the internet.

[0058] Furthermore, this application also discloses a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the various steps of the memory-protected database secure migration method disclosed in this application.

[0059] In the context of this application, a computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Specific examples include, but are not limited to, the following: (1) Portable computer disk: such as floppy disks and other removable magnetic storage media.

[0060] (2) Hard disk: including mechanical hard disks and solid-state hard disks and other fixed storage devices.

[0061] (3) Random Access Memory (RAM): A volatile storage medium used for temporary storage of data and program code.

[0062] (4) Read-only memory (ROM): a non-volatile storage medium used to store fixed programs and data.

[0063] (5) Erasable programmable read-only memory (EPROM) or flash memory: non-volatile storage media that supports multiple erasures and reprogrammings.

[0064] (6) Fiber optic storage devices: storage media based on fiber optic technology.

[0065] (7) Portable compact disc read-only memory (CD-ROM): a read-only medium that stores data in the form of an optical disc.

[0066] (8) Optical storage devices: such as DVDs, Blu-ray discs and other storage media based on optical principles.

[0067] (9) Magnetic storage devices: such as magnetic tapes, disks and other storage media based on magnetic principles.

[0068] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage needs.

[0069] These computer-readable storage media can be used to store the program code and related data described in this application to support program execution and persistent data storage.

[0070] Specifically, according to embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. This computer program contains program code for executing the memory-protected database secure migration method disclosed in this application. When the computer program is executed by a processing device, it can achieve the functions defined in the embodiments of this application.

[0071] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features. Furthermore, this application should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the foregoing disclosed concept.

[0072] Those skilled in the art should also understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of this application. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A database secure migration method based on memory protection, characterized in that, The method includes: S1. Start the migration task, establish a separate concurrent channel for reading and writing threads from the source database to the target database, and initialize memory statistics variables and preset backlog thresholds. S2. The read thread reads the source data into the memory buffer in batches, accumulates the current batch byte count in memory statistics in real time, and submits the batch write task to the write thread pool. S3. The write thread executes the batch write task asynchronously. When the task is submitted and completed, the corresponding number of bytes for the batch write task is deducted from the memory statistics variable to realize the dynamic release of the memory statistics variable. S4. Continuously compare memory statistics with the backlog threshold: If the memory statistics are less than or equal to the backlog threshold, maintain the original concurrency of the read and write threads; if the memory statistics are greater than the backlog threshold, immediately pause the read operation of the read thread and switch the read thread to "write task assistance mode" to pull pending tasks from the write thread pool to accelerate memory consumption. S5. Each time a read thread in the assist mode completes a write task, it updates the memory statistics variable and executes the comparison logic of S4 again until the memory statistics variable is less than or equal to the backlog threshold, at which point the original read action of the read thread is resumed. S6. Repeat S2-S5 until all data migration is complete.

2. The method according to claim 1, characterized in that, The memory statistics variable is the cumulative value of memory cached data.

3. The method according to claim 1, characterized in that, The backlog threshold is in GB and is dynamically set by the user through a web visualization interface based on the difference between the source end read rate and the target end write rate.

4. The method according to claim 1, characterized in that, During the assist mode, if the write thread pool is empty and the memory statistics variable is greater than the backlog threshold, the read thread will enter a timed sleep mode. After the sleep mode ends, it will fetch tasks again to prevent idle running.

5. The method according to claim 4, characterized in that, After completing the write task and releasing memory, if the write thread detects that the current memory statistics variable is less than or equal to the backlog threshold, it actively sends a wake-up signal to the sleeping read thread to wake it up and enable the read thread to immediately resume data reading operations.

6. The method according to claim 1, characterized in that, The batch size is fixed at 10MB. When the accumulated bytes reach 10MB, the batch data read operation of the read thread is immediately triggered.

7. The method according to claim 1, characterized in that, The update of the memory statistics variables adopts atomic operations to ensure measurement accuracy under multi-threaded concurrency.

8. A database security migration device based on memory protection, characterized in that, The device includes: The system settings module is used to start the migration task, establish a separate concurrent channel for reading and writing threads from the source database to the target database, and initialize memory statistics variables and preset backlog thresholds. The data reading module is used by the read thread to read source data into the memory buffer in batches, accumulate the current batch size in bytes to the memory statistics variable in real time, and submit the batch write task to the write thread pool. The data writing module is used to asynchronously execute the batch write tasks by the write thread. When the task submission is completed, the corresponding number of bytes for the batch write task is deducted from the memory statistics variable to realize the dynamic release of the memory statistics variable. The thread scheduling module continuously compares memory statistics with the backlog threshold: if the memory statistics are less than or equal to the backlog threshold, the original concurrency of read and write threads is maintained; if the memory statistics are greater than the backlog threshold, the read thread's read operation is immediately paused, and the read thread is switched to "write task assistance mode" to pull pending tasks from the write thread pool to accelerate memory consumption. The thread recovery module is used to update the memory statistics variable after each write task is completed by the read thread in assisted mode, and then execute the comparison logic of the thread scheduling module again until the memory statistics variable is less than or equal to the backlog threshold, after which the original read action of the read thread is resumed.

9. The apparatus according to claim 8, characterized in that, The device further includes: The wake-up module, located on the write thread side, is used to send a wake-up signal to the sleeping read thread when the memory statistics variable is less than or equal to the backlog threshold.

10. An electronic device, characterized in that, include: Memory and processor; Memory: Used to store computer programs; Processor: for executing the computer program to implement the steps of the memory-protected database secure migration method as described in any one of claims 1-7.

Images