A file migration system, method, apparatus, and storage medium

Abstract

The application discloses a file migration system, method, device and storage medium. The application relates to the field of file management. The system comprises a query thread management module, a download thread management module, an upload thread management module and an update thread management module. The query thread management module is used for assigning attribute information of a file to be migrated to a text object and transmitting the file to a waiting download queue; the download thread management module is used for assigning a file stream to the current text object and transmitting the file to a waiting upload queue; the upload thread management module is used for uploading the file stream to a migration database and transmitting the text object after the migration text identification is assigned to a waiting update queue; and the update thread management module is used for updating a migration file index table in the migration database. The technical scheme of the embodiment of the application splits the migration process of the file to be migrated into multiple stages such as query, download, upload and data table update, and different function threads are concurrently executed, so that the concurrent mechanism of the multithreading is fully utilized, and the file migration efficiency between databases is improved as a whole.

Description

Technical Field

[0001] This invention relates to the field of document management, and more particularly to a document migration system, method, apparatus, and storage medium. Background Technology

[0002] With the replacement of software products and physical machines, large-scale data migration is often required. Among them, the migration of unstructured data is particularly difficult due to the large number, size, and variety of file types.

[0003] To achieve the migration of unstructured data files, the existing migration scheme uses multiple threads to perform concurrent migration of multiple files. That is, each file to be migrated is migrated through a different thread, and each thread contains the complete file migration process for one file to be migrated.

[0004] However, in the above file migration method, the migration process of a file to be migrated is still executed serially within each thread, resulting in low file migration efficiency. At the same time, each thread is called for a long time, making it difficult to effectively monitor the intermediate migration status of each file to be migrated. Summary of the Invention

[0005] This invention provides a file migration system, method, apparatus, and storage medium to solve the problem of low efficiency in file migration between databases.

[0006] According to one aspect of the present invention, a file migration system is provided, comprising: a query thread management module, a download thread management module, an upload thread management module, and an update thread management module;

[0007] The query thread management module is used to assign attribute information of the file to be migrated to a text object based on the original file index table through query threads in the query thread pool, and to transmit the text object with the assigned attribute information to the waiting download queue; wherein, the attribute information includes the original text identifier;

[0008] The download thread management module is used to assign the corresponding file stream to the current text object based on the original text identifier of the current text object in the waiting download queue, and to transmit the current text object after the file stream assignment to the waiting upload queue.

[0009] The upload thread management module is used to upload the file stream of the current text object in the waiting upload queue to the migration database through the upload thread in the upload thread pool, assign the migration text identifier of the file stream to the current text object, and transmit the current text object after assigning the migration text identifier to the waiting update queue.

[0010] The update thread management module is used to update the migration file index table based on the migration text identifier of the current text object in the waiting update queue by using the update thread in the update thread pool.

[0011] According to another aspect of the present invention, a file migration method is provided, comprising:

[0012] The query thread management module uses query threads in the query thread pool to assign attribute information of the files to be migrated to text objects based on the original file index table, and then transmits the text objects with the assigned attribute information to the waiting download queue; wherein, the attribute information includes the original text identifier;

[0013] The download thread management module uses download threads in the download thread pool to assign the corresponding file stream to the current text object based on the original text identifier of the current text object in the waiting download queue, and then transmits the current text object with the assigned file stream to the waiting upload queue.

[0014] The upload thread management module uploads the file stream of the current text object in the waiting upload queue to the migration database through the upload thread in the upload thread pool, assigns the migration text identifier of the file stream to the current text object, and transmits the current text object with the assigned migration text identifier to the waiting update queue.

[0015] The update thread management module updates the migration file index table based on the migration text identifier of the current text object in the waiting update queue by updating the update thread in the update thread pool.

[0016] According to another aspect of the present invention, a file migration apparatus is provided, comprising:

[0017] The attribute information assignment module, configured in the query thread management module, is used to assign attribute information of the file to be migrated to a text object based on the original file index table through the query thread in the query thread pool, and to transmit the text object after attribute information assignment to the waiting download queue; wherein, the attribute information includes the original text identifier;

[0018] The file stream assignment module is configured in the download thread management module. It is used to assign the corresponding file stream to the current text object based on the original text identifier of the current text object in the waiting download queue through the download thread in the download thread pool, and to transmit the current text object after the file stream assignment to the waiting upload queue.

[0019] The migration file identifier assignment module is configured in the upload thread management module. It is used to upload the file stream of the current text object in the waiting upload queue to the migration database through the upload thread in the upload thread pool, assign the migration text identifier of the file stream to the current text object, and transmit the current text object with the migration text identifier assigned to it to the waiting update queue.

[0020] The migration file index table update module, configured in the update thread management module, is used to update the migration file index table based on the migration text identifier of the current text object in the waiting update queue by using the update thread in the update thread pool.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the file migration method described in any embodiment of the present invention.

[0022] The technical solution of this invention includes a query thread management module, a download thread management module, an upload thread management module, and an update thread management module in the file migration system. These modules are respectively used to obtain attribute information of the files to be migrated from the original database, download file streams from the original database, upload file streams to the migration database, and update information in the file index table of the migration database. This breaks down the migration process of the files to be migrated into multiple stages, such as querying, downloading, uploading, and updating data tables, which are executed concurrently by different functional threads. This fully utilizes the concurrency mechanism of multi-threading, improving the overall file migration efficiency. At the same time, each functional thread is not occupied by the current file to be migrated for a long time. Each functional thread can obtain the intermediate migration status of the current file after finishing its current processing task, thus realizing effective monitoring of the migration status of each file to be migrated. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the structure of a file migration system provided in Embodiment 1 of the present invention;

[0025] Figure 2 This is a schematic diagram of the structure of a file migration system provided in Embodiment 2 of the present invention;

[0026] Figure 3 This is a flowchart of a text migration method provided in Embodiment 3 of the present invention;

[0027] Figure 4 This is a schematic diagram of a file migration process provided in Embodiment 4 of the present invention;

[0028] Figure 5 This is a schematic diagram of a file migration device provided in Embodiment 5 of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] Example 1

[0032] Figure 1 A file migration system provided in Embodiment 1 of the present invention includes: a query thread management module 100, a download thread management module 200, an upload thread management module 300, and an update thread management module 400.

[0033] The query thread management module 100 is used to assign attribute information of the file to be migrated to a text object based on the original file index table through query threads in the query thread pool, and to transmit the text object after the attribute information is assigned to a waiting download queue; wherein, the attribute information includes the original text identifier.

[0034] Specifically, a file object is a Java object used to store the attribute fields of the migrated file. The original file index table is an index table in the original database that records and stores file attribute information. This attribute information can include not only the text identifier of the file stored in the original database, i.e., the original text identifier, but also relevant information describing the file status or file content, such as the number of downloads, the number of previews, and the publishing organization. Each file stored in the original database has one and only one original text identifier in the original file index table, and the original text identifiers of each file are different.

[0035] The query thread management module 100 obtains an idle query thread from the query thread pool and initializes a file object in that thread. Then, it queries the original file index table to assign attribute information such as the original text identifier of the file to be migrated to the text object, thereby establishing a one-to-one matching relationship between the text object and the file to be migrated. The query thread management module 100 then transmits the text object with the assigned attribute information to the waiting download queue, so that the download thread can obtain the matching file stream from the original database using the original text identifier of each text object in the waiting download queue.

[0036] Optionally, in this embodiment of the invention, the query thread management module 100 is further configured to create a first preset number of text objects through query threads in the query thread pool; wherein the text objects are matched one-to-one with the files to be migrated.

[0037] It should be noted that since the query thread management module 100 is used to assign attribute information of the files to be migrated to text objects, the data volume of the file attribute information is smaller than that of the file itself, and the processing speed of attribute information assignment is faster. Therefore, in order to make full use of query resources, multiple text objects can be created simultaneously through the currently idle thread. For example, when the first preset number is 500, one query thread can simultaneously create 500 text objects, query the attribute information of 500 files to be migrated, and assign the attribute information of the above 500 files to be migrated to the 500 text objects in a one-to-one matching manner. In this way, the attribute information assignment of multiple files to be migrated is realized through one query thread, making full use of query resources and improving file migration efficiency.

[0038] Optionally, in this embodiment of the invention, the attribute information further includes at least one of text type, text name, summary, and file migration status. The text type represents the text format of the file to be migrated, such as txt, doc, or pdf; the text name reflects the text title of the file; the summary describes the main content of the file; and the file migration status records the migration status of the file during the migration process. The query thread management module 100 obtains the file migration status of the file to be migrated from the original file index table through a query thread, indicating that the file has not yet been migrated to another database. This diverse attribute information comprehensively describes the basic information of the file to be migrated, improving the management and query efficiency after file storage.

[0039] The download thread management module 200 is used to assign the corresponding file stream to the current text object based on the original text identifier of the current text object in the waiting download queue, and to transmit the current text object after the file stream assignment to the waiting upload queue.

[0040] Specifically, when there is an idle download thread in the download thread pool, the download thread management module 200 obtains the current text object to be processed from the waiting download queue through the download thread, and downloads the file corresponding to the original text identifier from the original database by calling the download interface of the original database according to the original text identifier of the current text object; wherein, in this embodiment of the invention, the file is transmitted in the form of a file stream; the download thread management module 200 assigns the downloaded file to the text object, at which time the current text object includes the file body and the file's attribute information; and then transmits the assigned text object to the waiting upload queue.

[0041] Optionally, in this embodiment of the invention, the download thread management module 200 is further configured to set the file migration status of the current text object to download successful or download failed; if it is determined that the file migration status of the current text object is download failed, then the file stream of the current text object is downloaded again until the file stream of the current text object is downloaded successfully, or the number of downloads of the file stream of the current text object reaches the first threshold.

[0042] Specifically, the file migration status of the current text object is modified based on whether the file stream of the current text object was successfully downloaded. The file migration status can be either "download successful" or "download failed." If the file migration status is "download failed," the file stream of the current text object is downloaded again. If the download of the current text object's file stream still fails after reaching the initial threshold, continuing to download it would affect the migration speed of other files. Therefore, the download of the current file can be temporarily stopped, awaiting unified processing later. For example, when the number of text objects in the waiting download queue is insufficient, the files that failed to download can be re-downloaded using an idle thread.

[0043] For example, if the initial download threshold is set to 3, the download will temporarily stop when the file stream of the current text object reaches its third download attempt. The download thread management module 200 updates the download status of the current file to be migrated in real time by modifying the file migration status. This ensures that if a file download fails, a re-download mechanism can be used to ensure successful download. At the same time, setting the initial download threshold prevents the failure of a single corrupted file from causing other files to fail to migrate, thus ensuring the overall file migration speed.

[0044] The upload thread management module 300 is used to upload the file stream of the current text object in the waiting upload queue to the migration database through the upload thread in the upload thread pool, assign the migration text identifier of the file stream to the current text object, and transmit the current text object after assigning the migration text identifier to the waiting update queue.

[0045] Specifically, when there is an idle upload thread in the upload thread pool, the upload thread management module 300 uses this upload thread to retrieve the text object to be processed from the waiting upload queue, and uploads the file of the current text object to the migration database by calling the upload interface of the migration database; wherein, in this embodiment of the invention, the file is transmitted in the form of a file stream; the upload thread management module 300 retrieves the migration text identifier from the migration database and assigns the migration text identifier to the text object, at which time the current text object includes the file's attribute information and the migration text identifier; then the assigned text object is transmitted to the waiting update queue.

[0046] Optionally, in this embodiment of the invention, the upload thread management module 300 is further configured to set the file migration status of the current text object to upload success or upload failure; if it is determined that the file migration status of the current text object is upload failure, then the file stream of the current text object is downloaded again until the file stream of the current text object is successfully uploaded, or the number of uploads of the file stream of the current text object reaches the second threshold.

[0047] Specifically, the file migration status of the current text object is modified based on whether the file stream of the current text object was successfully uploaded. The file migration status can be either "upload successful" or "upload failed." If the file migration status is "upload failed," the file stream of the current text object is re-uploaded. If the upload attempt for the current text object's file stream fails after reaching the second threshold, continuing to upload it would affect the migration speed of other files. Therefore, the upload of the current file can be temporarily stopped, awaiting unified processing later. For example, when the number of text objects in the upload queue is insufficient, the files that failed to upload can be re-uploaded using an idle thread.

[0048] For example, if the second threshold is set to 3, the upload will temporarily stop when the file stream of the current text object reaches the third upload attempt. The upload thread management module 300 updates the file upload status of the current file to be migrated in real time by modifying the file migration status. This ensures that if a file upload fails, a re-upload mechanism can be used to ensure successful file upload. At the same time, the setting of the second threshold avoids the phenomenon that a single file corruption failure will prevent other files from being migrated, thus ensuring the overall file migration speed.

[0049] Optionally, in this embodiment of the invention, the upload thread management module 300 is further configured to obtain the upload file stream of the current text object according to the migration text identifier, and compare the similarity between the file stream of the current text object and the upload file stream to perform migration verification on the current text object, and update the file verification status and / or file verification count of the current text object.

[0050] For example, comparing the integrity of two file streams can be done by hashing the file streams. Specifically, before uploading a file stream, a hash function is used to calculate a hash value for the original byte stream (the file stream before upload, i.e., the downloaded file stream obtained from the original database). Then, the file is downloaded from the migration database, and the same hash function is used to calculate a hash value for the migrated byte stream (the downloaded file stream obtained from the migration database). After ensuring that the encoding of the new byte stream is consistent with the original byte stream, the same hash function is used to calculate its value. The resulting hash value is compared with the previously obtained hash value. If they are the same, the file integrity check passes; otherwise, the file integrity check fails. In particular, to avoid integrity checks reducing the overall migration efficiency of the database, a sampling method can be chosen for verification. For example, a file integrity check is performed every 20 migrated files.

[0051] Optionally, in this embodiment of the invention, the update thread management module 400 is used to update the migration file index table based on the migration text identifier of the current text object in the waiting update queue by using the update thread in the update thread pool.

[0052] Specifically, when there is an idle update thread in the update thread pool, the update thread management module 400 uses the update thread to obtain the current text object to be processed from the waiting update queue, and updates the attribute information of the current text object to the file index table of the migration database, i.e., the migration file index table, according to the migration text identifier of the current text object.

[0053] The update thread management module 400 is also used to set the file migration status of the current text object to update successful or update failed; if it is determined that the file migration status of the current text object is update failed, the attribute information of the current text object is updated again in the migration file index table until the attribute information of the current text object is updated successfully, or the number of times the attribute information of the current text object is updated reaches the third threshold.

[0054] Specifically, the attribute information of the current text object here includes other information besides the migration text identifier in the migration file index table and the file migration status. If any information fails to update, it can be considered an update failure. Based on whether the attribute information of the current text object was updated successfully, the file migration status of the current text object is modified; the file migration status can be either "update successful" or "update failed." If the file migration status is "update failed," the attribute information of the current text object is updated again. If the update of the current text object's attribute information fails after reaching the third update threshold, continuing to update it will affect the migration speed of other files. Therefore, the update of the current file can be temporarily stopped, awaiting later unified processing. For example, when the number of text objects in the waiting update queue is insufficient, the files that failed to update can be updated again using an idle thread.

[0055] For example, if the third threshold is set to 3, the update will temporarily stop when the attribute information of the current text object has been updated three times. The update thread management module 400 updates the file update status of the file to be migrated in real time by modifying the file migration status. This ensures that if a file update fails, a re-update mechanism can be used to ensure successful file updates. At the same time, setting the third threshold avoids the phenomenon that a single file corruption failure will prevent other files from being migrated, thus ensuring the overall file migration speed. In particular, the first, second, and third thresholds can all be pre-configured according to actual needs.

[0056] Optionally, in this embodiment of the invention, the query thread management module 100 is further configured to wake up the query thread in the query thread pool if it is determined that the number of text objects in the waiting download queue is less than or equal to a first quantity threshold and the number of text objects in the waiting update queue is less than or equal to a second quantity threshold; and / or the download thread management module 200 is further configured to wake up the download thread in the download thread pool if it is determined that the number of text objects in the waiting upload queue is less than or equal to a third quantity threshold and the number of text objects in the waiting update queue is less than or equal to a fourth quantity threshold; and / or the upload thread management module 300 is further configured to wake up the upload thread in the upload thread pool if it is determined that the number of text objects in the waiting update queue is less than or equal to a fifth quantity threshold.

[0057] It should be noted that, in order to avoid the accumulation of file objects, whether or not any thread is woken up depends on the data carrying capacity of the downstream communication queue, i.e., the number of text objects in the queue. Furthermore, since the waiting-to-update queue at the end of the migration process is the performance bottleneck of the entire migration process, whether or not other threads are woken up also depends on the number of text objects in the waiting-to-update queue.

[0058] The first threshold is determined by the data carrying capacity of the waiting download queue. If the number of text objects in the waiting download queue is less than or equal to the first threshold, it indicates that the waiting download queue has a large data carrying capacity. The second threshold is determined by the data carrying capacity of the waiting update queue and the text object processing efficiency of each download thread in the download thread pool. If the number of text objects in the waiting update queue is less than or equal to the second threshold, it indicates that the waiting update queue has a large data carrying capacity, and after a download thread processes the next text object and sends it to the downstream queue, subsequent operations by other threads will not cause a backlog of text objects in the update queue. Only when the waiting download queue and the waiting update queue have the capacity to carry new text objects will the query thread management module 100 wake up the idle query threads in the query thread pool to continue working.

[0059] The third threshold is determined by the data carrying capacity of the waiting upload queue. A number of text objects in the waiting upload queue that is less than or equal to the third threshold indicates that the waiting upload queue has a large data carrying capacity. The fourth threshold is determined by the data carrying capacity of the waiting update queue and the text object processing efficiency of each upload thread in the upload thread pool. A number of text objects in the waiting update queue that is less than or equal to the fourth threshold indicates that the waiting update queue has a large data carrying capacity, and that after an upload thread processes the next text object and sends it to the update queue, there will be no accumulation of text objects in the update queue. Only when the waiting upload queue and the waiting update queue have the capacity to accept new text objects will the download thread management module 200 wake up idle download threads in the download thread pool to continue working.

[0060] The fifth threshold is determined based on the data carrying capacity of the waiting update queue. A number of text objects in the waiting update queue that is less than or equal to the fifth threshold indicates that the waiting update queue has a large data carrying capacity. Only when the waiting update queue has a large data carrying capacity will the upload thread management module 300 wake up idle upload threads in the upload thread pool. It should be noted that the second, fourth, and fifth thresholds can be configured to the same value or different values.

[0061] The technical solution of this invention includes a query thread management module, a download thread management module, an upload thread management module, and an update thread management module in the file migration system. These modules are respectively used to obtain attribute information of the files to be migrated from the original database, download file streams from the original database, upload file streams to the migration database, and update information in the file index table of the migration database. This breaks down the migration process of the files to be migrated into multiple stages, such as querying, downloading, uploading, and updating data tables, which are executed concurrently by different functional threads. This fully utilizes the concurrency mechanism of multi-threading, improving the overall file migration efficiency. At the same time, each functional thread is not occupied by the current file to be migrated for a long time. Each functional thread can obtain the intermediate migration status of the current file after finishing its current processing task, thus realizing effective monitoring of the migration status of each file to be migrated.

[0062] Example 2

[0063] Figure 2 This invention provides a file migration system according to Embodiment 2 of the present invention. The relationship between this embodiment and the above embodiments is that the file migration system further includes a migration task management module 500.

[0064] After the file migration system starts up, the server creates and initializes a file migration scheduler using the file migration tool. This scheduler then creates a migration end-of-life scheduled task and a migration end-of-life trigger. The migration end-of-life scheduled task stops the query thread pool from querying new file objects by validating the pause query loop flag. This task is executed by the migration task management module 500. The migration end-of-life trigger sends a migration task end command to the migration task management module 500.

[0065] The migration task management module 500 is configured to, in response to receiving a migration task completion instruction, send a migration completion instruction to the query thread management module 100 to close active query threads in the query thread pool; in response to detecting that there are no active query threads in the query thread pool and no text objects in the waiting download queue, send a migration completion instruction to the download thread management module 200 to close active download threads in the download thread pool; in response to detecting that there are no active download threads in the download thread pool and no text objects in the waiting upload queue, send a migration completion instruction to the upload thread management module 300 to close active upload threads in the upload thread pool; and in response to detecting that there are no active upload threads in the upload thread pool and no text objects in the waiting update queue, send a migration completion instruction to the update thread management module 400 to close active update threads in the update thread pool.

[0066] Specifically, an active thread refers to a thread that is in a running state. When the migration task management module 500 receives a migration task termination instruction, it sends the instruction to the query thread management module 100. The query thread management module 100 will not call idle query threads after the currently active query thread has finished running. When the migration task management module 500 detects that there are no active query threads in the query thread pool and no text objects in the waiting download queue, it sends a migration termination instruction to the download thread management module 200. The download thread management module 200 will not call idle download threads after the currently active download thread has finished running. When the migration task management module 500 detects that there are no active download threads in the download thread pool and no text objects in the waiting upload queue, it sends a migration termination instruction to the upload thread management module 300. The upload thread management module 300 will not call idle upload threads after the currently active upload thread has finished running. When the migration task management module 500 detects that there are no active upload threads in the upload thread pool and no text objects in the waiting update queue, it sends a migration termination instruction to the update thread management module 400. The update thread management module 400 will not call idle update threads after the currently active update thread has finished running.

[0067] The technical solution of this invention, the migration task management module in the file migration system, uses the query thread as the starting point for the end of the migration task and sequentially closes each downstream thread pool. This ensures that after the downstream thread is closed, the upstream thread will not issue new text objects, thus avoiding the interruption of file migration due to unprocessed text objects and the resulting loss of file data.

[0068] Example 3

[0069] Figure 3 This is a flowchart of a text migration method provided in Embodiment 3 of the present invention. This embodiment is applicable to migrating files from an original database to a migration database. The method can be executed by a text migration device, which can be implemented in hardware and / or software. The text migration device can be configured in the text migration system of Embodiment 1 or Embodiment 2. The text migration system can be configured in electronic devices such as servers, typically in a data storage server. Figure 3 As shown, the method includes:

[0070] S310. The query thread management module uses query threads in the query thread pool to assign attribute information of the files to be migrated to text objects based on the original file index table, and transmits the text objects with the assigned attribute information to the waiting download queue; wherein, the attribute information includes the original text identifier.

[0071] S320. The download thread management module uses the download threads in the download thread pool to assign the corresponding file stream to the current text object based on the original text identifier of the current text object in the waiting download queue, and then transmits the current text object after the file stream assignment to the waiting upload queue.

[0072] S330, The upload thread management module uploads the file stream of the current text object in the waiting upload queue to the migration database through the upload thread in the upload thread pool, assigns the migration text identifier of the file stream to the current text object, and transmits the current text object with the assigned migration text identifier to the waiting update queue.

[0073] S340. The update thread management module updates the migration file index table based on the migration text identifier of the current text object in the waiting update queue by updating the update thread in the update thread pool.

[0074] The technical solution of this invention includes a query thread management module, a download thread management module, an upload thread management module, and an update thread management module in the file migration system. These modules are respectively used to obtain attribute information of the files to be migrated from the original database, download file streams from the original database, upload file streams to the migration database, and update information in the file index table of the migration database. This breaks down the migration process of the files to be migrated into multiple stages, such as querying, downloading, uploading, and updating data tables, which are executed concurrently by different functional threads. This fully utilizes the concurrency mechanism of multi-threading, improving the overall file migration efficiency. At the same time, each functional thread is not occupied by the current file to be migrated for a long time. Once each functional thread finishes its current processing task, it can obtain the intermediate migration status of the current file, thus achieving effective monitoring of the migration status of each file to be migrated.

[0075] Example 4

[0076] Figure 4 This is a schematic diagram of a file migration process provided in Embodiment 4 of the present invention, as shown below. Figure 4 As shown, the file migration process includes:

[0077] S401: The query thread management module initializes the text object; execute S401.

[0078] S402: The query thread management module queries the original text identifier, assigns the original text identifier to the current text object, and transmits the assigned text object to the waiting download queue; execute S402.

[0079] S403, The download thread management module downloads the file stream based on the original text identifier of the current text object in the waiting download queue; execute S404.

[0080] S404. The download thread management module determines whether the file stream has been downloaded successfully; if not, execute S405; if yes, execute S406.

[0081] S405, The download thread management module updates the file migration status of the current text object to download failed; execute S416.

[0082] S406. The download thread management module assigns the file stream to the current text object and then transmits the current text object after the file stream assignment to the waiting upload queue; execute S407.

[0083] S407. The upload thread management module will upload the file stream of the current text object in the upload queue to the migration database; execute S408.

[0084] S408. The upload thread management module determines whether the file stream of the current text object has been successfully uploaded; if not, execute S409; if yes, execute S410.

[0085] S409, The upload thread management module updates the file migration status of the current text object to upload failed; execute S416.

[0086] S410: The upload thread management module obtains the migration text identifier and assigns it to the current text object; execute S411.

[0087] S411, The upload thread management module performs integrity verification on the uploaded file stream; execute S412.

[0088] S412. The upload thread management module determines whether the verification was successful; if not, execute S413; if yes, execute S414.

[0089] S413, The upload thread management module updates the file verification status of the current text object to verification failure; execute S416.

[0090] S414. Update the migration text identifier of the current text object to the migration file index table in the thread management module; execute S415.

[0091] S415, Update the file migration status of the current text object to "update successful" in the thread management module; Execute S416.

[0092] S416, End.

[0093] The technical solution of this invention includes a query thread management module, a download thread management module, an upload thread management module, and an update thread management module in the file migration system. These modules are respectively used to obtain attribute information of the files to be migrated from the original database, download file streams from the original database, upload file streams to the migration database, and update information in the file index table of the migration database. This breaks down the migration process of the files to be migrated into multiple stages, such as querying, downloading, uploading, and updating data tables, all executed concurrently by different functional threads. This fully utilizes the concurrency mechanism of multi-threading, improving the overall efficiency of file migration. Furthermore, each functional thread is not occupied by the current file for an extended period; each thread can obtain the intermediate migration status of the current file after completing its current processing task, enabling effective monitoring of the migration status of each file to be migrated. In addition, after a file is successfully uploaded, it is downloaded again, and a hash value comparison is used to compare the previously uploaded file stream with the downloaded file stream, ensuring no data loss occurs during the file migration process and guaranteeing file integrity.

[0094] Example 5

[0095] Figure 5 This is a schematic diagram of a file migration device provided in an embodiment of the present invention. Figure 5As shown, the device includes: an attribute information assignment module 510, a file stream assignment module 520, a migration file identifier assignment module 530, and a migration file index table update module 540.

[0096] The attribute information assignment module 510 is configured in the query thread management module. It is used to assign the attribute information of the file to be migrated to the text object based on the original file index table through the query thread in the query thread pool, and to transmit the text object after the attribute information is assigned to the waiting download queue; wherein, the attribute information includes the original text identifier.

[0097] The file stream assignment module 520 is configured in the download thread management module. It is used to assign the corresponding file stream to the current text object based on the original text identifier of the current text object in the waiting download queue through the download thread in the download thread pool, and to transmit the current text object after the file stream assignment to the waiting upload queue.

[0098] The migration file identifier assignment module 530 is configured in the upload thread management module. It is used to upload the file stream of the current text object in the waiting upload queue to the migration database through the upload thread in the upload thread pool, assign the migration text identifier of the file stream to the current text object, and transmit the current text object with the migration text identifier assigned to it to the waiting update queue.

[0099] The migration file index table update module 540 is configured in the update thread management module and is used to update the migration file index table based on the migration text identifier of the current text object in the waiting update queue by the update thread in the update thread pool.

[0100] The technical solution of this invention includes a query thread management module, a download thread management module, an upload thread management module, and an update thread management module in the file migration system. These modules are respectively used to obtain attribute information of the files to be migrated from the original database, download file streams from the original database, upload file streams to the migration database, and update information in the file index table of the migration database. This breaks down the migration process of the files to be migrated into multiple stages, such as querying, downloading, uploading, and updating data tables, which are executed concurrently by different functional threads. This fully utilizes the concurrency mechanism of multi-threading, improving the overall file migration efficiency. At the same time, each functional thread is not occupied by the current file to be migrated for a long time. Each functional thread can obtain the intermediate migration status of the current file after finishing its current processing task, thus realizing effective monitoring of the migration status of each file to be migrated.

[0101] Example 6

[0102] In some embodiments, the file migration method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by a processor, one or more steps of the file migration method described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform the file migration method by any other suitable means (e.g., by means of firmware).

[0103] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0104] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0105] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0106] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0107] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0108] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0109] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0110] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A file migration system, characterized in that, include: Query the thread management module, download the thread management module, upload the thread management module, and update the thread management module; The query thread management module is used to assign attribute information of the file to be migrated to a text object based on the original file index table through query threads in the query thread pool, and to transmit the text object with the assigned attribute information to the waiting download queue; wherein, the attribute information includes the original text identifier; The download thread management module is used to assign the corresponding file stream to the current text object based on the original text identifier of the current text object in the waiting download queue, and to transmit the current text object after the file stream assignment to the waiting upload queue. The upload thread management module is used to upload the file stream of the current text object in the waiting upload queue to the migration database through the upload thread in the upload thread pool, assign the migration text identifier of the file stream to the current text object, and transmit the current text object after assigning the migration text identifier to the waiting update queue. The update thread management module is used to update the migration file index table based on the migration text identifier of the current text object in the waiting update queue by using the update thread in the update thread pool.

2. The file migration system according to claim 1, characterized in that, The attribute information also includes at least one of the following: text type, text name, summary, and file migration status.

3. The file migration system according to claim 2, characterized in that, The download thread management module is also used to set the file migration status of the current text object to download successful or download failed; if it is determined that the file migration status of the current text object is download failed, the file stream of the current text object is downloaded again until the file stream of the current text object is downloaded successfully, or the number of downloads of the file stream of the current text object reaches the first threshold. And / or the upload thread management module is also used to set the file migration status of the current text object to upload successful or upload failed; If the file migration status of the current text object is determined to be an upload failure, then the file stream of the current text object will be downloaded again until the file stream of the current text object is successfully uploaded, or the number of uploads of the file stream of the current text object reaches the second threshold. And / or the update thread management module is also used to set the file migration status of the current text object to update successful or update failed; if it is determined that the file migration status of the current text object is update failed, then the attribute information of the current text object is updated again in the migration file index table until the attribute information of the current text object is updated successfully, or the number of times the attribute information of the current text object is updated reaches the third threshold.

4. The file migration system according to claim 1, characterized in that, The attribute information also includes file verification status and / or the number of file verifications; The upload thread management module is further configured to obtain the upload file stream of the current text object based on the migration text identifier, compare the similarity between the file stream of the current text object and the upload file stream, perform migration verification on the current text object, and update the file verification status and / or file verification count of the current text object.

5. The file migration system according to claim 1, characterized in that, The query thread management module is also used to create a first preset number of text objects through query threads in the query thread pool; wherein the text objects are matched one-to-one with the files to be migrated.

6. The file migration system according to claim 1, characterized in that, The query thread management module is further configured to wake up the query thread in the query thread pool if it is determined that the number of text objects in the waiting download queue is less than or equal to a first quantity threshold and the number of text objects in the waiting update queue is less than or equal to a second quantity threshold. And / or the download thread management module is further configured to wake up the download threads in the download thread pool if it is determined that the number of text objects in the waiting upload queue is less than or equal to a third quantity threshold and the number of text objects in the waiting update queue is less than or equal to a fourth quantity threshold. And / or the upload thread management module is further configured to wake up the upload thread in the upload thread pool if it is determined that the number of text objects in the waiting update queue is less than or equal to a fifth quantity threshold.

7. The file migration system according to claim 1, characterized in that, The file migration system also includes: a migration task management module; The migration task management module is configured to, in response to receiving a migration task completion instruction, send a migration completion instruction to the query thread management module to close active query threads in the query thread pool; in response to detecting that there are no active query threads in the query thread pool and no text objects in the waiting download queue, send a migration completion instruction to the download thread management module to close active download threads in the download thread pool; in response to detecting that there are no active download threads in the download thread pool and no text objects in the waiting upload queue, send a migration completion instruction to the upload thread management module to close active upload threads in the upload thread pool; and in response to detecting that there are no active upload threads in the upload thread pool and no text objects in the waiting update queue, send a migration completion instruction to the update thread management module to close active update threads in the update thread pool.

8. A file migration method, characterized in that, include: The query thread management module uses query threads in the query thread pool to assign attribute information of the files to be migrated to text objects based on the original file index table, and then transmits the text objects with the assigned attribute information to the waiting download queue; wherein, the attribute information includes the original text identifier; The download thread management module uses download threads in the download thread pool to assign the corresponding file stream to the current text object based on the original text identifier of the current text object in the waiting download queue, and then transmits the current text object with the assigned file stream to the waiting upload queue. The upload thread management module uploads the file stream of the current text object in the waiting upload queue to the migration database through the upload thread in the upload thread pool, assigns the migration text identifier of the file stream to the current text object, and transmits the current text object with the assigned migration text identifier to the waiting update queue. The update thread management module updates the migration file index table based on the migration text identifier of the current text object in the waiting update queue by updating the update thread in the update thread pool.

9. A file migration device, characterized in that, include: The attribute information assignment module, configured in the query thread management module, is used to assign attribute information of the file to be migrated to a text object based on the original file index table through the query thread in the query thread pool, and to transmit the text object after attribute information assignment to the waiting download queue; wherein, the attribute information includes the original text identifier; The file stream assignment module is configured in the download thread management module. It is used to assign the corresponding file stream to the current text object based on the original text identifier of the current text object in the waiting download queue through the download thread in the download thread pool, and to transmit the current text object after the file stream assignment to the waiting upload queue. The migration file identifier assignment module is configured in the upload thread management module. It is used to upload the file stream of the current text object in the waiting upload queue to the migration database through the upload thread in the upload thread pool, assign the migration text identifier of the file stream to the current text object, and transmit the current text object with the migration text identifier assigned to it to the waiting update queue. The migration file index table update module, configured in the update thread management module, is used to update the migration file index table based on the migration text identifier of the current text object in the waiting update queue by using the update thread in the update thread pool.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to execute the file migration method of claim 8.

Images