Batch file migration method and device

Abstract

The present invention belongs to the field of big data technology. The present invention provides a method and device for migrating batch files. The method for migrating batch files includes: storing source files in a relational database; inserting the file list in the relational database into a distributed cache queue according to the global file database ID number and timestamp of the source file; generating a mapping table of the source file according to the distributed cache queue; and migrating the source file according to the mapping table. The present invention overcomes the stability problems, concurrent performance problems, and breakpoint batch resuming problems in the prior art, and fully realizes the efficiency and convenience of the redis distributed cache message queue.

Description

Technical Field

[0001] The present application belongs to the field of big data technology, and specifically relates to a method and device for migrating batch files. Background Art

[0002] With the rapid development of science and technology, the update and iteration time of Internet technology has been greatly shortened, and the upgrades of application systems with various functions have become more and more frequent. In the process of updating the system, the file data of the old system needs to be migrated to the new system. This process involves the following problems. Most traditional file migration systems adopt a monolithic system architecture. Because there was almost no distributed system framework technology at that time, the concurrency and performance level of traditional file migration systems were very limited. Secondly, the file migration system in the existing technology has low fault tolerance. If an error occurs during the migration process, most of them only do retries. In this way, once a problem occurs in batch migration, it is impossible to retry directly. Because the error point is not analyzed and recorded, it is very difficult to synchronize the data between the old and new systems after the error occurs, that is, it does not support the breakpoint resumption function. Finally, the existing file migration system is generally unable to integrate distributed components due to architectural design and other reasons. Summary of the invention

[0003] The present invention can be used in the technical field of big data technology applied in finance, and can also be used in any field other than the financial field. The present invention solves some pain points of the existing file migration system, including the concurrency, coupling, and scalability of file migration, and supports the breakpoint batch resumption function and the integration of distributed components.

[0004] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0005] In a first aspect, the present invention provides a method for migrating batch files, comprising:

[0006] Store source files in a relational database;

[0007] Inserting the file list in the relational database into the distributed cache queue according to the global file database ID number and timestamp of the source file;

[0008] Generate a mapping table of the source files according to the distributed cache queue;

[0009] The source file is migrated according to the mapping table.

[0010] In one embodiment, inserting the file list in the relational database into the distributed cache queue according to the global file database ID number and timestamp of the source file includes:

[0011] Sort the source files in the relational database according to the global file database ID number and timestamp;

[0012] Split the file list corresponding to the sorted source files;

[0013] Insert the split file list into the distributed cache redis queue.

[0014] In one embodiment, generating the mapping table of the source file according to the distributed cache queue includes:

[0015] A mapping table between a plurality of sub-files in the source file and the positions of the plurality of sub-files is generated according to the distributed cache redis queue.

[0016] In one embodiment, migrating the source file according to the mapping table includes:

[0017] Generate a producer queue according to the mapping table and the source file;

[0018] When the producer queue length is greater than 0, the producer queue is written into the consumer queue in a multi-threaded manner.

[0019] In one embodiment, the batch file migration method further includes:

[0020] When a source file migration breakpoint occurs, determining the subfile where the breakpoint occurs according to the breakpoint time and the mapping table;

[0021] The sub-files with breakpoints in the consumer queue are deleted, and the breakpoints are resumed according to the mapping table and the source file.

[0022] In a second aspect, the present invention provides a batch file migration device, the device comprising:

[0023] A source file storage module is used to store source files in a relational database;

[0024] A file list insertion module, used to insert the file list in the relational database into the distributed cache queue according to the global file database ID number and timestamp of the source file;

[0025] A mapping table generating module, used for generating a mapping table of the source file according to the distributed cache queue;

[0026] A source file migration module is used to migrate the source file according to the mapping table.

[0027] In one embodiment, the file list insertion module includes:

[0028] A source file sorting unit, used to sort the source files in the relational database according to the global file database ID number and timestamp;

[0029] A file list splitting unit, used for splitting the file list corresponding to the sorted source files;

[0030] A file list insertion unit, used to insert the split file list into the distributed cache redis queue;

[0031] In one embodiment, the mapping table generation module includes:

[0032] A mapping table generating unit is used to generate a mapping table between a plurality of sub-files in the source file and the positions of the plurality of sub-files according to the distributed cache redis queue.

[0033] In one embodiment, the source file migration module includes:

[0034] A producer queue generating unit, configured to generate a producer queue according to the mapping table and the source file;

[0035] A producer queue writing unit, used for writing the producer queue into the consumer queue in a multi-threaded manner when the producer queue length is greater than 0;

[0036] In one embodiment, the batch file migration device further includes:

[0037] A sub-file determination module, used for determining the sub-file where the breakpoint occurs according to the breakpoint time and the mapping table when a source file migration breakpoint occurs;

[0038] The source file resuming module is used to delete the sub-files with breakpoints in the consumer queue and to resume the breakpoint transfer according to the mapping table and the source file.

[0039] In a third aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a batch file migration method when executing the program.

[0040] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of a batch file migration method when executed by a processor.

[0041] From the above description, it can be seen that the embodiment of the present invention provides a method and device for migrating batch files. First, the source file is stored in a relational database; according to the global file database ID number and timestamp of the source file, the file list in the relational database is inserted into the distributed cache queue; then, a mapping table of the source file is generated according to the distributed cache queue; finally, the source file is migrated according to the mapping table. The present invention overcomes the stability problems, concurrency performance problems and breakpoint batch continuation problems in the prior art, and gives full play to the efficiency and convenience of the redis distributed cache message queue. Specifically, the present invention has the following beneficial effects: the present invention solves the concurrency, coupling, scalability and other problems of file migration encountered in the file migration process of the prior art, and supports the breakpoint batch continuation function, as well as the integration of distributed components. The goals of file migration are diverse, and can be migrated to the disk of a specified server or to the private cloud of OSS. There is no limit on the supported file types. Because it is a migration system that supports batches, the deployment cost is also very flexible, and the batch specifications and thread pool parameters of the system can be configured according to the actual budget cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 The process diagram of the batch file migration method in the embodiment of the present invention is as follows: Figure 1 ;

[0044] Figure 2 is a flow chart of step 200 in an embodiment of the present invention;

[0045] Figure 3 is a flow chart of step 300 in an embodiment of the present invention;

[0046] Figure 4 is a flow chart of step 400 in an embodiment of the present invention;

[0047] Figure 5 The process diagram of the batch file migration method in the embodiment of the present invention is as follows: Figure 2 ;

[0048] Figure 6 It is a flowchart of a method for migrating batch files in a specific implementation manner of the present invention;

[0049] Figure 7A mind map of a batch file migration method in a specific implementation manner of the present invention;

[0050] Figure 8 A block of a batch file migration device in an embodiment of the present invention Figure 1 ;

[0051] Fig. 9 is a block diagram of a file list inserting module 20 in an embodiment of the present invention;

[0052] Fig.10 is a block diagram of a mapping table generating module 30 in an embodiment of the present invention;

[0053] Fig.11 is a block diagram of a source file migration module 30 in an embodiment of the present invention;

[0054] Fig.12 A block of a batch file migration device in an embodiment of the present invention Figure 2 ;

[0055] Fig.13 It is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0058] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or apparatuses.

[0059] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0060] The embodiment of the present invention provides a specific implementation method of a batch file migration method, see Figure 1 , the method specifically includes the following contents:

[0061] Step 100: Store the source file in a relational database.

[0062] A relational database is a database that uses a relational model to organize data. It stores data in the form of rows and columns to facilitate user understanding. This series of rows and columns in a relational database is called a table, and a group of tables constitutes a database. Users retrieve data from the database through queries, and a query is an execution code used to limit certain areas in the database. The relational model can be simply understood as a two-dimensional table model, and a relational database is a data organization composed of two-dimensional tables and the relationships between them. Specifically, it has the following characteristics:

[0063] 1. Storage method: Traditional relational databases use tables for storage, where data is stored in rows and columns, making it very convenient to read and query.

[0064] 2. Storage structure: Relational databases store data in a structured manner. Each data table must have each field defined (that is, the table structure must be defined first), and then the data is stored according to the table structure. The advantage of this is that the form and content of the data have been defined before the data is stored, so the reliability and stability of the entire data table are relatively high. However, the problem is that once the data is stored, it will be very difficult to modify the structure of the data table if necessary.

[0065] 3. Storage specification: In order to avoid duplication, normalize data and make full use of storage space, relational databases store data in the form of minimum relational tables, so that data management can become very clear and easy to understand. Of course, this is mainly the case with one data table. If there are multiple tables, the situation is different. Since the data involves multiple data tables, there are complex relationships between the data tables. As the number of data tables increases, data management will become more and more complicated.

[0066] 4. Expansion method: Since relational databases store data in data tables, the bottleneck of data operations appears in the operations of multiple data tables, and the more data tables there are, the more serious this problem is. If you want to alleviate this problem, you can only improve the processing power, that is, choose a faster and higher-performance computer. Although this method can provide a certain amount of expansion space, such expansion space must be very limited, that is, relational databases only have the ability to expand vertically.

[0067] 5. Query method: Relational databases use structured query language (SQL) to query the database. SQL has long been supported by various database manufacturers and has become the standard of the database industry. It can support database CRUD (add, query, update, delete) operations and has very powerful functions. SQL can use index-like methods to speed up query operations.

[0068] 6. Normalization: In the process of database design and development, developers usually need to operate on one or more data entities (including arrays, lists and nested data) at the same time. In a relational database, a data entity must first be divided into multiple parts, and then the divided parts must be normalized. After normalization, they are stored in multiple relational data tables. This is a complicated process. The good news is that with the development of software technology, many software development platforms provide some simple solutions. For example, the ORM layer (that is, object-relational mapping) can be used to map the object model in the database to a SQL-based relational database and convert data between different types of systems.

[0069] 7. Transactionality: Relational databases emphasize ACID rules (Atomicity, Consistency, Isolation, Durability), which can meet data operations with high transactional requirements or complex data queries, and can fully meet the requirements of high performance and operational stability of database operations. In addition, relational databases emphasize strong consistency of data and provide good support for transaction operations. Relational databases can control the fine-grained atomicity of transactions, and can roll back transactions immediately if there is an error or need.

[0070] 8. Read and write performance: Relational databases place great emphasis on data consistency and pay a huge price for this by reducing read and write performance. Although relational databases have good reliability in storing and processing data, their efficiency becomes very poor when faced with processing massive amounts of data, especially when encountering high-concurrency read and write operations, the performance will drop significantly.

[0071] Step 200: inserting the file list in the relational database into the distributed cache queue according to the global file database ID number and timestamp of the source file.

[0072] Specifically, after the source files are stored in the relational database table, the file list in the database table is segmented and split according to the parameters in the configuration file such as sorting, batch specification, and inserted into the producer queue implemented by the distributed cache redis.

[0073] Step 300: Generate a mapping table of the source file according to the distributed cache queue.

[0074] It can be understood that, through the mapping table in step 300, a unified mapping relationship can be established between different positions in the source file and sub-files of different formats.

[0075] Step 400: Migrate the source file according to the mapping table.

[0076] Specifically, we first calculate the number of migrations and then the number of concurrent threads of the system program and the expected completion time. We then load the metadata of the batch file mapping table into the migration system through the configuration file, load and process it in batches (10,000 files per batch by default), and write it to the consumer queue implemented by redis in a multi-threaded manner.

[0077] From the above description, it can be seen that the embodiment of the present invention provides a method for migrating batch files, firstly storing the source files in a relational database; inserting the file list in the relational database into the distributed cache queue according to the global file database ID number and timestamp of the source files; then generating a mapping table of the source files according to the distributed cache queue; finally migrating the source files according to the mapping table. The present invention overcomes the stability problems, concurrent performance problems and breakpoint batch resuming problems in the prior art, and gives full play to the efficiency and convenience of the redis distributed cache message queue.

[0078] In one embodiment, see Figure 2 , step 200 further comprises:

[0079] Step 201: Sort the source files in the relational database according to the global file database ID number and timestamp;

[0080] Customize sorting of all source files based on global file database ID number, time and other parameters

[0081] Step 202: splitting the file list corresponding to the sorted source files;

[0082] After sorting, the file list in the database table is segmented and split according to the batch specifications and other parameters in the configuration file.

[0083] Step 203: insert the split file list into the distributed cache redis queue.

[0084] In one embodiment, see Figure 3 , step 300 further comprises:

[0085] Step 301: Generate a mapping table between a plurality of sub-files in the source file and the positions of the plurality of sub-files according to the distributed cache redis queue.

[0086] Redis supports a total of 5 data types, each of which corresponds to a different data structure, including simple String type, compressed string, dictionary, skip table, etc. The skip table is a relatively new data structure, often used for high-performance search, which can reach a query speed of log2N. Moreover, compared with the red-black tree, the skip table has fewer nodes changed during update, making it easier to implement concurrent operations.

[0087] Redis is implemented using a single thread, so no content that is too large will be stored in the distributed cache redis queue, otherwise it will block other requests.

[0088] Because cache operations are all memory operations with only a few computing operations, the performance is very good under a single thread. The single-threaded non-blocking network I / O model implemented by Redis is suitable for fast operation logic, but will affect performance when there is complex and long logic. For long logic, multiple instances should be configured to improve the utilization of multi-core CPUs, that is, multiple instances can be configured using a single machine with multiple ports.

[0089] In one embodiment, see Figure 4 , step 400 further comprises:

[0090] Step 401: Generate a producer queue according to the mapping table and the source file;

[0091] Specifically, the file list is scanned and collected through the file mapping system and stored in a relational database, and then the file list in the database is segmented and inserted into the redis queue of the distributed cache component through a segmentation algorithm. At this time, the queue is called a producer queue. In this way, a unified mapping table is established for files in different locations and formats, which is convenient for subsequent analysis, transmission and processing.

[0092] Step 402: When the length of the producer queue is greater than 0, the producer queue is written into the consumer queue in a multi-threaded manner.

[0093] Specifically, after integrating the redis producer queue, the scheduled scanning queue event will scan whether the producer queue length is greater than 0. If so, it will start receiving producer queue data and transmitting segmented file metadata to the batch file transfer system. The transmission system is implemented by the springboot scheduled task. The configuration file has been loaded at startup, and the transmission type (that is, the migration type, such as server to server type, server to OSS type, etc.) is obtained. After the target server data, the file metadata is sent to the enabled consumer task queue. The consumer queue is a concurrent task queue based on the thread pool.

[0094] In one embodiment, see Figure 5 , the batch file migration method also includes:

[0095] Step 500: When a source file migration breakpoint occurs, the sub-file where the breakpoint occurs is determined according to the breakpoint time and the mapping table;

[0096] Step 600: Delete the sub-file with the breakpoint in the consumer queue, and perform breakpoint resume according to the mapping table and the source file.

[0097] In step 500 and step 600, if any migration file fails during batch file migration, it will be recorded in the breakpoint file metadata list database, and the batch breakpoint resumption function will start after the batch breakpoint timing scanning event is started. Specifically, the sub-file where the breakpoint occurs in the source file is first determined, and then the location of the sub-file is determined according to the mapping table, and the sub-file where the breakpoint occurs is deleted, and finally the source file is resumed.

[0098] In a specific implementation, the present invention provides a specific implementation in a method for migrating batch files.

[0099] A batch file migration system is also provided herein, the system specifically comprising: a batch file mapper and a batch file transmitter, specifically:

[0100] Batch file mapper: The source file server stores batch files to be migrated. Before the batch files are migrated, they need to be scanned by the system of the present invention. The file name, size, and other file metadata are mapped to the database table, and the mapping timestamp is recorded to facilitate subsequent sorting and other operations. After being stored in the database table, the file list in the database table is segmented and split after sorting, batch specifications, and other parameters in the configuration file, and inserted into the producer queue implemented by the distributed cache redis. This process workflow is the batch file mapper.

[0101] Batch file transfer: The migration system first periodically scans the length of the redis producer queue. If the length is greater than 0, it starts to traverse the queue, takes out the list of file segments that have been segmented according to the batch specification, organizes them one by one into thread tasks, and sends them to the consumer queue implemented by the thread pool, and starts batch migration of files. During the migration process, the migration is carried out according to the configured target server type parameters, and finally the file migration is completed. This process task is the batch file transfer.

[0102] See also Figure 6 Based on the above batch file migration system, the batch file migration method provided in this specific implementation includes the following steps:

[0103] Based on the technical defects in the prior art, the current scenario requires a new file migration method, which must be stable and fault-tolerant, and must also consider performance and support the scalability of distributed components. The most important thing is to support the function of batch resuming breakpoints. Therefore, the present invention studies and designs a batch file migration system based on these problems. Starting from the file itself as metadata, the present invention supports migration regardless of whether the file is in the server disk or in a distributed database. The file list is scanned and collected by the file mapping system and stored in a relational database. The file list in the database is then segmented and inserted into the redis queue of the distributed cache component through a segmentation algorithm. At this time, the queue is called a producer queue. In this way, a unified mapping table is established for files in different locations and formats, which is convenient for subsequent analysis, transmission and processing. The producer queue is then analyzed and processed by a batch file transmitter, and the file migration task is performed in a multi-threaded concurrent manner, which greatly improves performance. In addition, all frameworks of the entire system are basically implemented by the springboot framework, so it supports seamless integration of distributed components, and finally solves the problems in the file migration process. Specifically:

[0104] S1: Generate a mapping table corresponding to the source file.

[0105] See also Figure 7, the batch file mapper will load the configuration file when it starts, and after starting, it will start scanning the source files and integrate all the files in the source file system into the producer message queue through file metadata collection, sorting, storage and other methods. This part of the method and program is the batch file mapper system. The file mapper and the file transfer both support configurable functions, that is, the batch specification and other parameters can be set, and the collected file list is extracted and integrated into the redis queue through the batch specification and other parameters. The implementation method is: all files are customized according to the global file database ID number and time and other parameters, and parameter settings are supported according to the configuration file of the batch migration system, including the number of batches, whether to open breakpoint resume, and thread pool parameter configuration. The algorithm is automatically calculated according to the parameter configuration in the configuration file to obtain the number of migrations required in the end, and then the number of concurrent threads of the system program and the expected completion time are automatically calculated. The metadata of the batch file mapping table is loaded into the migration system through the configuration file, and loaded and processed in batches (the default is 10,000 files per batch), and written to the consumer queue implemented by redis in a multi-threaded manner.

[0106] S2: Migrate the source files according to the mapping table.

[0107] After integrating the redis producer queue, the scheduled scanning queue event will scan that the producer queue length is greater than 0, and start receiving producer queue data, and transmit the segmented file metadata to the batch file transfer system. The transmission system is implemented by the springboot scheduled task. The configuration file has been loaded at startup, and the transmission type (that is, the type of this migration, such as server to server type, server to OSS type, etc.) is obtained. After the target server data, the file metadata is sent to the enabled consumer task queue. The consumer queue is a task concurrent task queue based on the thread pool. The default thread pool parameters are 10 core threads and 13 maximum threads. The configuration file in the system can modify the parameters. The consumer task queue is based on the thread pool, so the migration file task can be executed in parallel. The idea is that different file transfer threads will be started for each file transfer according to the transmission type. Each thread is responsible for transferring a file. At this time, the file migration task starts. If any migration file fails during the period, it will be recorded in the breakpoint file metadata list database, and wait for the batch breakpoint scheduled scanning event to start before starting the batch breakpoint resume function. The process of batch breakpoint resume is the same as the principle of the producer queue and the consumer queue, so I will not repeat it here.

[0108] From the above description, it can be seen that the embodiment of the present invention provides a method for migrating batch files, firstly storing the source file in a relational database; inserting the file list in the relational database into the distributed cache queue according to the global file database ID number and timestamp of the source file; then, generating a mapping table of the source file according to the distributed cache queue; finally, migrating the source file according to the mapping table. The present invention overcomes the stability problems, concurrency performance problems and breakpoint batch resume problems in the prior art, and gives full play to the efficiency and convenience of the redis distributed cache message queue. Specifically, the present invention has the following beneficial effects: the present invention solves the concurrency, coupling, scalability and other problems of file migration encountered in the file migration process of the prior art, and supports the breakpoint batch resume function, as well as supports the integration of distributed components. The goals of file migration are diversified, and can be migrated to the disk of a specified server or to the private cloud of OSS. There is no limit on the supported file types. Because it is a migration system that supports batches, the deployment cost is also very flexible, and the batch specifications and thread pool parameters of the system can be configured according to the actual budget cost.

[0109] Based on the same inventive concept, the embodiments of the present application also provide a migration device for batch files, which can be used to implement the method described in the above embodiments, such as the following embodiments. Since the principle of solving the problem by the migration device for batch files is similar to that of the migration method for batch files, the implementation of the migration device for batch files can refer to the implementation of the migration method for batch files, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0110] The embodiment of the present invention provides a specific implementation of a batch file migration device capable of implementing a batch file migration method, see Figure 8 The batch file migration device specifically includes the following contents:

[0111] The source file storage module 10 is used to store the source file in a relational database;

[0112] A file list inserting module 20, for inserting the file list in the relational database into the distributed cache queue according to the global file database ID number and timestamp of the source file;

[0113] A mapping table generating module 30, configured to generate a mapping table of the source file according to the distributed cache queue;

[0114] The source file migration module 40 is used to migrate the source file according to the mapping table.

[0115] In one embodiment, see Fig. 9 , the file list inserting module 20 comprises:

[0116] A source file sorting unit 201 is used to sort the source files in the relational database according to the global file database ID number and timestamp;

[0117] A file list splitting unit 202, used to split the file list corresponding to the sorted source files;

[0118] The file list inserting unit 203 is used to insert the split file list into the distributed cache redis queue;

[0119] In one embodiment, see Fig.10 , the mapping table generating module 30 comprises:

[0120] The mapping table generating unit 301 is used to generate a mapping table between a plurality of sub-files in the source file and the positions of the plurality of sub-files according to the distributed cache redis queue.

[0121] In one embodiment, see Fig.11 , the source file migration module 40 includes:

[0122] A producer queue generating unit 401, configured to generate a producer queue according to the mapping table and the source file;

[0123] The producer queue writing unit 402 is used to write the producer queue into the consumer queue in a multi-threaded manner when the producer queue length is greater than 0;

[0124] In one embodiment, see Fig.12 , the batch file migration device also includes:

[0125] A sub-file determination module 50, for determining the sub-file where the breakpoint occurs according to the breakpoint time and the mapping table when a source file migration breakpoint occurs;

[0126] The source file resuming module 60 is used to delete the sub-files with breakpoints in the consumer queue and perform breakpoint resuming according to the mapping table and the source file.

[0127] From the above description, it can be seen that the embodiment of the present invention provides a batch file migration device, firstly, the source file is stored in a relational database; according to the global file database ID number and timestamp of the source file, the file list in the relational database is inserted into the distributed cache queue; then, a mapping table of the source file is generated according to the distributed cache queue; finally, the source file is migrated according to the mapping table. The present invention overcomes the stability problem, concurrency performance problem and breakpoint batch resume problem in the prior art, and gives full play to the efficiency and convenience of the redis distributed cache message queue. Specifically, the present invention has the following beneficial effects: the present invention solves the concurrency, coupling, scalability and other problems of file migration encountered in the file migration process of the prior art, and supports the breakpoint batch resume function, as well as supports distributed component integration. The target of file migration is diversified, and it can be migrated to the disk of a specified server or to the private cloud of OSS. There is no limit on the supported file types. Because it is a migration system that supports batches, the deployment cost is also very flexible, and the batch specifications and thread pool parameters of the system can be configured according to the actual budget cost.

[0128] The embodiment of the present application also provides a specific implementation of an electronic device capable of implementing all steps in the batch file migration method in the above embodiment, see Fig.13 , electronic equipment specifically includes the following:

[0129] Processor (processor) 1201, memory (memory) 1202, communication interface (CommunicationsInterface) 1203 and bus 1204;

[0130] The processor 1201, the memory 1202, and the communication interface 1203 communicate with each other through the bus 1204; the communication interface 1203 is used to realize information transmission between the server device and the client device and other related devices;

[0131] The processor 1201 is used to call the computer program in the memory 1202. When the processor executes the computer program, all the steps in the batch file migration method in the above embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0132] Step 100: Storing the source file in a relational database;

[0133] Step 200: inserting the file list in the relational database into the distributed cache queue according to the global file database ID number and timestamp of the source file;

[0134] Step 300: Generate a mapping table of the source file according to the distributed cache queue;

[0135] Step 400: Migrate the source file according to the mapping table.

[0136] The embodiment of the present application also provides a computer-readable storage medium capable of implementing all the steps in the batch file migration method in the above embodiment. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, all the steps in the batch file migration method in the above embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0137] Step 100: Storing the source file in a relational database;

[0138] Step 200: inserting the file list in the relational database into the distributed cache queue according to the global file database ID number and timestamp of the source file;

[0139] Step 300: Generate a mapping table of the source file according to the distributed cache queue;

[0140] Step 400: Migrate the source file according to the mapping table.

[0141] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0142] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0143] Although the present application provides method operation steps such as embodiments or flow charts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0144] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing the embodiments of this specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0145] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0146] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0147] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0148] The present specification embodiments may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present specification embodiments may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0149] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, in the absence of contradiction, a person skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0150] The above is only an example of the embodiment of the present specification and is not intended to limit the embodiment of the present specification. For those skilled in the art, the embodiment of the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiment of the present specification shall be included in the scope of the claims of the embodiment of the present specification.

Claims

1. A method for migrating batch files, characterized in that: include: Store source files in a relational database; Inserting the file list in the relational database into the distributed cache queue according to the global file database ID number and timestamp of the source file; Generate a mapping table of the source file according to the distributed cache queue; the mapping table is used to establish a unified mapping relationship between sub-files at different positions and in different formats in the source file; Migrating the source file according to the mapping table; Migrating the source file according to the mapping table includes: First, the number of migrations is calculated, and then the number of concurrent threads of the system program and the expected completion time are calculated. The metadata of the batch file mapping table is loaded into the migration system through the configuration file. The batch loading process is carried out according to the batch loading of 10,000 files, and the files are written to the consumer queue implemented by redis in a multi-threaded manner.

2. The method for migrating batch files according to claim 1, characterized in that: The method of inserting the file list in the relational database into the distributed cache queue according to the global file database ID number and timestamp of the source file comprises: Sort the source files in the relational database according to the global file database ID number and timestamp; Split the file list corresponding to the sorted source files; Insert the split file list into the distributed cache redis queue.

3. The method for migrating batch files as claimed in claim 2, characterized in that: Generating the mapping table of the source file according to the distributed cache queue includes: A mapping table between a plurality of sub-files in the source file and the positions of the plurality of sub-files is generated according to the distributed cache redis queue.

4. The method for migrating batch files as claimed in claim 3, characterized in that: Migrating the source file according to the mapping table includes: Generate a producer queue according to the mapping table and the source file; When the producer queue length is greater than 0, the producer queue is written into the consumer queue in a multi-threaded manner.

5. The method for migrating batch files as claimed in claim 4, characterized in that: Also includes: When a source file migration breakpoint occurs, determining the subfile where the breakpoint occurs according to the breakpoint time and the mapping table; The sub-files with breakpoints in the consumer queue are deleted, and the breakpoints are resumed according to the mapping table and the source file.

6. A batch file migration device, characterized in that: include: A source file storage module is used to store source files in a relational database; A file list insertion module, used to insert the file list in the relational database into the distributed cache queue according to the global file database ID number and timestamp of the source file; A mapping table generation module, used to generate a mapping table of the source file according to the distributed cache queue; the mapping table is used to establish a unified mapping relationship between sub-files at different positions and in different formats in the source file; A source file migration module, used for migrating the source file according to the mapping table; According to the source file migration module, it is specifically used to first calculate the number of migrations and then calculate the thread concurrency of the system program and the expected completion time, load the batch file mapping table metadata into the migration system through the configuration file, and process the batch loading of 10,000 files per batch, and write them to the consumer queue implemented by redis in a multi-threaded manner.

7. The batch file migration device according to claim 6, characterized in that: The file list insertion module includes: A source file sorting unit, used to sort the source files in the relational database according to the global file database ID number and timestamp; A file list splitting unit, used for splitting the file list corresponding to the sorted source files; A file list insertion unit, used to insert the split file list into the distributed cache redis queue; The mapping table generation module comprises: A mapping table generating unit is used to generate a mapping table between a plurality of sub-files in the source file and the positions of the plurality of sub-files according to the distributed cache redis queue.

8. The batch file migration device according to claim 7, characterized in that: The source file migration module includes: A producer queue generating unit, configured to generate a producer queue according to the mapping table and the source file; A producer queue writing unit, used for writing the producer queue into the consumer queue in a multi-threaded manner when the producer queue length is greater than 0; The batch file migration device also includes: A sub-file determination module, used for determining the sub-file where the breakpoint occurs according to the breakpoint time and the mapping table when a source file migration breakpoint occurs; The source file resuming module is used to delete the sub-files with breakpoints in the consumer queue and to resume the breakpoint transfer according to the mapping table and the source file.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the batch file migration method according to any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the batch file migration method described in any one of claims 1 to 5 are implemented.

Images