File migration method and device and electronic equipment

Abstract

The invention discloses a file migration method and device and electronic equipment, and relates to the technical field of big data or the field of financial science and technology, and the method comprises the steps: determining the file type of each small file based on the file access records of L small files in a financial system; combining the small files with the same file type in the L small files to obtain a large file and a file mapping table corresponding to the M small files of the cold data type, and a virtual unit and a unit descriptor corresponding to the N small files of the hot data type; and based on the target concurrency and priority mechanism, carrying out file migration on the M small files through the large file and the file mapping table, and carrying out file migration on the N small files through the virtual unit and the unit descriptor. According to the file migration method and device, the technical problem that in the prior art, the file migration efficiency is low due to the fact that the TCP connection overhead proportion is high during file migration is solved.

Description

Technical Field

[0001] This application relates to the fields of big data technology or fintech, and more specifically, to a file migration method, apparatus, and electronic device. Background Technology

[0002] With the popularization of big data technology, distributed file systems need to frequently perform cross-cluster and cross-regional data migration. In actual business, small files (such as logs, images and sensor data) account for more than 80% of the distributed file systems.

[0003] In existing technologies, when migrating a large number of small files (≤16MB) using mainstream tools, a single-file-by-file processing approach is typically adopted. This approach has the following bottlenecks:

[0004] 1. Metadata service overload: High-frequency List Status (an API (Application Programming Interface) for listing files and directories in a directory, which returns a list containing file status, including filename, size, permissions, and other metadata) and file operation RPC (Remote Procedure Call) requests can overwhelm the NameNode (the primary data node, responsible for managing the file system namespace and client access metadata for files), causing service degradation or even Full GC (Full Garbage Collection, a garbage collection process performed by the Java Virtual Machine).

[0005] 2. Inefficient network transmission: When small files are transmitted independently, the TCP (Transmission Control Protocol) connection overhead accounts for a large proportion, resulting in a low effective transmission rate. Furthermore, the fixed concurrency strategy causes bandwidth contention between large and small files.

[0006] It is evident that traditional solutions cannot simultaneously satisfy metadata decompression and transmission optimization. Existing technologies perform file migration independently for each small file, resulting in metadata service overload and high TCP connection overhead, thus leading to low file migration efficiency.

[0007] There is currently no effective solution to the above problems. Summary of the Invention

[0008] This application provides a file migration method, apparatus, and electronic device to at least solve the technical problem of low file migration efficiency caused by the high proportion of TCP connection overhead in the prior art during file migration.

[0009] According to one aspect of this application, a file migration method is provided, comprising: determining the file type of each small file based on file access records of L small files in a financial system, wherein L is a positive integer, each small file is a file to be migrated whose file size is less than or equal to a preset size threshold, and the file type is either a cold data type or a hot data type; merging small files of the same file type among the L small files to obtain a large file and a file mapping table corresponding to M small files of the cold data type, and obtaining virtual units and unit descriptors corresponding to N small files of the hot data type, wherein M and N are both positive integers, and L is equal to the sum of N and M. The file mapping table includes at least the storage location identifiers of the file contents of M small files in the large file and the check codes of the M small files. The unit descriptor includes at least the unit identifier of the virtual unit and the file information of each small file corresponding to the virtual unit. Based on the target concurrency and priority mechanism, the M small files are migrated through the large file and the file mapping table, and the N small files are migrated through the virtual units and unit descriptors. The target concurrency is the minimum value among the maximum number of threads, the bandwidth factor, and the QPS factor of the file migration system. The QPS factor is used to characterize the processing efficiency of the file migration system for file query requests.

[0010] Optionally, the step of determining the file type of each small file based on the file access records of L small files in the financial system includes: determining the access popularity coefficient and access frequency coefficient of each small file based on the access records of each small file, wherein the access popularity is used to characterize the number of times the small file is accessed within the time window corresponding to the first access time to the most recent access time, and the access frequency is used to characterize the access frequency of the small file; if the access popularity coefficient is less than or equal to a preset popularity coefficient threshold, or if the access frequency coefficient is greater than a preset frequency coefficient threshold, the file type of the small file is determined to be a cold data type; if the access popularity coefficient is greater than the preset popularity coefficient threshold and the access frequency coefficient is less than or equal to the preset frequency coefficient threshold, the file type of the small file is determined to be a hot data type.

[0011] Optionally, the step of merging small files of the same file type from L small files to obtain a large file and a file mapping table corresponding to M small files of cold data type includes: scanning the storage directory of the small files of cold data type in the L small files to obtain M small files to be merged; sequentially reading the file content of the M small files and storing the sequentially read file content uniformly into an empty file of preset format to obtain a large file corresponding to the M small files; using the starting offset and storage length of the file content corresponding to each small file in the large file as the storage location identifier of the small file, and creating a check code corresponding to each small file in the M small files according to a preset algorithm; and generating a file mapping table corresponding to the M small files based on the storage location identifier and check code corresponding to each small file in the M small files.

[0012] Optionally, the step of merging small files of the same file type from L small files to obtain virtual units and unit descriptors corresponding to N small files of hot data type includes: logically merging small files of hot data type from L small files and belonging to the same storage directory to obtain X virtual units corresponding to N small files, where X is a positive integer and the unit size of each virtual unit is less than or equal to a preset unit size; determining the total data size of all small files corresponding to each virtual unit based on the sum of the data sizes of each small file corresponding to each virtual unit, and creating a checksum corresponding to each small file in the N small files according to a preset algorithm; and using the unit identifier, file list, total data size of all small files, and checksum of all small files corresponding to each virtual unit as the unit descriptor corresponding to the virtual unit.

[0013] Optionally, before migrating M small files using large files and file mapping tables based on the target concurrency and priority mechanism, and migrating N small files using virtual units and unit descriptors, the file migration method further includes: collecting performance metrics of the file migration system, wherein the performance metrics include at least the real-time bandwidth of the file migration system, the average data size of all small files migrated within a preset time period, the QPS upper limit, and the real-time QPS; using the product of the real-time bandwidth and the preset bandwidth weight as the bandwidth factor of the file migration system; using the product of the difference between the QPS upper limit and the real-time QPS and the preset QPS weight as the QPS factor of the file migration system; and using the minimum value among the maximum number of threads, the bandwidth factor, and the QPS factor of the file migration system as the target concurrency of the file migration system.

[0014] Optionally, the steps for migrating M small files using a large file and a file mapping table include: transferring the large file and file mapping table corresponding to the M small files to the target storage system, and deleting the M small files from the source storage system; after the target storage system receives the large file and file mapping table, parsing the file mapping table to obtain the storage location identifier and checksum of each of the M small files; extracting the file content from the large file based on the storage location identifier of each small file to obtain the file content of each small file; verifying the file content of each small file based on the checksum of each small file; and determining that the migration of the M small files is successful if the file content of each small file passes the verification.

[0015] Optionally, the steps of migrating N small files using virtual units and unit descriptors include: allocating a memory cache for each virtual unit based on its unit descriptor, wherein the size of the memory cache is equal to the total data size of all small files included in the unit descriptor; sequentially storing the file contents of all small files corresponding to each virtual unit into the content cache corresponding to that virtual unit to obtain a data block corresponding to each virtual unit; packaging the data blocks corresponding to each virtual unit to obtain X data packets corresponding to X virtual units; and migrating the N small files using the X data packets and X unit descriptors corresponding to the X virtual units.

[0016] Optionally, the step of migrating N small files using X data packets and X unit descriptors corresponding to X virtual units includes: transmitting the X data packets and X unit descriptors corresponding to the N small files to the target storage system; after the target storage system receives the X data packets and X unit descriptors, atomically splitting the data packets corresponding to the i-th unit descriptor based on the file information of each small file included in the i-th unit descriptor to obtain the file content of each small file corresponding to the i-th unit descriptor; verifying the file content of each small file corresponding to the i-th unit descriptor based on the checksum in the file information of each small file included in the i-th unit descriptor; and determining that the migration of the N small files is successful if the file content of each small file passes the verification.

[0017] According to another aspect of this application, a file migration apparatus is also provided, comprising: a type determination unit, configured to determine the file type of each small file based on file access records of L small files in a financial system, wherein L is a positive integer, each small file is a file to be migrated whose file size is less than or equal to a preset size threshold, and the file type is either a cold data type or a hot data type; and a merging unit, configured to merge small files of the same file type among the L small files to obtain a large file and a file mapping table corresponding to M small files of the cold data type, and to obtain a virtual unit and a unit descriptor corresponding to N small files of the hot data type, wherein M and N are both positive integers, and L is equal to N and N. The sum of M, the file mapping table includes at least the storage location identifiers of the file contents of M small files in the large file and the check codes of M small files, and the unit descriptor includes at least the unit identifier of the virtual unit and the file information of each small file corresponding to the virtual unit; the migration unit is used to migrate M small files through the large file and the file mapping table based on the target concurrency and priority mechanism, and to migrate N small files through the virtual unit and the unit descriptor. The target concurrency is the minimum value among the maximum number of threads of the file migration system, the bandwidth factor and the QPS factor. The QPS factor is used to characterize the processing efficiency of the file migration system for file query requests.

[0018] According to another aspect of this application, a computer program product is also provided, which stores a computer program, wherein, when the computer program is running, it controls the computer program product to execute any of the above-mentioned file migration methods.

[0019] According to another aspect of this application, an electronic device is also provided, wherein the electronic device includes one or more processors and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the file migration method described above.

[0020] In this application, firstly, based on the file access records of L small files in the financial system, the file type of each small file is determined, where L is a positive integer, and each small file is a file to be migrated whose file size is less than or equal to a preset size threshold, and whose file type is either a cold data type or a hot data type. Then, this application merges the small files of the same file type among the L small files to obtain the large files and file mapping tables corresponding to the M small files of the cold data type, and obtains the virtual units and unit descriptors corresponding to the N small files of the hot data type, where M and N are both positive integers, L is equal to the sum of N and M, and the file mapping table includes at least M... The file content of each small file is stored in the large file, and the checksums of the M small files are also included. The unit descriptor includes at least the unit identifier of the virtual unit and the file information of each small file corresponding to the virtual unit. Then, based on the target concurrency and priority mechanism, this application performs file migration on the M small files through the large file and the file mapping table, and performs file migration on the N small files through the virtual unit and the unit descriptor. The target concurrency is the minimum value among the maximum number of threads, the bandwidth factor, and the QPS factor of the file migration system. The QPS factor is used to characterize the processing efficiency of the file migration system for file query requests.

[0021] As can be seen from the above, this application adopts a cold and hot data classification and dual-track merging approach, and by constructing a file mapping table and virtual transmission units, it achieves the goal of reducing the pressure on metadata services and optimizing network transmission efficiency, thereby realizing high-performance and low-latency technical effects for file migration between big data clusters, and thus solving the technical problem of low file migration efficiency caused by high-frequency metadata requests and inefficient TCP connection overhead.

[0022] Specifically, this application addresses the characteristics of massive small files in financial systems by first classifying them into cold or hot data based on file access records. Then, it employs either physical merging (generating large files and file mapping tables) or logical merging (constructing virtual units and unit descriptors) strategies for these two types of files respectively. Furthermore, this application dynamically calculates the target concurrency to ensure reasonable resource allocation. An adaptive scheduling mechanism prioritizes the migration of virtual units and large files, effectively alleviating the pressure on metadata services while improving network efficiency for file migration. This avoids bandwidth contention between large and small files, ensuring smooth data migration and thus solving the technical problem of low file migration efficiency caused by high TCP connection overhead in existing technologies. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0024] Figure 1 This is a hardware structure block diagram of an optional computer terminal (or mobile device) for implementing a file migration method according to an embodiment of this application;

[0025] Figure 2 This is a flowchart of an optional file migration method according to an embodiment of this application;

[0026] Figure 3 This is a structural diagram of an optional file migration system according to an embodiment of this application;

[0027] Figure 4 This is a flowchart of an optional intelligent hot and cold data recognition engine according to an embodiment of this application;

[0028] Figure 5 This is a flowchart of an optional dual-track merging process for hot and cold data according to an embodiment of this application;

[0029] Figure 6 This is a timing flowchart of an optional cold data merging process according to an embodiment of this application;

[0030] Figure 7 This is a timing flowchart of an optional hot data merging process according to an embodiment of this application;

[0031] Figure 8 This is a flowchart of an optional adaptive parallel scheduling process according to an embodiment of this application;

[0032] Figure 9 This is a flowchart of an optional priority allocation process according to an embodiment of this application;

[0033] Figure 10 This is a schematic diagram of an optional file migration apparatus according to an embodiment of this application;

[0034] Figure 11 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] It should also be noted that all information and data (including but not limited to information used for display and analysis) and data (including but not limited to file access records of small files and file data stored in small files) involved in this application are information and data authorized by the user or fully authorized by all parties. For example, if there is an interface between this system and the relevant user or organization, before obtaining the relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain the relevant information after receiving the consent information from the aforementioned user or organization.

[0038] Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of relevant information and data involved in this application all comply with the relevant laws, regulations, and standards of the relevant regions, and necessary confidentiality measures have been taken. This application does not violate public order and good morals. In addition, this application provides a corresponding operation entry point for users to choose to agree to or refuse authorization. If the user chooses to refuse authorization, the corresponding expert decision-making process will be initiated.

[0039] This application provides a method for migrating massive small files based on a cold and hot dual-track merging strategy combined with adaptive intelligent scheduling, which can effectively solve the problems of metadata server overload and inefficient network transmission encountered during the transmission of massive small files on big data platforms.

[0040] The present invention will now be described in detail with reference to various embodiments.

[0041] Example 1

[0042] According to an embodiment of this application, an embodiment of a file migration method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0043] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 This is a hardware structure block diagram of an optional computer terminal (or mobile device) for implementing a file migration method according to an embodiment of this application. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0044] Under the aforementioned operating environment, this application provides a file migration system for executing the file migration method described in this application. Figure 2 This is a flowchart of an optional file migration method according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0045] Step S201: Based on the file access records of L small files in the financial system, determine the file type of each small file, where L is a positive integer, each small file is a file to be migrated whose file size is less than or equal to a preset size threshold, and the file type is either a cold data type or a hot data type.

[0046] Optionally, the L small files are small files that need to be migrated in the financial system. The size of the small files is less than or equal to a preset size threshold, such as 16MB.

[0047] Optionally, the file migration system can verify file access records for small files by associating with NameNode audit logs.

[0048] Optionally, cold data types refer to small files that are accessed infrequently and with long access intervals. These files are suitable for physical merging to reduce the overhead of metadata operations and network transmission.

[0049] Optionally, hot data types refer to small files that are accessed frequently and with short access intervals. These files are not suitable for physical merging and should be transmitted using logical aggregation to maintain their original access patterns.

[0050] In step S201, the file migration system accurately distinguishes between hot and cold data through intelligent analysis of file access records, and implements targeted data processing strategies. This effectively avoids over-processing of cold data and improper merging of hot data, thereby improving the transmission efficiency and resource utilization of the entire file migration system.

[0051] Step S202: Merge the small files of the same file type among the L small files to obtain the large file and file mapping table corresponding to the M small files of cold data type, and obtain the virtual unit and unit descriptor corresponding to the N small files of hot data type. Here, M and N are both positive integers, L is equal to the sum of N and M, the file mapping table includes at least the storage location identifier of the file content of the M small files in the large file and the check code of the M small files, and the unit descriptor includes at least the unit identifier of the virtual unit and the file information of each small file corresponding to the virtual unit.

[0052] Optionally, a file mapping table records the storage location identifiers (such as starting offsets) and check codes (such as CRC32 (Cyclic Redundancy Check 32-bit) codes) of M small files within a large file, used for file content recovery and integrity verification at the target end.

[0053] Optionally, a Virtual Transmission Unit (VTU) logically combines N small files into one unit for batch transmission of hot data, reducing network overhead.

[0054] Optionally, the unit descriptor contains the identifier of the virtual unit and the metadata of each small file, such as the file path and size, to guide the file splitting and writing on the target end.

[0055] In step S202, the file migration system reduces frequent requests to metadata services by using a dual-track processing method for both cold and hot data. At the same time, it optimizes network transmission efficiency and avoids data access delays that may be caused by the physical merging of hot data, thus achieving the dual goals of reducing cold data pressure and efficiently transmitting hot data.

[0056] Step S203: Based on the target concurrency and priority mechanism, M small files are migrated through large files and file mapping tables, and N small files are migrated through virtual units and unit descriptors. The target concurrency is the minimum value among the maximum number of threads, bandwidth factor and QPS factor of the file migration system. The QPS factor is used to characterize the processing efficiency of the file migration system for file query requests.

[0057] Optionally, the target concurrency, i.e. the number of concurrent executions of data transmission tasks, is the core parameter of the adaptive scheduling strategy, ensuring stable system operation while maximizing transmission efficiency.

[0058] Optionally, the maximum number of threads, the maximum number of threads that the file migration system can run simultaneously, reflects the system's hardware processing capabilities.

[0059] Optionally, the bandwidth factor, a parameter calculated based on the currently available bandwidth and average file size, is used to dynamically adjust the concurrency of transmission tasks to ensure efficient utilization of network resources.

[0060] Optionally, the QPS (Queries Per Second) factor, a parameter calculated based on the NameNode's current QPS and QPS limit, is used to limit the system's concurrency and prevent the NameNode from becoming overloaded.

[0061] Optionally, a priority mechanism ensures that virtual units corresponding to hot data are transmitted first, followed by large files composed of small files corresponding to cold data, and finally unaggregated small files, so as to ensure efficient and stable data migration.

[0062] In step S203, the file migration system can dynamically adjust the concurrency of transmission tasks through an adaptive parallel scheduling strategy, balance the system load, and prevent resource overload. At the same time, the priority mechanism ensures that important data (hot data) is transmitted first and fast, which improves the response speed of the entire system to sudden data migration needs and achieves optimal resource allocation and high efficiency and stability of data transmission.

[0063] As can be seen from the above, this application adopts a cold and hot data classification and dual-track merging approach, and by constructing a file mapping table and virtual transmission units, it achieves the goal of reducing the pressure on metadata services and optimizing network transmission efficiency, thereby realizing high-performance and low-latency technical effects for file migration between big data clusters, and thus solving the technical problem of low file migration efficiency caused by high-frequency metadata requests and inefficient TCP connection overhead.

[0064] Specifically, this application addresses the characteristics of massive small files in financial systems by first classifying them into cold or hot data based on file access records. Then, it employs either physical merging (generating large files and file mapping tables) or logical merging (constructing virtual units and unit descriptors) strategies for these two types of files respectively. Furthermore, this application dynamically calculates the target concurrency to ensure reasonable resource allocation. An adaptive scheduling mechanism prioritizes the migration of virtual units and large files, effectively alleviating the pressure on metadata services while improving network efficiency for file migration. This avoids bandwidth contention between large and small files, ensuring smooth data migration and thus solving the technical problem of low file migration efficiency caused by high TCP connection overhead in existing technologies.

[0065] In one alternative embodiment, Figure 3 This is a structural diagram of an optional file migration system according to an embodiment of this application, such as... Figure 3 As shown, the file migration system includes: a hot and cold data intelligent identification engine, a physical merging module, a logical aggregation and transmission module, and an adaptive scheduler.

[0066] Optionally, in the file migration system, the data in the source cluster is identified as cold or hot data after passing through the cold and hot data intelligent identification engine. Then, the physical merging module physically merges multiple small files of cold data type to obtain a large file. The logical aggregation transmission module logically merges multiple small files of hot data type to obtain VTU data packets. Then, according to the transmission rules defined by the adaptive scheduler, the merged large file and VTU data packets are transmitted to the target cluster.

[0067] In one optional embodiment, in order to accurately determine the file type of each small file, the file migration system first determines the access heat coefficient and access frequency coefficient of each small file based on the access records of each small file. The access heat coefficient is used to characterize the number of times the small file is accessed within the time window corresponding to the time of the first access to the most recent access. The access frequency coefficient is used to characterize the access frequency of the small file.

[0068] Optionally, the access popularity coefficient is calculated by multiplying the number of accesses by the logarithmic compression value of the time span within the time window, that is, access popularity coefficient = number of accesses × log(timestamp corresponding to the most recent access time - timestamp corresponding to the first access time), which is used to quantify the activity level of small files in a specific time period in the past. The preset popularity coefficient threshold δ corresponding to the access popularity coefficient is 5, which means that it is accessed more than 2 times within 72 hours.

[0069] Optionally, the access frequency coefficient represents the frequency of a small file being accessed per unit of time. It is usually obtained by calculating the number of accesses and the time interval within a recent period. The preset frequency coefficient threshold corresponding to the access frequency is 72 hours.

[0070] Optionally, by introducing access popularity coefficients and access frequency coefficients, the file migration system can more precisely identify and differentiate the usage characteristics of small files, especially for the dynamic changes of hot and cold data. This method not only considers the total number of accesses but also the time distribution of accesses, which can reflect the timeliness and importance of files to a certain extent, providing a more accurate data classification basis for subsequent data processing.

[0071] Optionally, if the access popularity coefficient is less than or equal to a preset popularity coefficient threshold, or if the access frequency coefficient is greater than a preset frequency coefficient threshold, the file migration system determines the file type of the small file as a cold data type.

[0072] Optionally, the file migration system automatically filters out cold data by defining clear threshold criteria, avoiding excessive processing of cold data and frequent network transmissions, reducing the pressure on the NameNode, reducing the waste of network resources, and improving the utilization of storage space.

[0073] Optionally, if the access popularity coefficient is greater than a preset popularity coefficient threshold and the access frequency coefficient is less than or equal to a preset frequency coefficient threshold, the file migration system determines the file type of the small file as a hot data type.

[0074] Optionally, hot data types, as opposed to cold data, refer to small files that are frequently accessed (i.e., accessed frequently in a short period of time). These files are more suitable for logical aggregation transmission, maintaining their original access patterns and data structures. Efficient hot data processing strategies can ensure that users or applications can quickly obtain the data they need when required, reducing data retrieval and transmission latency, improving user experience and system response speed, and avoiding the data access latency and storage architecture complexity problems that may result from physically merging hot data.

[0075] Optionally, Figure 4 This is a flowchart of an optional intelligent cold / hot data recognition engine according to an embodiment of this application, such as... Figure 4 As shown, the workflow of the hot and cold data intelligent recognition engine includes the following steps:

[0076] Step 1: Recursively traverse all files and mark small files based on the file size threshold T (default 16MB).

[0077] Step 2: Associate the NameNode audit logs and extract the access records for each small file, including: access count, first access timestamp, and last access timestamp.

[0078] Step 3: Calculate the access popularity coefficient corresponding to the small file (the calculation process uses logarithmic compression of the time range to avoid interference from frequent historical accesses). The access popularity coefficient Score = number of accesses × log (timestamp of the most recent access - timestamp of the first access).

[0079] Step 4: Set the preset popularity coefficient threshold δ (default 5, meaning more than 2 visits within 72 hours) and the preset frequency coefficient threshold (default 72 hours), and determine hot and cold data based on the thresholds:

[0080] Criteria for determining cold data: Popularity Score ≤ δ or (Current Time - Last Access Time) > Frequency Threshold;

[0081] The criteria for determining hot data are: heat score > δ and (current time - last access time) ≤ frequency threshold.

[0082] Step 5: Generate a category report with hot and cold tags, including metadata such as file path, size, access statistics, and category tags.

[0083] In one optional embodiment, during the merging of M small files of the same file type from L small files, and of the cold data type, the file migration system first scans the storage directory of the small files of the cold data type from the L small files to obtain the M small files to be merged. Then, the file migration system sequentially reads the file content of the M small files and stores the sequentially read file content uniformly into an empty file of a preset format to obtain a large file corresponding to the M small files. Then, the file migration system uses the starting offset and storage length of the file content corresponding to each small file in the large file as the storage location identifier of the small file, and creates a check code corresponding to each small file in the M small files according to a preset algorithm. Finally, the file migration system generates a file mapping table corresponding to the M small files based on the storage location identifier and check code corresponding to each small file in the M small files.

[0084] Optionally, the file migration system determines which small files need to be merged by scanning the storage directories of small files of cold data types. The storage directory refers to the specific directory path where the small files are stored, which may be distributed on different nodes of the big data cluster.

[0085] Optionally, the file migration system scans the storage directory and intelligently identifies small files of cold data types. The system can accurately locate files suitable for merging operations, avoiding unnecessary processing of hot data or other files that should not be merged, thereby improving storage efficiency, reducing the burden on the NameNode, and ensuring that the normal performance of data access is not affected.

[0086] Optionally, the preset format refers to the target format used to merge files, such as HAR (Hadoop Archive, an archive file format) or ORC (Optimized Row Columnar, an optimized row and column storage format), which typically have better performance and compression ratios in big data processing.

[0087] Optionally, the file migration system ensures the continuity and consistency of the small file data after merging by sequentially reading, that is, reading the file contents of the M small files in the natural order of their location in the storage directory.

[0088] Optionally, the file migration system significantly reduces the number of network connections and metadata operations in subsequent transmission stages by merging multiple small files into one large file and storing it in a preset format, thereby improving transmission efficiency and reducing the load on the NameNode. At the same time, the selection of the preset format also optimizes storage performance and data read speed.

[0089] Optionally, the starting offset refers to the position of the beginning of the content of each small file in a large file relative to the beginning byte position of the file.

[0090] Optionally, storage length refers to the number of bytes occupied by the content of each small file in a large file.

[0091] Optionally, the checksum is a digital signature calculated according to a preset algorithm (such as CRC32) used to verify the integrity of the small file content and the correctness of the transmission.

[0092] Optionally, by recording storage location identifiers and generating checksums, the file migration system can ensure that the target end accurately splits and restores a large file into the original M smaller files. At the same time, the use of checksums ensures data integrity and consistency during the data migration process, enhancing the reliability and robustness of the file migration system.

[0093] Optionally, the file mapping table contains the storage location identifiers and checksums of each of the M small files within the larger file. It serves as a crucial index for the target end to parse and recover the content of the small files. The generation of the file mapping table ensures that the target end can efficiently and accurately locate and extract the content of each small file. At the same time, the use of checksums to verify the content of the small files greatly improves the efficiency and accuracy of data migration. Furthermore, the file mapping table also supports the deployment of a virtual file system on the target end, which can further enhance the flexibility and efficiency of data access.

[0094] In one alternative embodiment, Figure 5 This is a flowchart of an optional dual-track merging process for hot and cold data according to an embodiment of this application; Figure 6 This is a timing flowchart of an optional cold data merging process according to an embodiment of this application; Figure 7 This is a timing flowchart of an optional hot data merging process according to an embodiment of this application.

[0095] Optionally, combined Figure 5 and Figure 6 The process for merging cold data is described below, which includes the following steps:

[0096] Step 1: Scan the cold data directory to determine which files need to be merged.

[0097] Step 2: Generate a large file. Read multiple small files sequentially and write them into a new large file (HAR or ORC format), preserving the content order of each small file during the merging process.

[0098] Step 3: Generate a mapping table. During the merging process, record the starting offset, length, and CRC32 checksum value of each original file in the merged large file. This information is serialized and stored in Protobuf (Protocol Buffers) format.

[0099] Step 4: Store the merged large file and mapping table together in the target storage system.

[0100] Step 5: Delete the original small files to save storage space.

[0101] In one optional embodiment, during the merging of N small files of the same file type from L small files, and the file type is a hot data type, the file migration system first logically merges the small files of the same file type from the L small files and belonging to the same storage directory, to obtain X virtual units corresponding to the N small files, where X is a positive integer, and the unit size of each virtual unit is less than or equal to a preset unit size. Then, the file migration system determines the total data size of all small files corresponding to each virtual unit based on the sum of the data sizes of each small file corresponding to each virtual unit, and creates a checksum corresponding to each small file in the N small files according to a preset algorithm. Then, the file migration system uses the unit identifier, file list, total data size of all small files, and checksum of all small files corresponding to each virtual unit as the unit descriptor corresponding to the virtual unit.

[0102] Optionally, the file migration system can reduce the overhead of small files in network transmission by logically merging N small files of hot data types into several virtual transfer units (VTUs), with the size of each unit being less than or equal to a preset unit size.

[0103] Optionally, the logical merging strategy can effectively reduce the overhead caused by frequent network connection establishment, while maintaining the original structure of hot data to meet the needs of real-time access, improve transmission efficiency, and reduce network latency. It is especially suitable for processing small files that are frequently accessed, ensuring the speed of data transmission and the stability of system performance.

[0104] Optionally, the total data size refers to the sum of the data volume of all small files within each virtual unit. It is used to guide the file migration system to allocate network resources and memory space reasonably for the virtual unit, thereby helping to optimize memory management and network bandwidth allocation, ensuring the effective use of resources. At the same time, the introduction of the checksum ensures the accuracy of data during transmission, enhances the reliability of data migration, and avoids the problem of data corruption or loss.

[0105] Optionally, the small file list refers to the list of small files contained within each virtual unit. The small file list lists the paths of the small files contained in each virtual unit and the basic data size information of each small file.

[0106] Optionally, the cell descriptor contains a set of information including the cell identifier (VTU_ID), a list of small files, the total data size, and checksums of all small files. This set of information is the basis for the target end to split and verify the data after receiving the virtual cell.

[0107] Optionally, the generation of cell descriptors enables the target end to accurately identify and process the received virtual cells, split the data through a file list, and verify the data integrity using a checksum. The entire process does not require additional metadata queries, which greatly improves the efficiency of data processing and reduces the dependence on the NameNode. At the same time, this mechanism also facilitates subsequent breakpoint resume and data consistency assurance, ensuring high performance and high reliability of hot data migration.

[0108] Optionally, combined Figure 5 and Figure 7 The process for merging and processing thermal data is described below, which includes the following steps:

[0109] Step 1: Construct a Virtual Transfer Unit (VTU). Logically aggregate multiple small files in the same directory into a single virtual unit (VTU) (default size is 64MB, but can be configured as needed).

[0110] Step 2: Generate VTU descriptors. This includes the VTU_ID (unique identifier), a file list (containing the path, size, etc. of each file), and the total size (used for memory allocation during transfer).

[0111] Step 3: Memory Aggregation Transfer. Based on the VTU descriptor, the contents of each small file are sequentially read into a memory buffer, forming a contiguous data block; the entire data block is then transmitted as a data packet over the network to the target end. (This method avoids the network overhead of multiple small file transfers, and because it operates in memory, it avoids disk I / O).

[0112] Step 4: Atomic Splitting at the Target End. After receiving the data packet, the target end extracts the content of each small file from the data packet according to the file list information in the VTU descriptor, based on the offset.

[0113] Step 5: Atomic Write. Each small file is written independently to the target storage system, ensuring that the write operation is atomic.

[0114] In one optional embodiment, before migrating M small files using large files and file mapping tables based on target concurrency and priority mechanisms, and migrating N small files using virtual units and unit descriptors, the file migration system first collects performance metrics. These performance metrics include at least the real-time bandwidth of the file migration system, the average data size of all small files migrated within a preset time period, the upper limit of QPS, and the real-time QPS. Then, the file migration system uses the product of the real-time bandwidth and a preset bandwidth weight as the bandwidth factor. Next, the file migration system uses the product of the difference between the upper limit of QPS and the real-time QPS and a preset QPS weight as the QPS factor. Finally, the file migration system uses the minimum value among the maximum number of threads, the bandwidth factor, and the QPS factor as the target concurrency of the file migration system.

[0115] Optionally, real-time bandwidth refers to the currently available network transmission rate, which is a key parameter for measuring the instantaneous transmission capability of a file migration system.

[0116] Optionally, the QPS cap refers to the maximum number of requests per second that the NameNode can handle, reflecting the processing capacity limit of the metadata service.

[0117] Optionally, real-time QPS refers to the number of requests per second that the NameNode is currently processing, used to monitor the load status of the file migration system.

[0118] Optionally, by monitoring the above performance indicators in real time, the file migration system can gain a comprehensive understanding of its current status and resource utilization. This step provides basic data for subsequent dynamic adjustments, ensuring that the system can make optimal decisions under real-time conditions, thereby achieving efficient data transmission and stable operation of metadata services.

[0119] Optionally, the file migration system calculates the bandwidth factor based on the current real-time bandwidth, which can dynamically adjust the concurrency of data migration tasks, ensuring the efficiency and stability of network transmission, avoiding network congestion caused by excessive concurrency or resource waste caused by insufficient concurrency, and realizing flexible scheduling and efficient use of bandwidth resources.

[0120] Optionally, the file migration system can dynamically assess the NameNode's load status by calculating the QPS factor, and then reasonably adjust the concurrency of file migration tasks, avoiding NameNode overload caused by high concurrency and ensuring the continuous and stable operation of metadata services. This is crucial for the efficient management and data migration of big data clusters.

[0121] Optionally, the maximum number of threads refers to the maximum number of tasks that the file migration system can process simultaneously, reflecting the upper limit of the file migration system's hardware capabilities and software design.

[0122] Optionally, the dynamic calculation of the target concurrency ensures the adaptability of the file migration system. The file migration system can intelligently adjust the parallel level of transmission tasks according to real-time network conditions and NameNode load status, maximizing resource utilization efficiency while maintaining system stability and data transmission reliability. This adaptive scheduling strategy avoids resource waste or system overload that may occur under fixed configuration, and is one of the key technical means to improve the migration efficiency of massive small files in big data clusters.

[0123] Optionally, Figure 8 This is a flowchart of an optional adaptive parallel scheduling process according to an embodiment of this application, such as... Figure 8 As shown, the file migration system adaptively schedules transmission by constructing a dynamic concurrency model and scheduling priorities. This process includes the following steps:

[0124] Step 1: Obtain real-time metrics. Collect current bandwidth, average file size, current QPS of the NameNode, and QPS limit.

[0125] Step 2: Calculate the bandwidth factor. This ensures efficient bandwidth utilization. Smaller files allow for higher concurrency; conversely, larger files reduce concurrency and prevent congestion.

[0126] Bandwidth factor = α × (bandwidth Mbps / average file size MB);

[0127] Bandwidth factor α = 0.8 (reserving 20% ​​bandwidth margin to cope with network fluctuations and sudden traffic surges);

[0128] Step 3: Calculate the QPS factor.

[0129] QPS factor = β × (NameNode's maximum QPS - current QPS);

[0130] QPS factor coefficient β = 0.6 (reserving a 40% safety margin to prevent QPS overload from causing downtime).

[0131] Step 4: Calculate concurrency. Concurrency is the minimum value of the maximum number of threads, the bandwidth factor, and the QPS factor. This ensures that the concurrency will not exceed the system's thread limit, bandwidth capacity, or NameNode processing capacity. In summary:

[0132] Concurrency = min(

[0133] MAX_THREADS, / / Maximum number of threads

[0134] α × (bandwidth / average file size)

[0135] β × (NameNode's QPS cap - current QPS) )

[0137] Step 5: Prioritize. Figure 9 This is a flowchart of an optional priority allocation process according to an embodiment of this application, such as... Figure 9 As shown, the first priority is VTU units, the second priority is large files, and the third priority is small files that are not aggregated.

[0138] In one optional embodiment, during the migration of M small files of cold data type, the file migration system first transmits the large file and file mapping table corresponding to the M small files to the target storage system, and deletes the M small files from the source storage system. Then, after receiving the large file and file mapping table from the target storage system, the file migration system parses the file mapping table to obtain the storage location identifier and checksum of each of the M small files. Next, based on the storage location identifier of each small file, the file migration system extracts the file content from the large file to obtain the file content of each small file. Furthermore, based on the checksum of each small file, the file migration system verifies the file content of each small file. Finally, if the file content of each small file passes the verification, the file migration system determines that the migration of the M small files is successful.

[0139] Optionally, the file migration system significantly reduces the number of network transfers by transmitting large files and file mapping tables in one go, thereby improving transmission efficiency and reducing the load on metadata processing. At the same time, the cleanup action at the source end saves storage resources and improves the overall performance of the storage system.

[0140] Optionally, the parsing of the file mapping table provides a fast and accurate file recovery method for the file migration system. By identifying the storage location, the target end can efficiently extract the contents of each small file, while the checksum ensures the accuracy of data transmission and avoids repeated transmissions caused by data corruption, thereby enhancing the reliability of data transmission.

[0141] Optionally, the file migration system extracts the contents of small files from large files based on storage location identifiers, ensuring the lossless nature of data migration. Each small file can be fully recovered, while maintaining the integrity and originality of the data, thus providing a guarantee for subsequent data use and processing.

[0142] In one optional embodiment, during the file migration of N small files of hot data type, the file migration system first allocates a memory cache for each virtual unit based on the unit descriptor of each of the X virtual units. The size of the memory cache is equal to the total data size of all small files included in the unit descriptor. Then, the file migration system sequentially stores the file content of all small files corresponding to each virtual unit into the content cache corresponding to that virtual unit, obtaining a data block corresponding to each virtual unit. Next, the file migration system packages the data blocks corresponding to each virtual unit to obtain X data packets corresponding to X virtual units. Finally, the file migration system performs file migration of the N small files using the X data packets and X unit descriptors corresponding to the X virtual units.

[0143] Optionally, the memory cache refers to the memory space reserved on the target side for each virtual unit to temporarily store file content. By allocating an appropriate size memory cache for each virtual unit based on its total data size, the file migration system ensures efficient loading and management of data blocks on the target side. The use of the memory cache avoids frequent disk read and write operations, speeds up data transmission and processing, improves the efficiency of file reconstruction on the target side, and also reduces requests for metadata services, thus reducing the burden on the NameNode.

[0144] Optionally, a data block refers to a continuous data segment that contains the contents of all small files within a virtual unit, which is transmitted from the source by the file migration system. The file migration system sequentially saves the contents of all small files into the content buffer, forming a complete data block. This process improves data continuity and transmission efficiency. Compared to transmitting each small file separately, this step significantly reduces the number of network connections established, reduces TCP overhead, and improves the effective utilization of network bandwidth.

[0145] Optionally, the file migration system further creates a complete transmission unit—a data packet—by packaging data blocks with their corresponding unit descriptors. This simplifies the transmission process, ensures the consistency of data and metadata, and facilitates the target end's parsing and processing of the data packet. This packaging method avoids matching errors that may result from the separate transmission of data and metadata, thus enhancing the reliability and efficiency of data transmission.

[0146] Optionally, the file migration system significantly reduces the number of transmissions and improves transmission efficiency by converting the transmission of N small files into the transmission of X data packets. At the same time, the information in the cell descriptor guides the target end on how to correctly unpack and reconstruct the small files, ensuring the accuracy and consistency of the entire migration process. When handling small files of hot data types, this mechanism maintains file access speed and efficiency while making full use of network resources, achieving high performance and low latency for file migration between big data clusters, and significantly improving network efficiency and data integrity during the file migration process.

[0147] In one optional embodiment, the file migration system first transmits X data packets and X unit descriptors corresponding to N small files to the target storage system. Then, after the target storage system receives the X data packets and X unit descriptors, the file migration system atomically splits the data packets corresponding to the i-th unit descriptor based on the file information of each small file included in the i-th unit descriptor, obtaining the file content of each small file corresponding to the i-th unit descriptor. Then, the file migration system verifies the file content of each small file corresponding to the i-th unit descriptor based on the checksum in the file information of each small file included in the i-th unit descriptor. Finally, if the file content of each small file passes the verification, the file migration system determines that the N small files have been successfully migrated.

[0148] Optionally, the joint transmission mechanism of data packets and unit descriptors significantly reduces the file migration overhead between big data clusters, especially when processing small files of hot data types. It avoids the high network overhead of transmitting individual small files and improves the overall transmission efficiency and system throughput.

[0149] Optionally, the file migration system performs atomic splitting at the target end, that is, decomposes the data packet into independent small file contents based on the file information in the unit descriptor. This ensures that the contents of each small file are accurately separated from the data packet, avoids data aliasing or omission, maintains the original structure and access characteristics of the data, and ensures that even in a concurrent transmission environment, the contents of the small files can reach the target end accurately.

[0150] Optionally, the file migration system uses checksums to verify file content, which not only improves the reliability of data migration but also enhances the robustness of the target system when receiving data. This mechanism can detect potential errors at the final stage of data transmission and take timely remedial measures to avoid subsequent processing problems caused by data corruption. Through a rigorous data verification process, the file migration system can ensure the integrity of all N small files after transmission. This confirmation mechanism provides the final quality assurance for the file migration operation. For fields such as the financial industry, which have extremely high requirements for data accuracy, the confirmation step for successful migration is particularly critical. This mechanism can prevent business interruptions or data risks that may be caused by data migration failure.

[0151] Optionally, the file migration system includes a data consistency guarantee mechanism to ensure file consistency before and after transmission through transmission verification and breakpoint resumption. This mechanism includes:

[0152] 1) Transmission verification: The CRC32 check value of each file is calculated at the source end and written into the VTU descriptor (for hot data) or mapping table (for cold data); the target end verifies it after receiving it.

[0153] 2) Resume interrupted transmission: Record the list of transmitted VTU_IDs (hot data) or merged large file blocks (cold data), and retransmit from the most recent complete unit in case of failure.

[0154] As can be seen from the above, this application adopts a cold and hot data classification and dual-track merging approach, and by constructing a file mapping table and virtual transmission units, it achieves the goal of reducing the pressure on metadata services and optimizing network transmission efficiency, thereby realizing high-performance and low-latency technical effects for file migration between big data clusters, and thus solving the technical problem of low file migration efficiency caused by high-frequency metadata requests and inefficient TCP connection overhead.

[0155] Specifically, this application addresses the characteristics of massive small files in financial systems by first classifying them into cold or hot data based on file access records. Then, it employs either physical merging (generating large files and file mapping tables) or logical merging (constructing virtual units and unit descriptors) strategies for these two types of files respectively. Furthermore, this application dynamically calculates the target concurrency to ensure reasonable resource allocation. An adaptive scheduling mechanism prioritizes the migration of virtual units and large files, effectively alleviating the pressure on metadata services while improving network efficiency for file migration. This avoids bandwidth contention between large and small files, ensuring smooth data migration and thus solving the technical problem of low file migration efficiency caused by high TCP connection overhead in existing technologies.

[0156] Example 2

[0157] This application embodiment can also provide a file migration device. It should be noted that the file migration device of this application embodiment can be used to execute the file migration method provided in this application embodiment. The file migration device provided in this application embodiment is described below.

[0158] According to an embodiment of this application, an apparatus for implementing the above-described file migration method is also provided. Figure 10 This is a schematic diagram of an optional file migration apparatus according to an embodiment of this application, such as... Figure 10 As shown, the device includes: a type determination unit 1001, a merging unit 1002, and a migration unit 1003.

[0159] Optionally, the type determination unit 1001 is used to determine the file type of each small file based on the file access records of L small files in the financial system, where L is a positive integer, each small file is a file to be migrated whose file size is less than or equal to a preset size threshold, and the file type is either a cold data type or a hot data type; the merging unit 1002 is used to merge the small files of the same file type among the L small files to obtain the large files and file mapping tables corresponding to the M small files of the cold data type, and to obtain the virtual units and unit descriptors corresponding to the N small files of the hot data type, where M and N are both positive integers, L is equal to the sum of N and M, and the file mapping table The system includes at least M small files whose contents are stored in the large file, along with checksums for the M small files. The unit descriptor includes at least the unit identifier of the virtual unit and the file information for each small file corresponding to the virtual unit. The migration unit 1003 is used to migrate M small files through the large file and file mapping table based on the target concurrency and priority mechanism, and to migrate N small files through the virtual unit and unit descriptor. The target concurrency is the minimum of the maximum number of threads, bandwidth factor, and QPS factor of the file migration system. The QPS factor is used to characterize the processing efficiency of the file migration system for file query requests.

[0160] As can be seen from the above, this application adopts a cold and hot data classification and dual-track merging approach, and by constructing a file mapping table and virtual transmission units, it achieves the goal of reducing the pressure on metadata services and optimizing network transmission efficiency, thereby realizing high-performance and low-latency technical effects for file migration between big data clusters, and thus solving the technical problem of low file migration efficiency caused by high-frequency metadata requests and inefficient TCP connection overhead.

[0161] Specifically, this application addresses the characteristics of massive small files in financial systems by first classifying them into cold or hot data based on file access records. Then, it employs either physical merging (generating large files and file mapping tables) or logical merging (constructing virtual units and unit descriptors) strategies for these two types of files respectively. Furthermore, this application dynamically calculates the target concurrency to ensure reasonable resource allocation. An adaptive scheduling mechanism prioritizes the migration of virtual units and large files, effectively alleviating the pressure on metadata services while improving network efficiency for file migration. This avoids bandwidth contention between large and small files, ensuring smooth data migration and thus solving the technical problem of low file migration efficiency caused by high TCP connection overhead in existing technologies.

[0162] In one optional embodiment, the type determination unit 1001 includes: a first determination subunit, a second determination subunit, and a third determination subunit.

[0163] Optionally, the first determining subunit is used to determine the access popularity coefficient and access frequency coefficient of each small file based on the access records of each small file, wherein the access popularity is used to characterize the number of times the small file is accessed within the time window corresponding to the first access time to the most recent access time, and the access frequency is used to characterize the access frequency of the small file; the second determining subunit is used to determine the file type of the small file as a cold data type when the access popularity coefficient is less than or equal to a preset popularity coefficient threshold, or when the access frequency coefficient is greater than a preset frequency coefficient threshold; the third determining subunit is used to determine the file type of the small file as a hot data type when the access popularity coefficient is greater than a preset popularity coefficient threshold and the access frequency coefficient is less than or equal to a preset frequency coefficient threshold.

[0164] In an optional embodiment, the merging unit 1002 includes: a scanning subunit, a first reading subunit, a storage identifier determination subunit, and a mapping table determination subunit.

[0165] Optionally, a scanning subunit is used to scan the storage directories of small files of type cold data in the L small files to obtain M small files to be merged; a first reading subunit is used to sequentially read the file content of the M small files and store the sequentially read file content into an empty file of a preset format to obtain a large file corresponding to the M small files; a storage identifier determination subunit is used to use the starting offset and storage length of the file content corresponding to each small file in the large file as the storage location identifier of the small file, and to create a check code corresponding to each small file in the M small files according to a preset algorithm; and a mapping table determination subunit is used to generate a file mapping table corresponding to the M small files based on the storage location identifier and check code corresponding to each small file in the M small files.

[0166] In an optional embodiment, the merging unit 1002 further includes: a logical merging subunit, a checksum creation subunit, and a descriptor determination subunit.

[0167] Optionally, the logical merging subunit is used to logically merge small files of type hot data that belong to the same storage directory among L small files, to obtain X virtual units corresponding to N small files, where X is a positive integer, and the unit size of each virtual unit is less than or equal to a preset unit size; the checksum creation subunit is used to determine the total data size of all small files corresponding to the virtual unit based on the sum of the data sizes of each small file corresponding to each virtual unit, and to create a checksum corresponding to each small file among the N small files according to a preset algorithm; the descriptor determination subunit is used to use the unit identifier, file list, total data size of all small files, and checksum of all small files corresponding to each virtual unit as the unit descriptor corresponding to the virtual unit.

[0168] In one optional embodiment, the file migration device further includes: a performance index acquisition unit, a bandwidth factor determination unit, a QPS factor determination unit, and a target concurrency determination unit.

[0169] Optionally, the performance metric acquisition unit is used to acquire performance metrics of the file migration system, wherein the performance metrics include at least the real-time bandwidth of the file migration system, the average data size of all small files migrated within a preset time period, the QPS upper limit, and the real-time QPS; the bandwidth factor determination unit is used to multiply the real-time bandwidth by a preset bandwidth weight as the bandwidth factor of the file migration system; the QPS factor determination unit is used to multiply the difference between the QPS upper limit and the real-time QPS by a preset QPS weight as the QPS factor of the file migration system; and the target concurrency determination unit is used to determine the minimum value among the maximum number of threads, the bandwidth factor, and the QPS factor of the file migration system as the target concurrency of the file migration system.

[0170] In one optional embodiment, the migration unit 1003 includes: a first transmission subunit, a parsing subunit, a first extraction subunit, a first verification subunit, and a first migration result determination subunit.

[0171] Optionally, the first transmission subunit is used to transmit the large file and file mapping table corresponding to the M small files to the target storage system, and delete the M small files from the source storage system; the parsing subunit is used to parse the file mapping table after the target storage system receives the large file and file mapping table, to obtain the storage location identifier and checksum of each of the M small files; the first extraction subunit is used to extract the file content of the large file based on the storage location identifier of each small file, to obtain the file content of each small file; the first verification subunit is used to verify the file content of each small file based on the checksum of each small file; and the migration result determination subunit is used to determine that the migration of the M small files is successful if the file content of each small file passes the verification.

[0172] In an optional embodiment, the migration unit further includes a cache allocation subunit, a storage subunit, a packing subunit, and a migration subunit.

[0173] Optionally, a cache allocation subunit is used to allocate a memory cache area for each virtual unit based on the unit descriptor of each of the X virtual units, wherein the size of the memory cache area is equal to the total data size of all small files included in the unit descriptor; a storage subunit is used to sequentially store the file contents of all small files corresponding to each virtual unit into the content cache area corresponding to the virtual unit, thereby obtaining a data block corresponding to each virtual unit; a packaging subunit is used to package the data block corresponding to each virtual unit, thereby obtaining X data packets corresponding to the X virtual units; and a migration subunit is used to migrate N small files using the X data packets and X unit descriptors corresponding to the X virtual units.

[0174] In one optional embodiment, the migration unit further includes: a second transmission subunit, a splitting subunit, a second verification subunit, and a second migration result determination subunit.

[0175] Optionally, the second transmission subunit is used to transmit X data packets and X unit descriptors corresponding to N small files to the target storage system; the splitting subunit is used to, after the target storage system receives X data packets and X unit descriptors, atomically split the data packets corresponding to the i-th unit descriptor based on the file information of each small file included in the i-th unit descriptor to obtain the file content of each small file corresponding to the i-th unit descriptor; the second verification subunit is used to verify the file content of each small file corresponding to the i-th unit descriptor based on the checksum in the file information of each small file included in the i-th unit descriptor; and the second migration result determination subunit is used to determine that the migration of N small files is successful if the file content of each small file passes the verification.

[0176] It should be noted here that the type determination unit 1001, merging unit 1002 and migration unit 1003 mentioned above correspond to steps S201 to S203 in the method embodiment. The three units and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of the device and can run in the computer terminal 10 provided in the embodiment.

[0177] Example 3

[0178] Embodiments of this application can also provide an electronic device. Figure 11 This is a structural block diagram of an electronic device according to an embodiment of this application, such as... Figure 11 As shown, the electronic device includes: one or more ( Figure 11 (Only one is shown) Processor 1102, memory 1104, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0179] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and devices in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the above-mentioned file migration method.

[0180] The memory may include high-speed random access memory (RAM), and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, which can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks (LANs), mobile communication networks, and combinations thereof.

[0181] The processor can call the information and application program stored in the memory through the transmission device to execute steps S201 to S203.

[0182] Those skilled in the art will understand that Figure 11 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, PDAs, mobile internet devices, PADs, and other terminal devices. Figure 11 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 11 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 11 The different configurations shown.

[0183] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0184] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0185] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0186] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0187] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0188] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0189] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0190] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A file migration method characterized by, The method comprises the steps of: determining the file type of each small file based on the file access records of L small files in a financial system, wherein L is a positive integer, each small file is a to-be-migrated file with a file size less than or equal to a preset size threshold, and the file type is a cold data type or a hot data type; merging small files with the same file type in the L small files to obtain a large file corresponding to M small files of the cold data type and a file mapping table, and to obtain a virtual unit corresponding to N small files of the hot data type and a unit descriptor, wherein M and N are both positive integers, L is equal to the sum of N and M, the file mapping table at least includes storage location identifiers of file contents of the M small files in the large file and check codes of the M small files, and the unit descriptor at least includes a unit identifier of the virtual unit and file information of each small file corresponding to the virtual unit; performing file migration on the M small files through the large file and the file mapping table and performing file migration on the N small files through the virtual unit and the unit descriptor based on a target concurrency and a priority mechanism, wherein the target concurrency is the minimum value of the maximum number of threads of a file migration system, a bandwidth factor and a QPS factor, and the QPS factor is used to represent the processing efficiency of the file migration system for file query requests.

2. The file migration method of claim 1, wherein, Based on the file access records of L small files in a financial system, the file type of each small file is determined, comprising: Based on the access records of each small file, the access heat coefficient and the access frequency coefficient of each small file are determined, wherein the access heat is used to represent the number of accesses of a small file within a time window corresponding to the first access time to the latest access time, and the access frequency is used to represent the access frequency of a small file; In the case where the access heat coefficient is less than or equal to a preset heat coefficient threshold, or the access frequency coefficient is greater than a preset frequency coefficient threshold, the file type of the small file is determined as the cold data type; In the case where the access heat coefficient is greater than the preset heat coefficient threshold, and the access frequency coefficient is less than or equal to the preset frequency coefficient threshold, the file type of the small file is determined as the hot data type.

3. The file migration method of claim 1, wherein, Merging small files with the same file type in the L small files to obtain a large file corresponding to M small files of the cold data type and a file mapping table, comprising: Scanning the storage directories of small files with the cold data type in the L small files to obtain M small files that need to be merged; sequentially reading the file contents of the M small files, and storing the sequentially read file contents into an empty file in a preset format to obtain a large file corresponding to the M small files; taking the starting offset and the storage length of the file contents of each small file in the M small files in the large file as the storage location identifier of the small file, and creating a check code corresponding to each small file in the M small files according to a preset algorithm; Generate a file mapping table corresponding to the M small files based on the storage location identifier and the check code corresponding to each of the M small files.

4. The file migration method of claim 1, wherein, Merge the small files of the same file type in the L small files to obtain a virtual unit and a unit descriptor corresponding to the N small files of the hot data type, including: Logically merge the small files of the hot data type and belonging to the same storage directory in the L small files to obtain X virtual units corresponding to the N small files, wherein X is a positive integer, and the unit size of each virtual unit is less than or equal to a preset unit size; Determine the total data size of all small files corresponding to each virtual unit based on the sum of the data sizes of each small file corresponding to the virtual unit, and create a check code corresponding to each small file in the N small files according to a preset algorithm; Take the unit identifier, file list, total data size of all small files, and check code of all small files corresponding to each virtual unit as the unit descriptor corresponding to the virtual unit.

5. The file migration method of claim 1, wherein, Before performing file migration on the M small files based on the target concurrency and priority mechanism through the large file and the file mapping table, and performing file migration on the N small files through the virtual unit and the unit descriptor, the file migration method further comprises: Collect performance indicators of the file migration system, wherein the performance indicators at least include real-time bandwidth of the file migration system, average data size of all small files migrated within a preset time period, QPS upper limit, and real-time QPS; Take the product of the real-time bandwidth and a preset bandwidth weight as the bandwidth factor of the file migration system; Take the product of the difference between the QPS upper limit and the real-time QPS and a preset QPS weight as the QPS factor of the file migration system; Take the minimum value among the maximum number of threads of the file migration system, the bandwidth factor, and the QPS factor as the target concurrency of the file migration system.

6. The file migration method of claim 1, wherein, Perform file migration on the M small files through the large file and the file mapping table, including: Transmit the large file corresponding to the M small files and the file mapping table to a target storage system, and delete the M small files from a source storage system; After the target storage system receives the large file and the file mapping table, parse the file mapping table to obtain the storage location identifier and the check code of each of the M small files; Extract the file content of each small file from the file content of the large file based on the storage location identifier of each small file; Verify the file content of each small file based on the check code of each small file; In the case that the file content of each small file passes the verification, determine that the M small files are migrated successfully.

7. The file migration method of claim 1, wherein, Perform file migration on the N small files through the virtual unit and the unit descriptor, including: allocating a memory cache area for each of the X virtual units based on a unit descriptor of each of the virtual units, wherein a size of the memory cache area is equal to a total data size of all small files included in the unit descriptor; storing file contents of all small files corresponding to each of the virtual units into a content cache area corresponding to the virtual unit sequentially to obtain a data block corresponding to each of the virtual units; packing the data block corresponding to each of the virtual units to obtain X data packets corresponding to the X virtual units; performing file migration on the N small files through the X data packets and the X unit descriptors corresponding to the X virtual units.

8. The file migration method of claim 7, wherein, performing file migration on the N small files through the X data packets and the X unit descriptors corresponding to the X virtual units, comprising: transmitting the X data packets and the X unit descriptors corresponding to the N small files to a target storage system; after the target storage system receives the X data packets and the X unit descriptors, atomically splitting a data packet corresponding to an i-th unit descriptor based on file information of each small file included in the i-th unit descriptor to obtain file contents of each small file corresponding to the i-th unit descriptor; verifying the file contents of each small file corresponding to the i-th unit descriptor based on a check code in the file information of each small file included in the i-th unit descriptor; in a case where the file contents of each small file pass the verification, determining that the migration of the N small files is successful.

9. A file migration apparatus characterized by comprising: comprising: a type determination unit configured to determine a file type of each small file based on file access records of L small files in a financial system, wherein L is a positive integer, the each small file is a to-be-migrated file with a file size less than or equal to a preset size threshold, and the file type is a cold data type or a hot data type; a merging unit configured to merge small files of the same file type in the L small files to obtain a large file corresponding to M small files of the cold data type and a file mapping table, and to obtain a virtual unit corresponding to N small files of the hot data type and a unit descriptor, wherein M and N are positive integers, L is equal to a sum of N and M, the file mapping table at least includes storage location identifiers of file contents of the M small files in the large file and check codes of the M small files, and the unit descriptor at least includes a unit identifier of the virtual unit and file information of each small file corresponding to the virtual unit; a migration unit configured to perform file migration on the M small files through the large file and the file mapping table based on a target concurrency and a priority mechanism, and to perform file migration on the N small files through the virtual unit and the unit descriptor, wherein the target concurrency is a minimum value of a maximum thread number of a file migration system, a bandwidth factor and a QPS factor, and the QPS factor is used to represent a processing efficiency of the file migration system on a file query request.

10. An electronic device, comprising: An apparatus, including one or more processors and memory storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the file migration method of any one of claims 1 to 8. An apparatus, including one or more processors and memory storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the file migration method of any one of claims 1 to 8.

Images