Data synchronization method and device, electronic device and storage medium

Abstract

The present disclosure relates to a data synchronization method and device, an electronic device, and a computer-readable storage medium, and relates to the field of data processing technology, and can be applied to scenarios in which data synchronization is performed between different types of databases. The method includes: in response to a received data synchronization task, obtaining the source file storage path corresponding to the data synchronization task; based on the current number of folders under the source file storage path, creating a listening thread object that matches the data synchronization task; based on the listening thread object, determining the number of files to be uploaded under each folder to determine the upload thread object corresponding to each folder; and storing the source data file under the source file storage path to the target file storage path through the upload thread object. The present disclosure creates a listening thread object by monitoring the number of files under the source file storage path, thereby achieving data synchronization between different databases, and avoiding waste of thread objects while ensuring file upload efficiency.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data synchronization method, a data synchronization device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Traditional relational databases use a relational model to organize data, storing data in rows and columns for easy understanding. Traditional relational databases are not well-suited for storing large amounts of data or performing online analytical processing (OLAP). Therefore, in practice, data can be synchronized from relational databases to databases such as Hadoop's data warehouse tool (Hive), using the Hive database as a data warehouse for long-term data storage and OLAP operations.

[0003] For example, there are two main options for synchronizing data from a relational database to Hive: The first is importing data using Java Database Connectivity (JDBC). The second is using open-source data migration tools such as Sqoop to extract data from the source database into Hive in batches. However, these methods may encounter performance bottlenecks during data synchronization, resulting in low synchronization efficiency.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0005] The purpose of the present disclosure is to provide a data synchronization method, a data synchronization device, an electronic device, and a computer-readable storage medium, thereby overcoming, at least to a certain extent, the problem of synchronizing data from a relational database to a Hive database by importing it via JDBC or extracting it in batches through tools such as Sqoop, which in turn generates a large load on the source database.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0007] According to a first aspect of the present disclosure, a data synchronization method is provided, including: in response to a received data synchronization task, obtaining a source file storage path corresponding to the data synchronization task; the source file storage path includes a file storage path in a first type of database; based on the current number of folders under the source file storage path, creating a listening thread object that matches the number of data synchronization tasks; based on the listening thread object, determining the number of files to be uploaded under each folder in the source file storage path, and determining an upload thread object corresponding to each folder according to the number of files to be uploaded; through the upload thread object, storing the source data file under the source file storage path to a target file storage path; the target file storage path includes a file storage path in a second type of database.

[0008] In an exemplary embodiment of the present disclosure, the method further includes: providing a task configuration interface through an external application, receiving a task configuration operation based on the task configuration interface; determining task configuration parameters corresponding to the task configuration operation, and generating the data synchronization task based on the task configuration parameters.

[0009] In an exemplary embodiment of the present disclosure, the task configuration parameters include any one or more combinations of source file storage path, target database, target data table, target file storage path, abnormal file storage path, blacklist file list, whitelist file list, number of upload thread resources and file upload policy name.

[0010] In an exemplary embodiment of the present disclosure, the creation of a listening thread object that matches the number of data synchronization tasks based on the current number of folders under the source file storage path includes: obtaining a pre-configured first file scanning cycle; performing a scanning operation on the source file storage path based on the first file scanning cycle to determine the current number of folders under the source file storage path at the current moment; determining the number of listening threads that match the data synchronization task based on the current number of folders; and creating or destroying the listening thread object in real time based on the number of listening threads.

[0011] In an exemplary embodiment of the present disclosure, determining the upload thread object corresponding to each of the folders based on the number of files to be uploaded includes: obtaining a pre-configured file number threshold; if the number of files to be uploaded is less than or equal to the file number threshold, using the listening thread object corresponding to the folder as the upload thread object; if the number of files to be uploaded is greater than the file number threshold, obtaining an idle thread object from the upload thread pool as the upload thread object.

[0012] In an exemplary embodiment of the present disclosure, obtaining an idle thread object from the upload thread pool as the upload thread object includes: splitting the source data file based on the file quantity threshold to obtain a split data file; and obtaining the idle thread object from the upload thread pool as the upload thread object corresponding to the split data file.

[0013] In an exemplary embodiment of the present disclosure, storing the source data file under the source file storage path to the target file storage path through the upload thread object includes: obtaining the upload thread object; and the upload thread object adopting a data overwrite mode and storing the source data file under the source file storage path to the target file storage path through the data storage interface.

[0014] In an exemplary embodiment of the present disclosure, storing the source data file under the source file storage path to the target file storage path includes: obtaining the source data file under the source file storage path from the temporary storage area of ​​the first type of database; obtaining the file upload policy corresponding to the source data file; generating a partition addition judgment instruction corresponding to the source data file in the second type of database based on the file upload policy; and storing the source data file according to the partition addition judgment instruction and the target file storage path.

[0015] In an exemplary embodiment of the present disclosure, the generating of a partition addition judgment instruction corresponding to the source data file in the second type database based on the file upload strategy includes: performing a name resolution operation on the source file name of the source data file based on the file upload strategy to generate a resolved storage path corresponding to the source data file; determining whether the existing storage path of the second type database contains the resolved storage path; and generating the creating a new partition instruction if the existing storage path does not contain the resolved storage path.

[0016] In an exemplary embodiment of the present disclosure, storing the source data file according to the partition addition judgment instruction and the target file storage path includes: storing the source data file to the target file storage path; in response to the partition addition judgment instruction being a create new partition instruction, executing a new partition operation to obtain a new data partition, and storing the source data file to the new data partition; if the new partition operation fails to be executed, moving the source data file from a temporary storage area to an abnormal file storage path; and deleting the source data file under the source file storage path.

[0017] In an exemplary embodiment of the present disclosure, the method further includes: responding to an abnormal monitoring response instruction of an abnormal file monitoring program, determining an abnormal file path based on the abnormal monitoring response instruction; and moving the abnormal file under the abnormal file path to a temporary storage area of ​​a first type of database to re-store the abnormal file.

[0018] In an exemplary embodiment of the present disclosure, the method further includes: obtaining the number of unuploaded files in the temporary storage area of ​​the first type database, and a pre-configured temporary file threshold; if the number of unuploaded files is greater than the temporary file threshold, generating an upload exception instruction; and generating exception alarm information based on the upload exception instruction.

[0019] In an exemplary embodiment of the present disclosure, the method further includes: obtaining a pre-configured second file scanning cycle; performing a file scanning operation on the data files under the archive file storage path based on the second file scanning cycle to filter out expired files; and deleting the expired files.

[0020] According to the second aspect of the present disclosure, a data synchronization device is provided, including: a parameter acquisition module, used to obtain the source file storage path corresponding to the data synchronization task in response to a received data synchronization task; the source file storage path includes the file storage path in the first type of database; a monitoring resource creation module, used to create a monitoring thread object matching the number of data synchronization tasks based on the current number of folders under the source file storage path; an upload resource creation module, used to determine the number of files to be uploaded under each folder in the source file storage path based on the monitoring thread object, and determine the upload thread object corresponding to each folder according to the number of files to be uploaded; a data storage module, used to store the source data file under the source file storage path to the target file storage path through the upload thread object; the target file storage path includes the file storage path in the second type of database.

[0021] In an exemplary embodiment of the present disclosure, the data synchronization device also includes a parameter configuration module, which is used to: provide a task configuration interface through an external application, receive task configuration operations based on the task configuration interface; determine the task configuration parameters corresponding to the task configuration operation, and generate the data synchronization task based on the task configuration parameters.

[0022] In an exemplary embodiment of the present disclosure, the listening resource creation module includes a listening resource creation unit, which is used to: obtain a pre-configured first file scanning cycle; perform a scanning operation on the source file storage path based on the first file scanning cycle to determine the current number of folders under the source file storage path at the current moment; determine the number of listening threads that match the data synchronization task based on the current number of folders; and create or destroy the listening thread object in real time based on the number of listening threads.

[0023] In an exemplary embodiment of the present disclosure, the upload resource creation module includes an upload resource determination unit, which is used to: obtain a pre-configured file number threshold; if the number of files to be uploaded is less than or equal to the file number threshold, then use the listening thread object corresponding to the folder as the upload thread object; if the number of files to be uploaded is greater than the file number threshold, then obtain an idle thread object from the upload thread pool as the upload thread object.

[0024] In an exemplary embodiment of the present disclosure, the thread resource determination unit includes a thread object acquisition sub-unit, which is used to: split the source data file based on the file quantity threshold to obtain a split data file; and obtain the idle thread object from the upload thread pool as the upload thread object corresponding to the split data file.

[0025] In an exemplary embodiment of the present disclosure, the data storage module includes a data storage unit, which is used to: obtain the upload thread object; and the upload thread object adopts a data overwrite mode and stores the source data file under the source file storage path to the target file storage path through the data storage interface.

[0026] In an exemplary embodiment of the present disclosure, the data storage unit includes a data storage sub-unit, which is used to: obtain the source data file under the source file storage path from the temporary storage area of ​​the first type of database; obtain the file upload policy corresponding to the source data file; based on the file upload policy, generate a partition addition judgment instruction corresponding to the source data file in the second type of database; and store the source data file according to the partition addition judgment instruction and the target file storage path.

[0027] In an exemplary embodiment of the present disclosure, the data storage subunit includes a new partition subunit, which is used to: perform a name resolution operation on the source file name of the source data file based on the file upload strategy, and generate a resolved storage path corresponding to the source data file; determine whether the existing storage path of the second type database contains the resolved storage path; if the existing storage path does not contain the resolved storage path, generate the instruction to create a new partition.

[0028] In an exemplary embodiment of the present disclosure, the data storage sub-unit is configured to execute: storing the source data file to the target file storage path; in response to the partition addition judgment instruction being a create new partition instruction, executing a new partition operation to obtain a new data partition, and storing the source data file to the new data partition; if the new partition operation fails to execute, moving the source data file from the temporary storage area to the abnormal file storage path; and deleting the source data file under the source file storage path.

[0029] In an exemplary embodiment of the present disclosure, the data synchronization device also includes an abnormal file processing module, which is used to: respond to the abnormal monitoring response instruction of the abnormal file monitoring program, determine the abnormal file path based on the abnormal monitoring response instruction; move the abnormal file under the abnormal file path to the temporary storage area of ​​the first type of database to re-store the abnormal file.

[0030] In an exemplary embodiment of the present disclosure, the data synchronization device also includes an alarm information generation module, which is used to: obtain the number of unuploaded files in the temporary storage area in the first type of database, and a pre-configured temporary file threshold; if the number of unuploaded files is greater than the temporary file threshold, generate an upload exception instruction; and generate abnormal alarm information based on the upload exception instruction.

[0031] In an exemplary embodiment of the present disclosure, the data synchronization device also includes an expired file processing module, which is used to: obtain a pre-configured second file scanning cycle; perform a file scanning operation on the data files under the archive file storage path based on the second file scanning cycle to filter out expired files; and delete the expired files.

[0032] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the data synchronization method according to any one of the above items is implemented.

[0033] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the data synchronization method according to any one of the above items is implemented.

[0034] The technical solutions provided by the present disclosure may have the following beneficial effects:

[0035] The data synchronization method in the exemplary embodiment of the present disclosure, on the one hand, provides a solution for data synchronization between different types of databases by monitoring the source data file path and uploading data to the target file storage path of the target database, thereby reducing the load on both the source and target databases. On the other hand, by creating a listening thread object based on the current number of folders to perform file scanning and obtaining an upload thread object corresponding to the number of files for file upload, data storage efficiency can be maximized while avoiding the waste of thread objects.

[0036] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0038] Figure 1 A schematic block diagram schematically illustrates a system architecture of an exemplary application scenario according to some embodiments of the present disclosure;

[0039] Figure 2 A flowchart of a data synchronization method according to an exemplary embodiment of the present disclosure is schematically shown;

[0040] Figure 3 The following schematically illustrates a flow chart of synchronizing NAS files to HDFS through a data synchronization task according to an exemplary embodiment of the present disclosure;

[0041] Figure 4 The overall flow chart of executing a data synchronization task according to an exemplary embodiment of the present disclosure is schematically shown;

[0042] Figure 5 Schematically shows a block diagram of a data synchronization device according to an exemplary embodiment of the present disclosure;

[0043] Figure 6A block diagram schematically illustrates an electronic device according to an exemplary embodiment of the present disclosure;

[0044] Figure 7 A schematic diagram schematically illustrates a computer-readable storage medium according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0045] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. Like reference numerals in the drawings represent like or similar parts, and thus repetitive description thereof will be omitted.

[0046] In addition, the described features, structures or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, well-known structures, methods, devices, implementations, materials or operations are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0047] The blocks shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. Specifically, these functional entities may be implemented in software, or in one or more software-hardened modules, or in different networks and / or processor devices and / or microcontroller devices.

[0048] Traditional first-type databases are not good at storing and performing OLAP operations on massive amounts of data. Therefore, in actual applications, data is synchronized from first-type databases to databases such as Hive, and the Hive database is used as a second-type database for long-term data storage and OLAP operations.

[0049] Currently, there are two main solutions for synchronizing data from Type 1 databases to Hive. The first is importing data using JDBC, which places a high load on Hive and has poor performance. The second is batch extraction using tools such as Sqoop. This method is more efficient but can place a high load on the source database, potentially impacting the normal use of the business database. This is unacceptable in actual production.

[0050] Based on this, in this example embodiment, a data synchronization method is first provided. Figure 1 , Figure 1 A schematic block diagram shows a system architecture of an exemplary application scenario in which a data synchronization method and apparatus according to an embodiment of the present disclosure can be applied.

[0051] like Figure 1 As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, a server 105, and databases 106 and 107. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc. The terminal devices 101, 102, 103 may be various electronic devices with display screens, including but not limited to desktop computers, portable computers, smart phones, tablet computers, etc. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as needed. For example, the server 105 may be a server cluster consisting of multiple servers.

[0052] The data synchronization method provided in the embodiments of the present disclosure is generally executed by the server 105, and accordingly, the data synchronization device is generally set in the server 105. For example, in an exemplary embodiment, a staff member may perform task configuration operations through the terminal devices 101, 102, and 103 to generate a data synchronization task, and upload the generated data synchronization task to the server 105. The server then synchronously uploads the source data files in the database 106 to the database 107 using the data synchronization method provided in the embodiments of the present disclosure.

[0053] Specifically, Figure 2 The following schematically illustrates a data synchronization method flow according to some embodiments of the present disclosure. Figure 2 , the data synchronization method may include the following steps:

[0054] Step S210: In response to the received data synchronization task, obtain a source file storage path corresponding to the data synchronization task; the source file storage path includes a file storage path in a first type of database.

[0055] According to some exemplary embodiments of the present disclosure, a data synchronization task may be a synchronization storage task for synchronizing data stored in a first-type database to a second-type database. The source file storage path may be a file storage path in the first-type database. The file storage path in the first-type database may be a storage path corresponding to a data file stored in the first-type database. For example, the first-type database may be a relational database, and the second-type database may be a data warehouse.

[0056] Different databases have corresponding characteristics. When using data, in order to make up for the shortcomings of one type of database, the data files in the database can be synchronized to other types of databases for corresponding data processing. For example, in order to make up for the shortcomings of traditional relational databases in storing massive data and performing OLAP operations, the data in the relational database can be synchronized to a data warehouse, such as a Hive database. When there is a need for data synchronization, a corresponding data synchronization task can be created based on the data synchronization need. When a data synchronization task is received, in response to the received data synchronization task, the source file storage path contained in the data synchronization task is obtained. The source file storage path can be the storage path for storing source data files in the first type of database. The data synchronization task can be to synchronize the source data files under the source data file storage path to the second type of database.

[0057] Step S220: Based on the current number of folders under the source file storage path, create a listening thread object that matches the number of data synchronization tasks.

[0058] According to some exemplary embodiments of the present disclosure, the current number of folders may be the number of folders in the source file storage path at the current moment, and the monitoring thread object may be a thread resource object for scanning the number of folders contained in each source file storage path.

[0059] After obtaining the source file storage path, the source data files stored there can be retrieved, and the number of folders under the source file storage path at the current moment can be determined. For example, during the production process, multiple machines may be put into production, and the production data of each machine can be stored in the corresponding machine. Each machine can correspond to multiple storage paths, and the storage paths of different machines can be used as source file storage paths. Different storage paths on a machine may store one or more folders. The current number of folders under the source file storage path is used to create a listening thread object that matches the number of data synchronization tasks.

[0060] Since each storage path contains a different number of folders, each folder is assigned a corresponding listening thread object. Therefore, the different number of listening thread objects required under each storage path can be determined according to the number of different folders. That is, the created listening thread objects can be automatically expanded and reduced according to the number of files at the current moment, so as to improve the efficiency of data file uploading while avoiding the waste of listening thread objects.

[0061] Step S230: Based on the monitoring thread object, determine the number of files to be uploaded in each folder in the source file storage path, and determine the upload thread object corresponding to each folder according to the number of files to be uploaded.

[0062] According to some exemplary embodiments of the present disclosure, the number of files to be uploaded may be the number of files contained in each folder. The upload thread object may be a thread object used to execute a data synchronization task of synchronizing data from a first type database to a second type database.

[0063] Since a folder can store one or more file objects, after determining the listener thread object corresponding to each folder in the source file storage path, the number of files stored in each folder, i.e., the number of files to be uploaded, can be determined based on the listener thread object. Furthermore, based on the number of files to be uploaded in each folder, the upload thread object corresponding to each folder is determined, and the file upload operation is performed based on the determined upload thread object.

[0064] Step S240: By uploading the thread object, the source data file under the source file storage path is stored in the target file storage path; the target file storage path includes the file storage path in the second type database.

[0065] According to some exemplary embodiments of the present disclosure, the source data file may be a data file stored in a source data file storage path, and the target file storage path may be a file storage path used by the second type database for storing data files.

[0066] After creating an upload thread object for processing data synchronization tasks in each folder, the source data files under the source file storage path can be stored in the target file storage path based on the upload thread object. For example, the target file storage path can be the file storage path in the data warehouse, that is, the target file storage path can be the storage path in the Hive database after the source data file is uploaded to the Hive database. After synchronizing the source data files in the relational database to the target file storage path in the data warehouse, the data files can be stored long-term in the data warehouse, and corresponding data operations can be performed based on the data warehouse, making up for the defect that the relational database cannot store data long-term and perform data operations.

[0067] The data synchronization method in this example embodiment, on the one hand, provides a solution for synchronizing data between different types of databases by monitoring the source data file path and uploading data to the target file storage path of the target database, reducing the load on both the source and target databases. On the other hand, by creating a listening thread object based on the current number of folders to scan files and obtaining an upload thread object corresponding to the number of files for file upload, data storage efficiency is maximized while avoiding wasted thread objects.

[0068] The data synchronization method in this example embodiment will be further described below. For the convenience of description, this embodiment takes the first type of database as a relational database and the second type of database as a data warehouse as an example to further describe the data synchronization method.

[0069] In an exemplary embodiment of the present disclosure, a task configuration interface is provided through an external application, and a task configuration operation based on the task configuration interface is received; task configuration parameters corresponding to the task configuration operation are determined, and a data synchronization task is generated based on the task configuration parameters.

[0070] The application can be a visual user interface for interacting with users and providing them with task configuration operations. The task configuration interface can be a user interface provided by the application for users to perform task configuration operations. Task configuration operations can be user operations for configuring parameters to generate a data synchronization task. Task configuration parameters can be task parameters used to execute the data synchronization task.

[0071] refer to Figure 3 , Figure 3 This diagram schematically illustrates a flow chart for synchronizing NAS files to HDFS using a data synchronization task, according to an exemplary embodiment of the present disclosure. The program server 310 executing the data synchronization task can connect to an external application (webapp) 320 and NAS storage 330. Application 320 can provide the user with a task configuration interface for performing task configuration operations. The task configuration interface includes task configuration parameters that require manual configuration by the user. After filling in these task configuration parameters, the user can click the Submit Data Synchronization Task action button to trigger the data synchronization task.

[0072] At this point, the application receives the task configuration parameters in JavaScript Object Notation (JSON) format entered by the user, uses them as input for the data synchronization task, generates the corresponding data synchronization task, and submits it to the server. This task configuration generates the data synchronization task to be executed.

[0073] In an exemplary embodiment of the present disclosure, the task configuration parameters include any one or more combinations of the source file storage path, target database, target data table, target file storage path, abnormal file storage path, blacklist file list, whitelist file list, number of upload thread resources and file upload policy name.

[0074] Among them, the target database can be a second type database for storing target data files, and the target database can include a corresponding target database name (Hive DB). The target data table can be a data table in the second type database for storing target data files, and the target data table contains a corresponding data table name (Hive Table). The abnormal file storage path can be a storage path for storing abnormal data files. For example, the abnormal file storage path can be used to store data files that failed to upload. The blacklist file list can be a list of data files that need to be intercepted. The whitelist file list can be a list of whitelist data files. In some specific scenarios, data files other than those on the whitelist cannot be passed. The number of upload thread resources can be the number of upload thread object resources used to process data synchronization tasks. The file upload policy name can be the name of the upload strategy used to perform data synchronization tasks.

[0075] The task configuration interface provided by the application allows users to configure task parameters such as the source file storage path, target database, target data table, target file storage path, abnormal file storage path, blacklist file list, whitelist file list, number of upload thread resources, and file upload policy name. Specifically, the source file storage path can be the NAS path corresponding to the data file. The NAS path can include a temporary storage path. The source data file to be uploaded can be temporarily stored in a temporary storage area (staging area) and have a corresponding storage path. The target database can be the second type of database required to store the source data file in the data synchronization task. In the target database, data files are also stored in data tables one by one. Therefore, the task configuration parameters can also include the target data table. In addition, after the source data file is stored in the target database, there is also a corresponding target file storage path in the target database. In the task configuration parameters, the target file storage path can also be configured.

[0076] During the data synchronization process, some source data files may fail to upload. In this case, these failed source data files can be stored in an exception file storage path. Therefore, you can configure the exception file storage path during task configuration. Additionally, during task configuration, you can configure blacklist and whitelist file lists. For example, if a data file poses a security risk or is at risk of being infiltrated by a network virus, you can blacklist it and add it to the blacklist. Source data files that are readily accepted can be added to the whitelist.

[0077] Furthermore, when executing a data synchronization task, you can also configure the number of upload thread objects used to execute the data synchronization task, that is, the number of core threads of the file upload thread pool, which represents the number of upload thread objects in the thread pool that can be used to perform data synchronization tasks. During the parameter configuration operation, the number of upload thread objects can be configured to a fixed value, such as 10. For different source data files, you can configure the file upload strategy and strategy name for performing data synchronization tasks according to actual needs. Among them, it can include multiple file upload strategies for processing data synchronization tasks. For example, the file upload strategy can include a general upload strategy, a data deletion after upload strategy, and a data merge strategy. After completing the above parameter configuration operation, the corresponding data synchronization task can be triggered.

[0078] In an exemplary embodiment of the present disclosure, for step S220, based on the current number of folders under the source file storage path, a listening thread object that matches the number of data synchronization tasks is created, including: obtaining a pre-configured first file scanning cycle; performing a scanning operation on the source file storage path based on the first file scanning cycle to determine the current number of folders under the source file storage path at the current moment; determining the number of listening threads that match the data synchronization task based on the current number of folders; and creating or destroying listening thread objects in real time based on the number of listening threads.

[0079] The first file scanning cycle may be a scanning cycle for scanning the number of folders stored in the source file storage path. The scanning operation may be an operation for obtaining file information of all source data files in the source file storage path and counting the number of folders containing source data files. The number of listening threads may be the number of thread objects used to scan and obtain the data files contained in each folder.

[0080] When performing a file upload operation, the folders of the source file storage path can be scanned according to a specific time period to determine the number of folders contained in each source file storage path. Specifically, a first file scanning period is obtained. For example, the first file scanning period can be configured to be 10 seconds (s), 20 seconds, etc. After obtaining the first file scanning period, the main thread of the program can scan the folders storing the source data files under the source file storage path at intervals of the first file scanning period, such as 20 seconds, to obtain the current number of folders under the source file storage path at the current moment.

[0081] Continue to refer Figure 3 After obtaining the current number of folders, the number of threads that matches the number of data synchronization tasks can be determined based on the current number of folders. Since the main thread of the program will repeatedly perform a file scanning operation every first file scanning cycle, the current number of folders under the data file NAS path is obtained, and a corresponding number of listening thread objects 340 are created based on the current number of folders. For example, at the first moment, the number of current folders obtained by scanning is 10. At this time, the number of configured listening threads is 10. At the second moment, the number of current folders obtained by scanning is 20. At this time, 10 listening threads can be added, and the number of listening threads required at the second moment is 20. At the third moment, the number of current folders obtained by scanning is 15, and 5 listening thread objects can be recycled to make the number of listening thread objects match the number of folders under the source file storage path.

[0082] The main thread scans a specific NAS path, such as the source file storage area or temporary storage area, and dynamically creates a listener thread pool based on the number of folders on the machine within that path, automatically scaling it up or down. This automatically scales the number of listener threads in the thread pool based on real-time changes in the number of folders, ensuring efficient data synchronization while avoiding wasted listener threads.

[0083] In an exemplary embodiment of the present disclosure, for step S230, the upload thread object corresponding to each folder is determined according to the number of files to be uploaded, including: obtaining a pre-configured file number threshold; if the number of files to be uploaded is less than or equal to the file number threshold, then using the listening thread object corresponding to the folder as the upload thread object; if the number of files to be uploaded is greater than the file number threshold, then obtaining an idle thread object from the upload thread pool as the upload thread object.

[0084] The file quantity threshold may be a pre-configured quantity threshold for comparing the size with the current folder quantity.

[0085] Continue to refer Figure 3For each folder 350 in the source file storage path, the number of files to be uploaded corresponding to the folder can be determined by monitoring the thread object, and then determining whether the number of files to be uploaded in the folder exceeds the threshold that can be carried by an upload thread object. Specifically, a pre-configured file number threshold is obtained. For example, the number of files that can be uploaded by a thread object does not exceed 200 data files. Therefore, the pre-configured file number threshold may be 200.

[0086] If the number of files to be uploaded in a certain folder is less than or equal to the file number threshold, the listening thread object 340 corresponding to the folder is used as the upload thread object, as the thread object for performing the file upload operation, that is, the source data file in the folder is directly uploaded using the listening thread object. If the current number of folders determined is greater than the file number threshold, multiple thread objects 360 are obtained from the upload thread pool as upload thread objects, so as to use the obtained multiple upload thread objects to perform the file upload operation. According to the number of files to be uploaded, a matching number of upload thread objects are obtained from the upload thread pool to perform the file upload operation. For example, during the task parameter configuration process, the number of upload thread objects is configured to 10, then a maximum of 10 upload thread objects can be used for file upload operations. If there are currently 5 upload thread objects occupied, only 5 upload thread objects can be used for file upload operations in the folder. After the remaining 5 upload thread objects are released, all 10 upload thread objects can perform file upload operations.

[0087] In an exemplary embodiment of the present disclosure, an idle thread object is obtained from an upload thread pool as an upload thread object, including: splitting the source data file based on a file quantity threshold to obtain a split data file; and obtaining an idle thread object from the upload thread pool as the upload thread object corresponding to the split data file.

[0088] The splitting process may include splitting the source data file based on a file number threshold to ensure that the number of files contained in the split data file does not exceed the file number threshold. The split data file may be a data file obtained by splitting the source data file based on the file number threshold. The idle thread object may be an unoccupied upload thread object in the upload thread pool.

[0089] When the determined number of files to be uploaded is greater than the file number threshold, the source data file may be segmented according to the file number threshold (ie, the maximum number of files uploaded by a single thread) to obtain segmented data files.

[0090] For example, the upper limit on the number of files that can be uploaded at a time by each upload thread object, i.e., the file number threshold, is 100. If the number of files to be uploaded in the folder is determined to be 580, the source data file is segmented according to the file number threshold of 100, resulting in 6 segmented data files. At this point, an idle thread object can be obtained from the upload thread pool as the upload thread object to perform file upload processing. Through the above processing operations, the source data files in a folder can be segmented and file upload processing can be performed based on the segmented data files.

[0091] In an exemplary embodiment of the present disclosure, for step S240, the source data file under the source file storage path is stored in the target file storage path through the upload thread object, including: obtaining the upload thread object; the upload thread object adopts a data overwrite mode and stores the source data file under the source file storage path to the target file storage path through the data storage interface.

[0092] The data overwriting mode may be a data writing mode in which all existing data is deleted and new data is written in. The data storage interface may be an application program interface used to synchronize source data files in the first type database to the second type database.

[0093] After determining the upload thread object corresponding to a folder, the upload thread object can call the Hadoop Distributed System (HDFS) Application Programming Interface (API) and use the data overwrite mode (overwrite mode) to upload the data file to the HDFS file path corresponding to Hive, that is, to the target file storage path. In addition, in other exemplary embodiments of the present disclosure, other modes can also be used for data synchronization operations, and this disclosure does not make any special limitations on this.

[0094] Through the above operation steps, the source data files in the first type database can be synchronized to the target file storage path in the second type database, completing data synchronization so that the data stored therein can be stored and operated on for a long time through the second type database.

[0095] In an exemplary embodiment of the present disclosure, a source data file under a source file storage path is stored in a target file storage path, including: obtaining the source data file under the source file storage path from a temporary storage area of ​​a first type of database; obtaining a file upload policy corresponding to the source data file; generating a partition addition judgment instruction corresponding to the source data file in a second type of database based on the file upload policy; and storing the source data file according to the partition addition judgment instruction and the target file storage path.

[0096] The temporary storage area may be a data temporary storage area in a Network Attached Storage (NAS) storage. The file upload policy may be a storage policy used for data synchronization operations. The data table in the second-type database may be a data table stored in the second-type database. For example, the data table in the second-type database may exist in a key-value pair format. The partition addition judgment instruction may be an operation instruction indicating whether to add a data partition to the data table in the second-type database.

[0097] When performing data synchronization tasks, a temporary storage area can be set up for data files in a relational database. Data files in the relational database can be temporarily stored in the temporary storage area. When uploading files, the source data files under the source file storage path can be retrieved from the temporary storage area. Different file upload policies can be applied to different source data files due to different file storage requirements. For example, the file upload policy corresponding to a source data file can be the file policy. This means that after the source data file is stored from the relational database to the data warehouse, the source data stored in the relational database will be deleted.

[0098] When synchronizing data based on a file upload policy, you can generate a new partition addition determination instruction for the source data file in the data warehouse based on the file upload policy. For example, if the source data files in a batch of data synchronization tasks need to be stored in different data partitions in the data warehouse due to different file upload policies, you can determine whether a matching data partition exists in the data warehouse's data table and generate a partition addition determination instruction to determine whether to add the new data partition.

[0099] The partition addition judgment instruction can indicate whether the source data file should be stored in the newly added data partition when storing the file in the data warehouse. Therefore, data synchronization operations in the data warehouse can be completed based on the partition addition judgment instruction and the target file storage path. Through this operation, the source data file in the relational database can be stored in the corresponding data partition, so that the data warehouse can subsequently perform long-term data storage and data operations.

[0100] In an exemplary embodiment of the present disclosure, based on the file upload strategy, a new partition addition judgment instruction corresponding to the source data file in the second type of database is generated, including: based on the file upload strategy, performing a name resolution operation on the source file name of the source data file to generate a resolution storage path corresponding to the source data file; determining whether the existing storage path of the second type of database contains the resolution storage path; if the existing storage path does not contain the resolution storage path, generating an instruction to create a new partition.

[0101] Among them, the file upload strategy can be a storage strategy adopted to synchronize the source data file to the target file storage path. The source file name can be the file name corresponding to the source data file. The name resolution operation can be a processing operation of resolving the source file name to obtain the target storage path corresponding to the source data file. The resolved storage path can be a target storage path determined based on the source file name. The existing storage path can be a storage path for file storage that already exists in the second type database. The create new partition instruction can be an operation instruction to add a new data partition in the data table of the second type database.

[0102] Because the file upload policy may include information about storing data in different locations in the second-type database, when creating a new partition in the second-type database, a name resolution operation can be performed on the source file name of the source data file based on the file upload policy to generate the first partition information corresponding to the source data file. For example, if the source file name consists of specified characters, the creation time, and random characters, such as the source file name "a_2022-01-01_gjghsueh.csv", after obtaining the source file name, the source file name can be resolved, such as by resolving the multi-level path corresponding to the source data file based on the separator "_", and then splicing them together to obtain the resolved storage path corresponding to the original data file, that is, the HDFS path corresponding to the source data file.

[0103] After obtaining the parsed storage path (i.e., HDFS path), it can be determined whether the existing storage path of the second type database contains the HDFS path obtained by parsing, that is, whether the parsed HDFS path exists in the second type database. If the parsed storage path exists in the second type database, the data can be directly stored in the corresponding target storage path. If the path does not exist, it is necessary to create a data partition for storing the file in the second type database to construct the corresponding parsed storage path, so an instruction to create a new partition can be generated. For example, a new data partition can be created by executing a HiveQL statement through Java Database Connectivity (JDBC), alter table XXXX add partition. The source data file is stored in the newly created data partition corresponding to the second type database to complete the data storage.

[0104] In an exemplary embodiment of the present disclosure, storing source data files according to a partition addition judgment instruction and a target file storage path includes: storing the source data files to the target file storage path; in response to the partition addition judgment instruction being a create new partition instruction, executing a new partition operation to obtain a new data partition, and storing the source data files to the new data partition; if the new partition operation fails to be executed, moving the source data files from a temporary storage area to an abnormal file storage path; and deleting the source data files under the source file storage path.

[0105] The instruction to create a new partition may be an instruction to create a new data partition in a data table of the second type of database. The operation to create a new partition may be an operation to create a new data partition in the data table of the second type of database. The abnormal file storage path may be a storage path for storing abnormal files.

[0106] refer to Figure 4 , Figure 4 The following schematically illustrates an overall flow chart for executing a data synchronization task according to an exemplary embodiment of the present disclosure. The thread pool resource can call the HDFS API and use data overwrite mode to store source data files in the target file storage path. For example, a file upload program can upload files from a NAS staging area to HDFS. During the file upload process, a partition addition determination instruction can be used to determine whether to add partitions to the Hive table in the second-type database, automatically adding the corresponding partition data.

[0107] If the partition addition judgment instruction is a create partition instruction, the partition addition operation is performed on the data table, a data partition is added to the second type database, and the source data file is stored in the newly added data partition. If the HDFS upload is successful, the file in the listening path is copied to the target file storage path, and then the file in the listening path is deleted.

[0108] If the partition addition operation fails, the source data files are moved from the temporary storage area (NAS staging area) to the abnormal file storage path (i.e., the NAS abnormal path), and the files in the monitoring path are deleted. The files that failed to be uploaded are moved to the abnormal file storage path so that the data files in the abnormal file storage path can be uploaded again later.

[0109] In an exemplary embodiment of the present disclosure, in response to an abnormal monitoring response instruction of an abnormal file monitoring program, an abnormal file path is determined based on the abnormal monitoring response instruction; the abnormal file under the abnormal file path is moved to a temporary storage area of ​​a first type of database to re-store the abnormal file.

[0110] The abnormal monitoring response instruction may be a response instruction generated by monitoring whether an abnormality occurs during the data synchronization operation. The abnormal file may be a data file that has an abnormality during the data synchronization operation.

[0111] During the execution of the data synchronization task, the file upload may fail due to network transmission factors or other factors. In order to ensure the high fault tolerance of data synchronization, the abnormal file monitoring program 410 can monitor whether there are abnormal files in the data synchronization task. The abnormal file monitoring program 410 can scan the upload abnormal paths of all tasks every specific time period, such as 10s, and move the files therein to the corresponding temporary storage area (i.e., the upload monitoring path) so that they can be consumed and uploaded again. For example, the abnormal file monitoring program 410 can monitor the folders under the NAS abnormal path and move the files that failed to be uploaded to the corresponding NAS staging area so that they can be listened to and consumed by the file upload program again, thereby providing a highly fault-tolerant data synchronization mechanism.

[0112] In an exemplary embodiment of the present disclosure, the number of unuploaded files in the temporary storage area of ​​the first type of database and a pre-configured temporary file threshold are obtained; if the number of unuploaded files is greater than the temporary file threshold, an upload exception instruction is generated; and abnormal alarm information is generated based on the upload exception instruction.

[0113] The number of unuploaded files may be the number of source data files in the temporary storage area that have not yet been uploaded to the second-type database. The temporary file threshold may be a file number value used for comparison with the number of unuploaded files. The upload exception instruction may be a response instruction indicating an abnormality in data synchronization. The abnormality alarm information may be an alarm message indicating an abnormal state during the data synchronization operation.

[0114] During data synchronization tasks, the temporary storage area has a corresponding storage threshold. During task execution, the task listener monitors the number of files of valid file types in the listening path of each task's corresponding NAS staging area, that is, the number of unuploaded files. After determining the number of unuploaded files, it compares the number with the pre-configured temporary file threshold to determine the magnitude. If the number of unuploaded files exceeds the temporary file threshold, an upload exception instruction is generated. Based on the upload exception instruction, an exception alarm is generated, which can be subsequently sent via email or instant messaging.

[0115] In addition, the service status monitoring program 420 can also monitor the program running status of the file upload program. When the status of the file upload program is abnormal, the alarm information can also be sent via email or instant messaging to remind relevant personnel that the data synchronization operation is abnormal.

[0116] In an exemplary embodiment of the present disclosure, a preconfigured second file scanning cycle is obtained; a file scanning operation is performed on data files in the archive file storage path based on the second file scanning cycle to filter out expired files; and expired files are deleted.

[0117] The second file scanning period may be a time period for periodically scanning data files in the target file storage path. The file scanning operation may be an information acquisition operation for acquiring relevant file information of the data files in the target file storage path. Expired files may be data files in the target file storage path whose file retention period exceeds a preset time threshold.

[0118] In order to further improve the fault tolerance of the data synchronization task, the uploaded data files, i.e., the archive files, can also be scanned regularly. Specifically, a pre-configured second file scanning cycle can be obtained, and the archive file listening program 430 can scan the data files under the archive file storage path based on the second file scanning cycle. For example, the archive path listening program can regularly scan the files under the NAS archive path, filter out files that have exceeded the retention period (e.g., more than 24 hours), treat them as expired files, delete the expired files, and release NAS space. Through the above processing steps, the storage status of the data can be grasped in a timely manner, and the waste of storage space can be avoided.

[0119] It should be noted that the terms "first" and "second" used in the present disclosure are only used to distinguish different file scanning cycles and should not impose any limitations on the present disclosure.

[0120] In summary, the data synchronization method disclosed herein, in response to a received data synchronization task, obtains the source file storage path corresponding to the data synchronization task; the source file storage path includes the file storage path in the first type of database; based on the current number of folders under the source file storage path, creates a listening thread object that matches the number of data synchronization tasks; based on the listening thread object, determines the number of files to be uploaded under each folder in the source file storage path, and determines the upload thread object corresponding to each folder according to the number of files to be uploaded; through the upload thread object, stores the source data file under the source file storage path to the target file storage path; the target file storage path includes the file storage path in the second type of database. On the one hand, by uploading data to the target file storage path of the target database through the listening source data file path, a solution for data synchronization between different types of databases is provided, which reduces the load pressure on the source library and the target library. On the other hand, by performing file scanning based on the listening thread object created based on the current number of folders and obtaining the upload thread object corresponding to the number of files for file upload, it is possible to maximize data storage efficiency while avoiding the waste of thread objects. On the other hand, by providing abnormal file monitoring programs, service status monitoring programs, etc., while achieving high fault tolerance for data synchronization tasks, the final consistency of data synchronization in HIVE by importing data files into HDFS is guaranteed.

[0121] It should be noted that although the steps of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in this specific order, or that all steps must be performed to achieve the desired results. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0122] In addition, in this exemplary embodiment, a data synchronization device is also provided. Figure 5 The data synchronization device 500 may include: a parameter acquisition module 510 , a resource creation module 520 , an upload resource creation module 530 and a data storage module 540 .

[0123] Specifically, the parameter acquisition module 510 is used to obtain the source file storage path corresponding to the data synchronization task in response to the received data synchronization task; the source file storage path includes the file storage path in the first type of database; the monitoring resource creation module 520 is used to create a monitoring thread object that matches the number of data synchronization tasks based on the current number of folders under the source file storage path; the upload resource creation module 530 is used to determine the number of files to be uploaded under each folder in the source file storage path based on the monitoring thread object, and determine the upload thread object corresponding to each folder according to the number of files to be uploaded; the data storage module 540 is used to store the source data file under the source file storage path to the target file storage path through the upload thread object; the target file storage path includes the file storage path in the second type of database.

[0124] In an exemplary embodiment of the present disclosure, the data synchronization device 500 also includes a parameter configuration module, which is used to: provide a task configuration interface through an external application, receive task configuration operations based on the task configuration interface; determine the task configuration parameters corresponding to the task configuration operation, and generate a data synchronization task based on the task configuration parameters.

[0125] In an exemplary embodiment of the present disclosure, the monitoring resource creation module 520 includes a monitoring resource creation unit, which is used to: obtain a pre-configured first file scanning cycle; perform a scanning operation on the source file storage path based on the first file scanning cycle to determine the current number of folders under the source file storage path at the current moment; determine the number of monitoring threads that match the data synchronization task based on the current number of folders; and create or destroy monitoring thread objects in real time based on the number of monitoring threads.

[0126] In an exemplary embodiment of the present disclosure, the upload resource creation module 530 includes an upload resource determination unit, which is used to: obtain a pre-configured file number threshold; if the number of files to be uploaded is less than or equal to the file number threshold, then use the listening thread object corresponding to the folder as the upload thread object; if the number of files to be uploaded is greater than the file number threshold, then obtain an idle thread object from the upload thread pool as the upload thread object.

[0127] In an exemplary embodiment of the present disclosure, the thread object acquisition unit includes a thread object acquisition sub-unit, which is used to: split the source data file based on a file quantity threshold to obtain a split data file; and obtain an idle thread object from the upload thread pool as the upload thread object corresponding to the split data file.

[0128] In an exemplary embodiment of the present disclosure, the data storage module 540 includes a data storage unit for: obtaining an upload thread object; and storing the source data file under the source file storage path to the target file storage path by the upload thread object in a data overwrite mode and through a data storage interface.

[0129] In an exemplary embodiment of the present disclosure, the data storage unit includes a data storage sub-unit, which is used to: obtain the source data file under the source file storage path from the temporary storage area of ​​the first type of database; obtain the file upload policy corresponding to the source data file; based on the file upload policy, generate a partition addition judgment instruction corresponding to the source data file in the second type of database; and store the source data file according to the partition addition judgment instruction and the target file storage path.

[0130] In an exemplary embodiment of the present disclosure, the data storage subunit includes a new partition subunit, which is used to: perform a name resolution operation on the source file name of the source data file based on the file upload strategy, and generate a resolution storage path corresponding to the source data file; determine whether the existing storage path of the second type of database contains the resolution storage path; if the existing storage path does not contain the resolution storage path, generate an instruction to create a new partition.

[0131] In an exemplary embodiment of the present disclosure, the data storage sub-unit is configured to execute: storing the source data file to the target file storage path; in response to the partition addition judgment instruction being a create new partition instruction, executing a new partition operation to obtain a new data partition, and storing the source data file to the new data partition; if the new partition operation fails to be executed, moving the source data file from the temporary storage area to the abnormal file storage path; and deleting the source data file under the source file storage path.

[0132] In an exemplary embodiment of the present disclosure, the data synchronization device 500 also includes an abnormal file processing module, which is used to: respond to the abnormal monitoring response instruction of the abnormal file monitoring program, determine the abnormal file path based on the abnormal monitoring response instruction; move the abnormal file under the abnormal file path to the temporary storage area of ​​the first type of database to re-store the abnormal file.

[0133] In an exemplary embodiment of the present disclosure, the data synchronization device 500 also includes an alarm information generation module, which is used to: obtain the number of unuploaded files in the temporary storage area in the first type of database, and a pre-configured temporary file threshold; if the number of unuploaded files is greater than the temporary file threshold, generate an upload exception instruction; and generate abnormal alarm information based on the upload exception instruction.

[0134] In an exemplary embodiment of the present disclosure, the data synchronization device 500 also includes an expired file processing module, which is used to: obtain a pre-configured second file scanning cycle; perform a file scanning operation on the data files under the archive file storage path based on the second file scanning cycle to filter out expired files; and delete expired files.

[0135] The specific details of the virtual modules of the above-mentioned data synchronization devices have been described in detail in the corresponding data synchronization methods, and therefore will not be repeated here.

[0136] It should be noted that although several modules or units of the data synchronization device are mentioned in the detailed description above, such division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in a single module or unit. Conversely, the features and functions of a single module or unit described above can be further divided and embodied by multiple modules or units.

[0137] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0138] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods, or program products. Accordingly, various aspects of the present invention may be implemented as a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or a combination of hardware and software embodiments, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0139] Reference below Figure 6 6 to describe the electronic device 600 according to such an embodiment of the present disclosure. Figure 6 The electronic device 600 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0140] like Figure 6 As shown, electronic device 600 is implemented as a general-purpose computing device. Components of electronic device 600 may include, but are not limited to, the aforementioned at least one processing unit 610, the aforementioned at least one storage unit 620, a bus 630 connecting various system components (including storage unit 620 and processing unit 610), and a display unit 640.

[0141] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0142] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 621 and / or a cache memory unit 622 , and may further include a read-only memory unit (ROM) 623 .

[0143] The storage unit 620 may include a program / utility 624 having a set (at least one) of program modules 625, such program modules 625 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0144] Bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0145] The electronic device 600 can also communicate with one or more external devices 670 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 650. Furthermore, the electronic device 600 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 660. As shown, the network adapter 660 communicates with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 600, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0146] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0147] In exemplary embodiments of the present disclosure, a computer-readable storage medium is also provided, storing a program product capable of implementing the aforementioned methods of this specification. In some possible embodiments, various aspects of the present invention may also be implemented in the form of a program product comprising program code. When the program product is executed on a terminal device, the program code is configured to cause the terminal device to perform the steps according to various exemplary embodiments of the present invention described in the "Exemplary Methods" section of this specification.

[0148] refer to Figure 7 As shown, a program product 700 for implementing the above method according to an embodiment of the present invention is described. The program product 700 may be a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0149] The program product may be implemented in any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0150] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0151] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0152] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0153] Furthermore, the above-described figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention and are not intended to be limiting. It is readily understood that the processes illustrated in the above-described figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0154] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow from the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

[0155] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data synchronization method, characterized in that: include: In response to the received data synchronization task, obtaining a source file storage path corresponding to the data synchronization task; The source file storage path includes a file storage path in a first type of database; Based on the current number of folders under the source file storage path, create a listening thread object that matches the number of data synchronization tasks; Based on the listening thread object, determine the number of files to be uploaded in each folder in the source file storage path, and determine the upload thread object corresponding to each folder according to the number of files to be uploaded; The source data file under the source file storage path is stored in the target file storage path through the upload thread object; The target file storage path includes a file storage path in a second type database.

2. The method according to claim 1, characterized in that The method further comprises: Providing a task configuration interface through an external application and receiving a task configuration operation based on the task configuration interface; Determine task configuration parameters corresponding to the task configuration operation, and generate the data synchronization task based on the task configuration parameters.

3. The method according to claim 2, characterized in that The task configuration parameters include any one or more combinations of source file storage path, target database, target data table, target file storage path, abnormal file storage path, blacklist file list, whitelist file list, upload thread resource number and file upload policy name.

4. The method according to claim 1, wherein The step of creating a listening thread object that matches the number of data synchronization tasks based on the current number of folders under the source file storage path includes: Obtaining a pre-configured first file scanning period; Scanning the source file storage path based on the first file scanning period to determine the number of current folders in the source file storage path at a current moment; Determine the number of listening threads that match the data synchronization task according to the current number of folders; The listening thread object is created or destroyed in real time based on the number of the listening threads.

5. The method according to claim 1, wherein Determining the upload thread object corresponding to each folder according to the number of files to be uploaded includes: Get the pre-configured file number threshold; If the number of files to be uploaded is less than or equal to the file number threshold, the listening thread object corresponding to the folder is used as the uploading thread object; If the number of files to be uploaded is greater than the file number threshold, an idle thread object is obtained from the upload thread pool as the upload thread object.

6. The method according to claim 5, characterized in that The obtaining of an idle thread object from the upload thread pool as the upload thread object includes: Based on the file quantity threshold, the source data file is segmented to obtain segmented data files; From the upload thread pool, the idle thread object is obtained as the upload thread object corresponding to the segmented data file.

7. The method according to claim 1, characterized in that The step of storing the source data file under the source file storage path to the target file storage path through the upload thread object includes: Get the upload thread object; The upload thread object adopts the data overwrite mode and stores the source data file under the source file storage path to the target file storage path through the data storage interface.

8. The method according to claim 7, characterized in that Storing the source data file under the source file storage path to the target file storage path includes: Acquire the source data file under the source file storage path from the temporary storage area of ​​the first type database; Obtaining a file upload policy corresponding to the source data file; Based on the file upload strategy, generating a new partition addition judgment instruction corresponding to the source data file in the second type database; The source data file is stored according to the partition newly added judgment instruction and the target file storage path.

9. The method according to claim 8, characterized in that The generating, based on the file upload policy, a partition addition judgment instruction corresponding to the source data file in the second type database includes: Based on the file upload strategy, performing a name resolution operation on the source file name of the source data file to generate a resolution storage path corresponding to the source data file; determining whether the existing storage paths of the second type database include the parsed storage path; If the existing storage path does not include the parsed storage path, a create new partition instruction is generated.

10. The method according to claim 8, characterized in that The storing of the source data file according to the partition newly added judgment instruction and the target file storage path includes: Storing the source data file in the target file storage path; In response to the partition adding judgment instruction being a create new partition instruction, executing a new partitioning operation to obtain a new data partition, and storing the source data file in the new data partition; If the operation of adding a new partition fails, the source data file is moved from the temporary storage area to the abnormal file storage path; Delete the source data file under the source file storage path.

11. The method according to any one of claims 1 to 10, characterized in that: The method further comprises: In response to an abnormal monitoring response instruction of the abnormal file monitoring program, determining an abnormal file path based on the abnormal monitoring response instruction; The abnormal files under the abnormal file path are moved to a temporary storage area of ​​the first type database to restore the abnormal files.

12. The method according to any one of claims 1 to 10, characterized in that: The method further comprises: Obtaining the number of unuploaded files in the temporary storage area of ​​the first type of database and a pre-configured temporary file threshold; If the number of the unuploaded files is greater than the temporary file threshold, an upload exception instruction is generated; Generate abnormal warning information based on the upload abnormal instruction.

13. The method according to any one of claims 1 to 10, characterized in that: The method further comprises: Obtaining a pre-configured second file scanning period; Performing a file scanning operation on the data files in the archive file storage path based on the second file scanning period to filter out expired files; Delete the expired files.

14. A data synchronization device, characterized in that: include: A parameter acquisition module, configured to, in response to a received data synchronization task, acquire a source file storage path corresponding to the data synchronization task; The source file storage path includes a file storage path in a first type of database; A monitoring resource creation module is used to create a monitoring thread object that matches the number of data synchronization tasks based on the current number of folders under the source file storage path; An upload resource creation module is used to determine the number of files to be uploaded in each folder in the source file storage path based on the monitoring thread object, and determine the upload thread object corresponding to each folder according to the number of files to be uploaded; A data storage module, configured to store the source data file under the source file storage path to the target file storage path through the upload thread object; The target file storage path includes a file storage path in a second type database.

15. An electronic device, characterized in that: include: processor; as well as A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the data synchronization method according to any one of claims 1 to 13. 16 . A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the data synchronization method according to claim 1 is implemented.

Images