Data distribution method, device and system, equipment, storage medium and product

Abstract

The invention discloses a data distribution method, device, system and equipment, a storage medium and a product, and the method comprises the steps: initializing a thread pool according to an available computing power resource of a target source server, dividing a source data position list sent by a metadata server into a plurality of sub-batch lists, creating a data migration task according to the sub-batch lists, and transmitting the data migration task to the target source server; and submitting the tasks to a thread pool, independently executing the tasks by threads in the thread pool, and migrating the to-be-processed data in the data source cluster to a data management platform. By adopting the embodiment of the invention, the position of the to-be-processed data in the source data cluster can be accurately found according to the source data position list generated by the metadata server; according to the method, the source data position list is split, and the thread pool is initialized based on the available computing power resource of the target source server, so that data distribution in batches is realized, the high efficiency of data migration and the concurrent data processing capability are ensured, and the data distribution efficiency is improved from the perspective of the data source cluster.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data distribution method, device, system, equipment, storage medium and product. Background Art

[0002] Since data sources are stored in different platforms or systems, the data storage platform needs to obtain the information that needs to be processed from different databases or systems (such as Ceph cluster, MySQL database, HBase database, etc.), and data needs to be distributed frequently between each cluster and the data storage platform.

[0003] The prior art has proposed some methods to achieve data distribution for multiple databases. The data source splits the original data to be distributed according to the network performance and distributes the split original data. This distribution technology only considers the performance of the distribution channel and cannot effectively exert the performance of the data source system, thereby failing to effectively improve the data distribution efficiency from the perspective of the data source system. Summary of the invention

[0004] The present invention provides a data distribution method, device, system, equipment, storage medium and product to solve the defect in the prior art that the data distribution efficiency cannot be fully improved from the perspective of the data source system.

[0005] To achieve the above object, an embodiment of the present invention provides a data distribution method, which is applied to a data source cluster. The method includes:

[0006] Initialize the pre-set thread pool based on the available computing resources of the target source server;

[0007] Divide the source data location list sent from the metadata server to obtain multiple sub-batch lists; wherein the source data location list is determined by the metadata server based on the mapping relationship between metadata and data source location according to the data acquisition request initiated by the data management platform; the data acquisition request carries metadata corresponding to the data to be processed, and the data to be processed is the source data requested to be obtained by the data management platform;

[0008] Creating a data migration task according to the sub-batch list, and submitting the data migration task to the thread pool;

[0009] Each of the data migration tasks is independently executed based on the thread pool to migrate the data to be processed in the data source cluster to the data management platform; wherein the target source server is a server in the data source cluster that stores the data to be processed.

[0010] As a preferred solution, the data acquisition request is initiated by the data management platform in the following manner:

[0011] Verifying the configuration information of the data management platform;

[0012] After the configuration information is verified, a request body in JSON format is constructed, wherein the request body includes metadata corresponding to the data to be processed;

[0013] The data acquisition request carrying the request body is sent to the metadata server.

[0014] As a preferred solution, the source data location list is an object key list, and the object key list is generated by the metadata server and sent to the data source cluster in the following manner:

[0015] Receiving a data acquisition request initiated by the data management platform;

[0016] Parsing the request body in the data acquisition request to obtain metadata corresponding to the data to be processed;

[0017] Creating an SQL query statement according to metadata corresponding to the data to be processed;

[0018] Establishing communication with a database connection pool to obtain a database connection, wherein the database connection is a connection between the metadata server and a database storing metadata;

[0019] Use PreparedStatement to execute the SQL query statement on the database connection to obtain a query result set;

[0020] Traversing the query result set, and extracting the object key in the query result set;

[0021] All object keys are stored in an object key list and synchronized to the data source cluster.

[0022] As a preferred solution, after independently executing each of the data migration tasks based on the thread pool to migrate the to-be-processed data in the data source cluster to the data management platform, the method further includes:

[0023] During the migration of a single pending data in the data source cluster, if the migration is abnormal, the migration operation of the pending data with the migration abnormality is re-executed until the pending data is successfully migrated or the number of re-migrated times reaches the set number of retries;

[0024] Each time a data migration task is successfully executed, a count value in a preset migration counter is increased by one;

[0025] After completing all the data migration tasks, closing the thread pool in the data source cluster;

[0026] All migration results are summarized and fed back to the data management platform, and the migration results include information on the success and failure of data migration.

[0027] To achieve the above objective, in a second aspect, an embodiment of the present invention provides a data distribution device, the device comprising:

[0028] The thread pool initialization module is used to initialize the pre-set thread pool according to the available computing resources of the target source server;

[0029] A list partitioning module is used to partition the source data location list sent from the metadata server to obtain multiple sub-batch lists; wherein the source data location list is determined by the metadata server based on the mapping relationship between metadata and data source location according to the data acquisition request initiated by the data management platform; the data acquisition request carries metadata corresponding to the data to be processed, and the data to be processed is the source data requested to be obtained by the data management platform;

[0030] A task creation module, used to create a data migration task according to the sub-batch list, and submit the data migration task to the thread pool;

[0031] A task execution module is used to independently execute each of the data migration tasks based on the thread pool to migrate the unprocessed data in the data source cluster to the data management platform; wherein the target source server is a server in the data source cluster that stores the unprocessed data.

[0032] To achieve the above-mentioned purpose, in a third aspect, an embodiment of the present invention further provides a data distribution system, including: a data source cluster, a metadata server and a data management platform;

[0033] The data management platform is used to generate a data acquisition request and send it to the metadata server; wherein the data acquisition request carries metadata corresponding to the data to be processed, and the data to be processed is the source data requested to be acquired by the data management platform;

[0034] The metadata server is used to receive the data acquisition request, and based on the mapping relationship between the metadata and the data source location, determine the source data location list according to the data acquisition request to send to the data source cluster;

[0035] The data source cluster is used to initialize a preset thread pool according to the available computing resources of the target source server; wherein the target source server is a server in the data source cluster that stores the data to be processed;

[0036] The data source cluster is further used to receive the source data location list, divide the source data location list to obtain multiple sub-batch lists, create data migration tasks according to the sub-batch lists, and submit the data migration tasks to the thread pool, and independently execute each data migration task based on the thread pool to migrate the pending data in the data source cluster to the data management platform.

[0037] As a preferred solution, the data source cluster is also used for:

[0038] During the migration of a single pending data in the data source cluster, if the migration is abnormal, the migration operation of the pending data with the migration abnormality is re-executed until the pending data is successfully migrated or the number of re-migrated times reaches the set number of retries;

[0039] Each time a data migration task is successfully executed, a count value in a preset migration counter is increased by one;

[0040] After completing all the data migration tasks, closing the thread pool in the data source cluster;

[0041] All migration results are summarized and fed back to the data management platform, and the migration results include information on the success and failure of data migration.

[0042] To achieve the above-mentioned purpose, an embodiment of the present invention also provides a data distribution device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, it implements the data distribution method as described in any one of the first aspects.

[0043] To achieve the above-mentioned purpose, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the data distribution method as described in any one of the first aspects is implemented.

[0044] To achieve the above-mentioned purpose, an embodiment of the present invention further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the data distribution method as described in any one of the first aspects.

[0045] Compared with the prior art, the implementation of the embodiments of the present invention has the following beneficial effects: based on the source data location list generated by the metadata server, the location of the data to be processed in the source data cluster can be accurately found; the source data location list is split, and the thread pool is initialized based on the available computing resources of the target source server to realize batch data distribution, thereby ensuring the high efficiency of data migration and the ability to process data concurrently, and improving the data distribution efficiency from the perspective of the data source cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical features of the embodiments of the present invention, the drawings required for use in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 It is a flowchart of a data distribution method provided by an embodiment of the present invention;

[0048] Figure 2 It is a detailed flow chart of a data distribution method provided by an embodiment of the present invention;

[0049] Figure 3 It is a schematic diagram of the result feedback refinement process provided by the preferred embodiment of the present invention;

[0050] Figure 4 is a structural schematic diagram of a data distribution device provided by an embodiment of the present invention;

[0051] Figure 5 It is a structural diagram of a data distribution system provided by an embodiment of the present invention;

[0052] Figure 6 It is a structural diagram of a data distribution device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] In the description of the present invention, it should be noted that the step numbers in the text are only for the convenience of explanation of specific embodiments and do not serve to limit the order in which the steps are executed. The terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0055] See also Figure 1 , Figure 11 is a flow chart of a data distribution method provided by an embodiment of the present invention. Specifically, the data distribution method comprises steps S11 to S14:

[0056] S11: Initialize a pre-set thread pool according to the available computing resources of the target source server.

[0057] It is worth noting that the thread pool is not newly created, but may have stored unfinished tasks. Therefore, the thread pool needs to be initialized and all tasks cleared so that when a new task arrives, it can immediately use the idle thread to start working without waiting for the creation of a new thread. In a web server, each client request can be regarded as an independent task. By using thread pool technology, the server can handle concurrent requests more efficiently, respond quickly even under high load conditions, and make reasonable use of network resources to avoid jamming caused by excessive CPU usage.

[0058] S12: Divide the source data location list sent from the metadata server into multiple sub-batch lists; wherein the source data location list is determined by the metadata server based on a mapping relationship between metadata and data source location and according to a data acquisition request initiated by a data management platform; the data acquisition request carries metadata corresponding to the data to be processed, and the data to be processed is the source data requested to be obtained by the data management platform.

[0059] Specifically, the metadata server will obtain a complete list of source data locations, which contains the location information of all data to be processed (such as file path, network address, file name, etc.). Based on the currently available computing resources (such as the number of nodes in the cluster, the memory capacity on each node, the number of CPU cores, etc.) and business requirements (such as the time limit for task completion), the workload between sub-batches is balanced to avoid overloading some nodes while other nodes are idle. The simplest strategy is to evenly divide the list by a fixed size or number of data items so that each sub-batch contains the same number of data blocks. If the data itself has some structural characteristics (such as data sorted by time), it can be divided according to the time characteristics so that the relationship between the data can be better utilized in subsequent processing.

[0060] It is worth noting that all metadata can be stored in one metadata server or in a metadata cluster formed by multiple metadata servers, which is not limited here.

[0061] S13: Creating a data migration task according to the sub-batch list, and submitting the data migration task to the thread pool.

[0062] The divided sub-batch lists are assigned to different threads, and each thread is responsible for processing the sub-batch list data it obtains. It is worth noting that the data distribution task is dynamic, and a real-time monitoring system can be set up to adjust the data distribution strategy in real time according to indicators such as execution efficiency, latency, throughput, and error rate during the data distribution process.

[0063] S14: independently executing each of the data migration tasks based on the thread pool to migrate the data to be processed in the data source cluster to the data management platform; wherein the target source server is a server in the data source cluster that stores the data to be processed.

[0064] As an example, see Figure 2 The detailed flow chart of the data distribution method is as follows. First, you need to evaluate the available hardware resources, such as the number of CPU cores and memory capacity, to determine the maximum number of threads that can run simultaneously. Use the Java Executors class to create a fixed-size thread pool. For example, you can initialize an ExecutorService (thread pool) containing a specified number of threads. The sizes of thread pools 1 to n do not need to be evenly divided. Consider whether you need to set other parameters of the thread pool, such as the queue size, to ensure that the thread pool can effectively manage tasks. Logically divide the object key list into multiple batches and sub-batches. Determine the number of objects that each batch should contain based on factors such as the performance characteristics of a single operation (such as I / O speed) and the throughput limit of the data management platform. Divide the entire object key list into several batches, and each batch is further divided into smaller sub-batches to avoid overload caused by submitting too many requests at one time. Considering factors such as network latency and bandwidth utilization, you may also need to adjust the size of batches and sub-batches to achieve the best data migration effect. For each sub-batch, create and submit a data migration task to the thread pool. Create a separate task unit for each sub-batch, which contains all the logic required to obtain data from the data source cluster and migrate it to the data management platform. Use the ExecutorService created previously to submit these tasks to the thread pool by calling its execute method. The executor is responsible for managing and scheduling these tasks to achieve concurrent execution.

[0065] Compared with the prior art, this implementation performs task segmentation based on the original data information provided by metadata, divides the original data to be distributed into blocks according to rules, obtains multiple subtasks, and assigns these subtasks to different thread pools to execute the distribution tasks, thereby efficiently utilizing the concurrent processing capabilities of the data source cluster and improving data processing efficiency.

[0066] In one implementation, the data acquisition request is initiated by the data management platform in the following manner:

[0067] Verify the configuration information of the data management platform; specifically, before initiating a request, the data management platform must first check the relevant parameters of its own configuration file, such as API keys, access tokens, etc.

[0068] After the configuration information is verified, a request body in JSON format is constructed, wherein the request body includes metadata corresponding to the data to be processed;

[0069] The data acquisition request carrying the request body is sent to the metadata server.

[0070] It is worth noting that metadata refers to metadata composition and logical relationships, and is used to express shared data attribute information. By sorting out shared distribution data types, defining metadata node types and attribute ranges, and forming metadata information of shared data in JSON format. The data management platform constructs a JSON request body containing necessary metadata based on the type of data to be processed. JSON supports a variety of data types, including strings, numbers, Boolean values, arrays, etc., and can be used to define required parameters such as data sources, fields, time ranges, etc.

[0071] As an improvement to the above implementation, the source data location list is an object key list, and the object key list is generated by the metadata server and sent to the data source cluster in the following manner:

[0072] Receiving a data acquisition request initiated by the data management platform;

[0073] Parsing the request body in the data acquisition request to obtain metadata corresponding to the data to be processed;

[0074] Creating an SQL query statement according to metadata corresponding to the data to be processed;

[0075] Establishing communication with a database connection pool to obtain a database connection, wherein the database connection is a connection between the metadata server and a database storing metadata;

[0076] Use PreparedStatement to execute the SQL query statement on the database connection to obtain a query result set;

[0077] Traversing the query result set, and extracting the object key in the query result set;

[0078] All object keys are stored in an object key list and synchronized to the data source cluster.

[0079] As an example, suppose that in a procurement system, the data management platform needs to integrate relevant data from multiple heterogeneous data sources (such as order databases and user behavior logs). In this process, the metadata server plays an important intermediary role between the data management platform and different data sources. When the data management platform requests to obtain the shopping records of all users in the past month, the request clearly specifies the time range and data type of the required data. After receiving the request, the metadata server first parses the relevant information in the request body to determine the specific query parameters. Based on the metadata obtained by parsing, the server further constructs the corresponding SQL query statement to accurately locate the required shopping records. The metadata server directly establishes a connection with the database storing metadata through the database connection pool. Therefore, it does not need to create a new connection when receiving a request. The connection pool provides an already established connection, which greatly reduces the waiting time. In order to ensure the security of the query operation and prevent the risk of SQL injection attacks, the query statement is executed in a pre-compiled manner. After execution, the system returns a result containing all relevant order records in the queried time period. The metadata server traverses the result set and extracts the unique identifier of each order (i.e., object key, such as order ID). All extracted object keys will be summarized into a list and synchronized to each node in the data source cluster to ensure that each component in the cluster can identify the specific order data involved in the current analysis task.

[0080] In one embodiment, after independently executing each of the data migration tasks based on the thread pool to migrate the to-be-processed data in the data source cluster to the data management platform, the method further includes:

[0081] During the migration of a single pending data in the data source cluster, if the migration is abnormal, the migration operation of the pending data with the migration abnormality is re-executed until the pending data is successfully migrated or the number of re-migrated times reaches the set number of retries;

[0082] Each time a data migration task is successfully executed, a count value in a preset migration counter is increased by one;

[0083] After completing all the data migration tasks, closing the thread pool in the data source cluster;

[0084] All migration results are summarized and fed back to the data management platform, and the migration results include information on the success and failure of data migration.

[0085] Specifically, the types of exceptions may include insufficient permissions, network anomalies, missing data, data errors, etc. Each error type has specific retry rules. If multiple retries still cannot solve the problem, a limited number of retries must be set to avoid occupying resources for a long time. Based on exception capture technology, the information about distribution failure is fed back to the data management platform. For common problems and errors, automatic error correction and retry are performed; for new errors, they are recorded in log files and uploaded to the work order system for administrators to study. For the convenience of management, a counter is set in the data source cluster to record the number of heterogeneous data distributed from different data sources to the data management platform. After successful distribution, the amount of distributed data is returned, and the counter quantity information is fed back to the data management platform after the migration is completed. The data management platform compares the amount of data to be distributed according to the previous metadata statistics. When the data amount is consistent, it indicates that the distribution is successful.

[0086] As an example, see Figure 3 The result feedback refinement process diagram is shown in Figure 1. First, a reasonable number of retries and interval time are determined based on factors such as network stability and service availability. When performing migration operations, use try-catch blocks to capture possible exceptions. If an exception is encountered and the current number of attempts is less than the preset maximum number of retries, wait for a specified period of time before trying again. According to the exception type and the preset retry logic. Different exceptions may require different processing methods. For example, for temporary network problems, data loss problems, etc., retries are performed immediately, while for non-temporary errors such as insufficient permissions, no retries are required. Only specific types of exceptions trigger the retry mechanism. You can determine whether to retry by checking the exception instance or its message content. Relevant information, such as exception details, number of retries, etc., should be recorded before and after each retry, so that feedback users can analyze abnormal behavior. For the completion of each task in the thread pool, check its result and increase the migration counter if successful. The results of task execution can be obtained through methods provided by the Future interface, such as get(). If the task is successfully completed, the method returns the result, otherwise it displays an exception. Whenever a task is successfully completed, update the global migration counter or other state variables to track progress.

[0087] If the task fails, in addition to possible retries, the failure counter can also be set and updated and detailed error information can be recorded. After completing all migration tasks, call the shutdown() method of ExecutorService to shut down the thread pool and wait for all tasks to end. It is worth noting that it will not stop all running tasks immediately, but will no longer accept new task submissions. If it fails to shut down normally within a reasonable time, you can consider using shutdownNow() to force the thread pool to shut down, but this may cause unfinished tasks to be interrupted. Finally, summarize the migration results, including detailed information on success and failure, and feedback to the caller or record it to the log system. The data management platform traverses the results of all tasks and counts the number of successes and failures. Prepare a detailed report including the total number of objects, the number of objects successfully migrated to the data management platform, the list of failed objects and their specific error information, and send the above report to the relevant monitoring system or print it directly to the console / file log to ensure that relevant personnel can understand the status of the migration in a timely manner.

[0088] Compared with the prior art, an embodiment of the present invention provides a data distribution method, which first initializes a pre-set thread pool according to the available computing resources of the target source server; secondly, divides the source data location list sent from the metadata server to obtain multiple sub-batch lists; wherein the source data location list is determined by the metadata server based on the mapping relationship between metadata and data source location and according to a data acquisition request initiated by a data management platform; the data acquisition request carries metadata corresponding to the data to be processed, and the data to be processed is the source data requested to be obtained by the data management platform; then, a data migration task is created according to the sub-batch list, and the data migration task is submitted to the thread pool; finally, each of the data migration tasks is independently executed based on the thread pool to migrate the data to be processed in the data source cluster to the data management platform; wherein the target source server is a server in the data source cluster that stores the data to be processed. It can be seen that the embodiment of the present invention can accurately find the location of the data to be processed in the source data cluster based on the source data location list generated by the metadata server; split the source data location list, and initialize the thread pool based on the available computing resources of the target source server to realize batch data distribution, thereby ensuring the high efficiency of data migration and the ability to process data concurrently, and improving the data distribution efficiency from the perspective of the data source cluster.

[0089] See also Figure 4 The embodiment of the present invention further provides a data distribution device 20, comprising:

[0090] The thread pool initialization module 21 is used to initialize a preset thread pool according to the available computing resources of the target source server;

[0091] A list partitioning module 22 is used to partition the source data location list sent from the metadata server to obtain multiple sub-batch lists; wherein the source data location list is determined by the metadata server based on the mapping relationship between metadata and data source location according to the data acquisition request initiated by the data management platform; the data acquisition request carries metadata corresponding to the data to be processed, and the data to be processed is the source data requested to be obtained by the data management platform;

[0092] A task creation module 23, used to create a data migration task according to the sub-batch list, and submit the data migration task to the thread pool;

[0093] The data distribution module 24 is used to independently execute each of the data migration tasks based on the thread pool to migrate the unprocessed data in the data source cluster to the data management platform; wherein the target source server is a server in the data source cluster that stores the unprocessed data.

[0094] It is worth noting that the working principle of the data distribution device provided in the above embodiments can refer to the working process of the data distribution method provided in any of the above embodiments, which will not be described in detail here.

[0095] Compared with the prior art, the data distribution device provided in the embodiment of the present invention can accurately find the location of the data to be processed in the source data cluster based on the source data location list generated by the metadata server; split the source data location list, and initialize the thread pool based on the available computing power resources of the target source server to realize batch data distribution, thereby ensuring the high efficiency of data migration and the concurrent data processing capability, and improving the data distribution efficiency from the perspective of the data source cluster.

[0096] See also Figure 5 , an embodiment of the present invention also provides a data distribution system, including a data source cluster 1, a metadata server 2 and a data management platform 3;

[0097] The data management platform 3 is used to generate a data acquisition request and send it to the metadata server 2; wherein the data acquisition request carries metadata corresponding to the data to be processed, and the data to be processed is the source data requested to be acquired by the data management platform 3;

[0098] The metadata server 2 is used to receive the data acquisition request, and based on the mapping relationship between the metadata and the data source location, determine the source data location list according to the data acquisition request, and send it to the data source cluster 1;

[0099] The data source cluster 1 is used to initialize a preset thread pool according to the available computing resources of the target source server; wherein the target source server is a server in the data source cluster 1 that stores the data to be processed;

[0100] The data source cluster 1 is further used to receive the source data location list, divide the source data location list into multiple sub-batch lists, create a data migration task according to the sub-batch list, and submit the data migration task to the thread pool, and independently execute each data migration task based on the thread pool to migrate the pending data in the data source cluster 1 to the data management platform 3.

[0101] In one implementation, the data source cluster 1 is further used for:

[0102] During the migration of a single pending data in the data source cluster 1, if the migration is abnormal, the migration operation of the pending data with the migration abnormality is re-executed until the pending data is successfully migrated or the number of re-migrated times reaches the set number of retries;

[0103] Each time a data migration task is successfully executed, a count value in a preset migration counter is increased by one;

[0104] After completing all the data migration tasks, close the thread pool of the data source cluster 1;

[0105] All migration results are summarized and fed back to the data management platform 3, and the migration results include information on the success and failure of data migration.

[0106] See also Figure 6 The embodiment of the present invention further provides a data distribution device 30, including a processor 31, a memory 32, and a computer program stored in the memory and configured to be executed by the processor 31. When the processor 31 executes the computer program, the steps in the above data distribution method embodiment are implemented, for example Figure 1 S11 to S14 in the above description; or, the processor 31 implements the functions of each module in the above-mentioned device embodiments when executing the computer program.

[0107] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory 32 and executed by the processor 31 to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the data distribution device. For example, the computer program may be divided into multiple modules, and the specific functions of each module are as follows:

[0108] The thread pool initialization module 21 is used to initialize a preset thread pool according to the available computing resources of the target source server;

[0109] A list partitioning module 22 is used to partition the source data location list sent from the metadata server to obtain multiple sub-batch lists; wherein the source data location list is determined by the metadata server based on the mapping relationship between metadata and data source location according to the data acquisition request initiated by the data management platform; the data acquisition request carries metadata corresponding to the data to be processed, and the data to be processed is the source data requested to be obtained by the data management platform;

[0110] A task creation module 23, used to create a data migration task according to the sub-batch list, and submit the data migration task to the thread pool;

[0111] The data distribution module 24 is used to independently execute each of the data migration tasks based on the thread pool to migrate the unprocessed data in the data source cluster to the data management platform; wherein the target source server is a server in the data source cluster that stores the unprocessed data.

[0112] The specific working process of each module can refer to the working process of the data distribution device described in the above embodiment, which will not be repeated here.

[0113] The data distribution device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The data distribution device may include, but is not limited to, a processor 31 and a memory 32. Those skilled in the art will appreciate that the data distribution device may also include an input / output device, a network access device, a bus, etc.

[0114] The processor 31 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 31 is the control center of the data distribution device, and uses various interfaces and lines to connect various parts of the entire data distribution device.

[0115] The memory 32 can be used to store the computer program and / or module. The processor 31 realizes various functions of the data distribution device by running or executing the computer program and / or module stored in the memory 32 and calling the data stored in the memory 32. The memory 32 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as an image playback function, etc.); the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory 32 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0116] Wherein, if the module integrated in the data distribution device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor 31. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0117] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the data distribution method as described in any of the above embodiments when executed by a processor.

[0118] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A data distribution method, characterized in that: Applicable to data source clusters, including: Initialize the pre-set thread pool based on the available computing resources of the target source server; Divide the source data location list sent from the metadata server to obtain multiple sub-batch lists; wherein the source data location list is determined by the metadata server based on the mapping relationship between metadata and data source location according to the data acquisition request initiated by the data management platform; the data acquisition request carries metadata corresponding to the data to be processed, and the data to be processed is the source data requested to be obtained by the data management platform; Creating a data migration task according to the sub-batch list, and submitting the data migration task to the thread pool; Each of the data migration tasks is independently executed based on the thread pool to migrate the data to be processed in the data source cluster to the data management platform; wherein the target source server is a server in the data source cluster that stores the data to be processed.

2. The data distribution method according to claim 1, characterized in that: The data acquisition request is initiated by the data management platform in the following manner: Verifying the configuration information of the data management platform; After the configuration information is verified, a request body in JSON format is constructed, wherein the request body includes metadata corresponding to the data to be processed; The data acquisition request carrying the request body is sent to the metadata server.

3. The data distribution method according to claim 2, characterized in that: The source data location list is an object key list, which is generated by the metadata server and sent to the data source cluster in the following manner: Receiving a data acquisition request initiated by the data management platform; Parsing the request body in the data acquisition request to obtain metadata corresponding to the data to be processed; Creating an SQL query statement according to metadata corresponding to the data to be processed; Establishing communication with a database connection pool to obtain a database connection, wherein the database connection is a connection between the metadata server and a database storing metadata; Use PreparedStatement to execute the SQL query statement on the database connection to obtain a query result set; Traversing the query result set, and extracting the object key in the query result set; All object keys are stored in an object key list and synchronized to the data source cluster.

4. The data distribution method according to claim 1, wherein: After independently executing each of the data migration tasks based on the thread pool to migrate the to-be-processed data in the data source cluster to the data management platform, the method further includes: During the migration of a single pending data in the data source cluster, if the migration is abnormal, the migration operation of the pending data with the migration abnormality is re-executed until the pending data is successfully migrated or the number of re-migrated times reaches the set number of retries; Each time a data migration task is successfully executed, a count value in a preset migration counter is increased by one; After completing all the data migration tasks, closing the thread pool in the data source cluster; All migration results are summarized and fed back to the data management platform, and the migration results include information on the success and failure of data migration.

5. A data distribution device, characterized in that: include: The thread pool initialization module is used to initialize the pre-set thread pool according to the available computing resources of the target source server; A list partitioning module is used to partition the source data location list sent from the metadata server to obtain multiple sub-batch lists; wherein the source data location list is determined by the metadata server based on the mapping relationship between metadata and data source location according to the data acquisition request initiated by the data management platform; the data acquisition request carries metadata corresponding to the data to be processed, and the data to be processed is the source data requested to be obtained by the data management platform; A task creation module, used for creating a data migration task according to the sub-batch list, and submitting the data migration task to the thread pool; A data distribution module is used to independently execute each of the data migration tasks based on the thread pool to migrate the unprocessed data in the data source cluster to the data management platform; wherein the target source server is a server in the data source cluster that stores the unprocessed data.

6. A data distribution system, characterized in that: Includes data source cluster, metadata server and data management platform; The data management platform is used to generate a data acquisition request and send it to the metadata server; wherein the data acquisition request carries metadata corresponding to target data, and the target data is the source data requested to be acquired by the data management platform; The metadata server is used to receive the data acquisition request, and based on the mapping relationship between the metadata and the data source location, determine the source data location list according to the data acquisition request to send to the data source cluster; The data source cluster is used to initialize a preset thread pool according to the available computing resources of the target source server; wherein the target source server is a server in the data source cluster that stores the target data; The data source cluster is further used to receive the source data location list, divide the source data location list into multiple sub-batch lists, create data migration tasks according to the sub-batch lists, and submit the data migration tasks to the thread pool, and independently execute each data migration task based on the thread pool to migrate the target data in the data source cluster to the data management platform.

7. The data distribution system according to claim 6, characterized in that: The data source cluster is also used for: During the migration of a single pending data in the data source cluster, if the migration is abnormal, the migration operation of the pending data with the migration abnormality is re-executed until the pending data is successfully migrated or the number of re-migrated times reaches the set number of retries; Each time a data migration task is successfully executed, a count value in a preset migration counter is increased by one; After completing all the data migration tasks, closing the thread pool in the data source cluster; All migration results are summarized and fed back to the data management platform, and the migration results include information on the success and failure of data migration.

8. A data distribution device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the data distribution method according to any one of claims 1 to 4 when executing the computer program.

9. A computer-readable medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the data distribution method according to any one of claims 1 to 4.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the data distribution method according to any one of claims 1 to 4 is implemented.

Images