Distributed data backhaul method, device, medium, and equipment

Abstract

The present invention provides a distributed data return method and apparatus, medium, and equipment. The method includes: collecting model data through a data collection service and saving it to a database; generating a data push task based on scheduled task execution information through a data push service, adding the data push task to a thread queue, and adding data table information to a blocking queue; executing at least one data push task in a thread queue in parallel through at least one thread in a thread pool; obtaining the corresponding data table in the database based on the data table information retrieved by each thread; determining incremental data through a data push service, encapsulating the incremental data, obtaining a push message, and sending the push message to a Kafka cluster; pulling push messages from the Kafka cluster through a data receiving service, saving the incremental data to a data lake, and submitting a pull offset to the Kafka cluster. An embodiment of the present invention provides an efficient, reliable, and secure model data return solution.

Description

Technical Field

[0001] The present invention relates to the field of data storage technology, and in particular to a distributed data return method and apparatus, medium, and equipment. Background Art

[0002] The model data generated by each distributed cloud center needs to be centrally stored to facilitate relevant analysis using the model data from all distributed cloud centers. Therefore, it is necessary to provide a solution for transmitting distributed model data back to meet this demand. Summary of the Invention

[0003] In response to at least one of the above technical problems, embodiments of the present invention provide a distributed data backhaul method, apparatus, medium, and device.

[0004] According to a first aspect, an embodiment of the present invention provides a distributed data backhaul method comprising:

[0005] Collecting model data generated by the large model through a data acquisition service, and saving the model data in a database connected to the data acquisition service; wherein the database is a local database of the distributed cloud center;

[0006] Generate a data push task through the data push service according to pre-configured scheduled task execution information, add the data push task to the thread queue of the thread pool, and add the data table information of the model data in the database to the blocking queue; execute at least one data push task in the thread queue in parallel through at least one thread in the thread pool, and each thread is responsible for retrieving the corresponding data table information in the blocking queue;

[0007] Establishing a connection channel with the database through the data push service, obtaining the corresponding data table in the database through the connection channel according to the data table information retrieved by each thread, and closing the connection channel;

[0008] Determine the incremental data in the data table through the data push service, encapsulate the incremental data to obtain a push message, and send the push message to the Kafka cluster;

[0009] Pull the push message from the Kafka cluster by subscribing to the data receiving service of the corresponding topic, save the incremental data in the pulled push message into the data lake, and submit its own pull offset to the Kafka cluster.

[0010] In one embodiment, before collecting the model data generated by the large model through the data collection service, the method further includes:

[0011] A data table is created in the database, and the table name of the data table created in the database is the same as the table name of the corresponding data table in the data lake, and the structured table fields of the data table created in the database are the same as the structured table fields of the corresponding data table in the data lake.

[0012] In one embodiment, before generating the data push task by the data push service according to the pre-configured scheduled task execution information, the method further includes:

[0013] Configuring first relevant information corresponding to the data push service; wherein the first relevant information includes information about the data table created in the database, information about the Kafka cluster to be pushed to, published topic information, the number of threads in the thread pool, and scheduled task execution information;

[0014] Starting the data push service;

[0015] Call the initialization method so that the initialization method creates the blocking queue, creates a thread pool according to the number of threads in the first relevant information, creates the thread queue in the thread pool, and publishes the topic information in the first relevant information; wherein, the thread queue is used to store the data push task.

[0016] In one embodiment, the method further comprises:

[0017] Configure the second relevant information corresponding to the data receiving service; wherein the second relevant information includes information about the data lake, information about the Kafka cluster to be consumed, and information about the subscribed topic;

[0018] The data receiving service is started, and the subject information in the second related information is subscribed.

[0019] In one embodiment, determining the incremental data in the data table through the data push service includes:

[0020] The timestamp in the last push information is obtained through the data push service; based on the timestamp, the data generated in the data table after the last push operation is used as the incremental data, and the timestamp of this push operation is recorded in this push information.

[0021] In one embodiment, the process of saving the incremental data in the pulled push message into the data lake by subscribing to the data receiving service of the corresponding topic includes:

[0022] The push message is structurally parsed through the data receiving service; if the parsing is normal, the incremental data obtained by the parsing is written in batches into the corresponding data table in the data lake through the transaction control mode.

[0023] In one embodiment, the method further comprises:

[0024] If all push messages stored in the Kafka cluster are pulled and saved to the data lake through the data receiving service, the data receiving service is closed to release the used resources.

[0025] According to a second aspect, an embodiment of the present invention provides a distributed data backhaul device including:

[0026] A data acquisition module, configured to acquire model data generated by a large model through a data acquisition service, and save the model data to a database connected to the data acquisition service; wherein the database is a local database of a distributed cloud center;

[0027] The data push module is used to generate a data push task according to pre-configured scheduled task execution information through the data push service, add the data push task to the thread queue of the thread pool, and add the data table information of the model data in the database to the blocking queue; execute at least one data push task in the thread queue in parallel through at least one thread in the thread pool, and each thread is responsible for retrieving the corresponding data table information in the blocking queue; establish a connection channel with the database through the data push service, obtain the corresponding data table in the database through the connection channel according to the data table information retrieved by each thread, and close the connection channel; determine the incremental data in the data table through the data push service, encapsulate the incremental data, obtain a push message, and send the push message to the Kafka cluster;

[0028] The data receiving module is used to pull the push message from the Kafka cluster by subscribing to the data receiving service of the corresponding topic, save the incremental data in the pulled push message to the data lake, and submit its own pull offset to the Kafka cluster.

[0029] According to a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method provided in the first aspect.

[0030] According to a fourth aspect, an embodiment of the present invention provides a computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method provided in the first aspect is implemented.

[0031] The distributed data transmission method, apparatus, medium, and device provided by the embodiments of the present invention involve a data collection service that collects model data and persistently stores it in a local database in a cloud center. The data push service then initiates multi-threaded access to the local database. After reading the data, the data push service acts as a message producer and sends the model data to a Kafka cluster via an established network link. The data receiving service acts as a consumer, reading messages from the Kafka cluster and, after completing the read, persisting the model data in a data lake. As can be seen, the embodiments of the present invention, by utilizing Kafka and multi-threading technology, can process large amounts of data in parallel, significantly improving the speed of data collection and push and reducing data latency. The high availability and fault tolerance of the Kafka cluster improve the reliability of data transmission. By storing data in a data lake, the present invention facilitates data analysis and processing, while also facilitating long-term data preservation and archiving, and enabling centralized data storage and management. In summary, the embodiments of the present invention provide an efficient, reliable, and secure solution for collecting and pushing large model data, providing strong support for big data analysis and artificial intelligence applications, with significant technical advantages and practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of a distributed data transmission method according to an embodiment of the present invention;

[0033] Figure 2 FIG. 4 is a structural block diagram of a distributed data return device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In a first aspect, an embodiment of the present invention provides a distributed data return method, which involves a data collection service, a data push service, a Kafka cluster, a data receiving service, and a data lake.

[0035] See also Figure 1 The method includes the following steps S110 to S150:

[0036] S110, collecting model data generated by the large model through a data collection service, and saving the model data to a database connected to the data collection service; wherein the database is a local database of the distributed cloud center;

[0037] Specifically, after startup, the data collection service connects to the local database of the distributed cloud center, such as a local MySQL database. The data collection service records the time when the large model generates model data, the APIs involved, and the specific data generated by the large model, and saves the time, API, and specific data as model data in the local database.

[0038] In one embodiment, before collecting the model data generated by the large model through the data collection service in S110, the method may further include:

[0039] A data table is created in the database, and the table name of the data table created in the database is the same as the table name of the corresponding data table in the data lake, and the structured table fields of the data table created in the database are the same as the structured table fields of the corresponding data table in the data lake.

[0040] As you can see, before data is collected, a data table identical to the one in the data lake is created in the database. At this point, the tables created in the database and the data lake are empty. After collecting model data, the data collection service saves the collected model data to a table in the database. Then, through subsequent steps, the data in the table is transferred to the same table in the database. This ensures data consistency and integrity and prevents data transfer failures.

[0041] S120: Generate a data push task through the data push service according to pre-configured scheduled task execution information, add the data push task to a thread queue of a thread pool, and add the data table information of the model data in the database to a blocking queue; execute at least one data push task in the thread queue in parallel through at least one thread in the thread pool, with each thread being responsible for retrieving the corresponding data table information from the blocking queue;

[0042] In one embodiment, before generating the data push task by the data push service according to the pre-configured scheduled task execution information in S120, the method may further include:

[0043] 1. Configure the first relevant information corresponding to the data push service; wherein the first relevant information includes information about the data table created in the database, information about the Kafka cluster to be pushed to, information about the published topic, the number of threads in the thread pool, and scheduled task execution information;

[0044] The scheduled task execution information includes, for example, an execution time interval of 5 minutes.

[0045] 2. Start the data push service;

[0046] 3. Call the initialization method to enable the initialization method to create the blocking queue, create a thread pool based on the number of threads in the first relevant information, create the thread queue in the thread pool, and publish the topic information in the first relevant information; wherein, the thread queue is used to store the data push task.

[0047] Specifically, after the data push service is started, it is instantiated into at least one data push unit, which then executes step 3. In step 3, the instantiated data push unit calls an initialization method. This initialization method creates a blocking queue; creates a thread pool based on the number of threads in the first relevant information, places threads with that number in the thread pool, and creates a thread queue in the thread pool, which can be used to place data push tasks; and publishes the topic information in the first relevant information to facilitate the data receiving service to subscribe to the topic.

[0048] After completing the above three steps, the data push service, specifically the data push unit, can generate a data push task every 5 minutes, and then place the data push task in the thread queue, while adding the information of the data table created in the database to the blocking queue. When there is at least one data push task in the thread queue, at least one thread will be started, with one thread responsible for one data push task, and these threads will execute these data push tasks in parallel. The process of each thread executing a data push task includes: retrieving the corresponding data table information in the blocking queue, so as to know which data table needs to be retrieved from the database, facilitating the execution of subsequent S130.

[0049] The thread can retrieve data table information from the blocking queue using the take method. If the blocking queue is empty, the thread will remain blocked in the take method, monitoring the blocking queue until the next scheduled task, at which point the data table information will be added to the blocking queue.

[0050] S130: Establish a connection channel with the database through the data push service, obtain the corresponding data table in the database through the connection channel according to the data table information retrieved by each thread, and close the connection channel;

[0051] Specifically, the data push unit can use the connection pool to obtain a database connection, connect to the database in the distributed cloud center, and then pull the corresponding data table from the database based on the data table information retrieved from the blocking queue by each thread. After the pull is completed, the connection can be closed to release resources.

[0052] S140: Determine incremental data in the data table through the data push service, encapsulate the incremental data to obtain a push message, and send the push message to the Kafka cluster;

[0053] In one embodiment, determining the incremental data in the data table through the data push service in S140 may include: obtaining the timestamp in the last push information through the data push service; based on the timestamp, using the data generated in the data table after the last push operation as the incremental data, and recording the timestamp of this push operation in this push information.

[0054] That is, after each data push, the push information including the timestamp is recorded. Therefore, in S140, the data in the data table can be determined as newly generated data based on the timestamp in the last push information, and the newly generated data is used as incremental data. The timestamp of this push operation is also recorded to facilitate the determination of incremental data during the next push.

[0055] As you can see, after the incremental data is determined, it is packaged into a push message, which is then sent to the Kafka cluster. The Kafka cluster then stores the message in a local log file for consumption by consumers, completing the push process. Message retry and dead letter queue mechanisms can be added to ensure reliable message delivery.

[0056] S150. Pull the push message from the Kafka cluster by subscribing to the data receiving service of the corresponding topic, save the incremental data in the pulled push message into the data lake, and submit its own pull offset to the Kafka cluster.

[0057] In one embodiment, before S150, the method further includes:

[0058] 1. Configure the second relevant information corresponding to the data receiving service; wherein the second relevant information includes information about the data lake, information about the Kafka cluster to be consumed, and information about the subscribed topic;

[0059] 2. Start the data receiving service and subscribe to the subject information in the second relevant information.

[0060] It can be seen that before using the data receiving service, you must first configure the data receiving service, then start the data receiving service, and then subscribe to related topics.

[0061] In one embodiment, the process of saving the incremental data pulled from the push message into the data lake by subscribing to the data receiving service of the corresponding topic in S150 may include: performing structured parsing on the push message by the data receiving service; if the parsing is normal, writing the incremental data obtained by the parsing into the corresponding data table in the data lake in batches through the transaction control mode.

[0062] Furthermore, the method may also include: if all push messages stored in the Kafka cluster are pulled and saved in the data lake through the data receiving service, closing the data receiving service to release the used resources.

[0063] As can be seen, the data receiving service pulls push messages from the Kafka cluster using the pull method. The pull method blocks until a new message is available or the specified timeout is reached. A message confirmation mechanism ensures that messages are processed at least once to prevent data loss. After pulling the push message, the data receiving service performs structured parsing on the push message. If the parsing fails, the message is not processed further. If the parsing is correct, the parsed data is saved to the data lake using batch writes and transaction control. After the data is saved, the data receiving service submits its own pull offset to the Kafka cluster using the commit method, so that consumption can resume from the correct position if the data receiving service is restarted or fails. After the data receiving service consumes all push messages from the subscribed topic, it closes the consumer to free up resources and ensure that the offset is committed.

[0064] The above text involves several professional terms, which are explained as follows:

[0065] (1) Kafka cluster: A distributed stream processing platform and data storage system. It allows applications to exchange messages by decoupling producers and consumers. Producers publish messages to Kafka topics, and consumers read messages from topics. It is used to process large-scale data, achieve decoupling between systems, and provide high availability and fault tolerance. A Kafka cluster consists of one or more brokers. Brokers are server nodes in a Kafka cluster. Each broker is responsible for storing data and servicing client requests. These brokers work together to provide high throughput, scalability, and fault tolerance.

[0066] (2) Kafka Producer: refers to the entity that publishes messages to the Kafka cluster. The producer is responsible for creating messages and sending them to the Kafka cluster.

[0067] (3) Kafka Consumer: refers to the entity that reads messages from the Kafka cluster. Consumers subscribe to one or more topics and process the messages received from these topics.

[0068] (4) Topic: A category or channel used to store messages. Producers publish messages to topics, and consumers read messages from topics.

[0069] (5) Thread pool: Several threads are created in advance in the system and placed in a pool. When a task needs to be executed, the thread pool will take a thread from the pool to execute the task. After the task is completed, the thread is not destroyed but put back into the pool for subsequent tasks. This can significantly improve system performance and resource utilization.

[0070] (6) Bidirectional blocking queue: A thread-safe double-ended queue that supports insertion and deletion operations at both ends of the queue. It is often used in producer-consumer models that require high concurrency. The consumer thread can wait when there are no elements in the queue until the producer thread adds a new element.

[0071] (7) Data Lake: A data storage architecture used to store large amounts of raw data, including structured, semi-structured, and unstructured data. It can store data in its original format without first converting or structuring it. All data can be centrally managed in a central repository, facilitating data analysis and processing.

[0072] In actual scenarios, before executing the method provided by the embodiment of the present invention, it is necessary to deploy a Kafka cluster. The specific process is: first download the latest binary package on each target server, then unzip it to the appropriate directory, modify the server property configuration, refer to the ID of the server node, and specify the listening port of the listener. Finally, start the cluster service on each server. In addition to deploying the Kafka cluster, it is also necessary to open up the network link. Since the data acquisition service, Kafka cluster, and data receiving service of the distributed cloud center are not in the same network area, it is necessary to open up the network connectivity of each node in advance.

[0073] The embodiment of the present invention provides a method for collecting and pushing large model data based on Kafka and multi-threading technology, aiming to realize the transmission of model data from distributed cloud centers to the data lake. The embodiment of the present invention involves data collection services, data push services, Kafka clusters, data receiving services, data lakes, etc. The method includes the data collection service collecting model data and persistently storing it in the local database of the cloud center, and then the data push service starts multi-threading to read the local database. After the data push service finishes reading the data, it acts as a message producer to send the model data to the Kafka cluster through the established network link. The data receiving service acts as a consumer to read the message from the Kafka cluster, and after reading the message, it persists the model data in the data lake. It can be seen that the embodiment of the present invention solves the problems of collecting, transmitting and centrally storing model data generated by distributed cloud centers, so that the generated model data can be regularly transmitted back to the data lake, and finally the data of all distributed cloud centers can be analyzed and operated.

[0074] It can be seen that the embodiment of the present invention can process large amounts of data in parallel by using Kafka and multi-threading technology, significantly improving the speed of data collection and push and reducing data latency. The high availability and fault tolerance of the Kafka cluster are utilized to improve the reliability of data transmission. By storing data in the data lake, the present invention makes data analysis and processing more convenient, while also facilitating the long-term preservation and archiving of data, and realizing centralized storage and management of data. In general, the embodiment of the present invention provides an efficient, reliable, and secure large-model data collection and push solution through a series of technical means, providing strong support for big data analysis and artificial intelligence applications, and has significant technical advantages and practical value.

[0075] In a second aspect, an embodiment of the present invention provides a distributed data backhaul device, see Figure 2 , the apparatus 100 comprises:

[0076] The data acquisition module 110 is used to acquire model data generated by the large model through the data acquisition service and save the model data to a database connected to the data acquisition service; wherein the database is a local database of the distributed cloud center;

[0077] The data push module 120 is configured to generate a data push task according to pre-configured scheduled task execution information through a data push service, add the data push task to a thread queue of a thread pool, and add the data table information of the model data in the database to a blocking queue; execute at least one data push task in the thread queue in parallel through at least one thread in the thread pool, with each thread being responsible for retrieving the corresponding data table information from the blocking queue; establish a connection channel with the database through the data push service, obtain the corresponding data table in the database through the connection channel based on the data table information retrieved by each thread, and close the connection channel; determine the incremental data in the data table through the data push service, encapsulate the incremental data to obtain a push message, and send the push message to the Kafka cluster;

[0078] The data receiving module 130 is used to pull the push message from the Kafka cluster by subscribing to the data receiving service of the corresponding topic, save the incremental data in the pulled push message into the data lake, and submit its own pull offset to the Kafka cluster.

[0079] In one embodiment, before collecting model data generated by the large model through the data collection service, the data collection module is further used to: create a data table in the database, and the table name of the data table created in the database is the same as the table name of the corresponding data table in the data lake, and the structured table fields of the data table created in the database are the same as the structured table fields of the corresponding data table in the data lake.

[0080] In one embodiment, the apparatus further comprises:

[0081] The first configuration module is used to configure the first relevant information corresponding to the data push service before the data push module generates a data push task through the data push service according to the pre-configured scheduled task execution information; wherein, the first relevant information includes information about the data table created in the database, information about the Kafka cluster to be pushed, published topic information, the number of threads in the thread pool and scheduled task execution information; start the data push service; call the initialization method so that the initialization method creates the blocking queue, creates a thread pool according to the number of threads in the first relevant information, creates the thread queue in the thread pool, and publishes the topic information in the first relevant information; wherein, the thread queue is used to store the data push task.

[0082] In one embodiment, the apparatus may further include:

[0083] The second configuration module is used to configure the second related information corresponding to the data receiving service; wherein the second related information includes information about the data lake, information about the Kafka cluster to be consumed, and subscribed topic information; start the data receiving service and subscribe to the topic information in the second related information.

[0084] In one embodiment, the data push module determines the incremental data in the data table through the data push service, including: obtaining the timestamp in the last push information through the data push service; based on the timestamp, using the data generated in the data table after the last push operation as the incremental data, and recording the timestamp of this push operation in this push information.

[0085] In one embodiment, the process of saving the incremental data pulled from the push message into the data lake by subscribing to the data receiving service of the corresponding topic in the data receiving module includes: performing structured parsing on the push message through the data receiving service; if the parsing is normal, writing the incremental data obtained by the parsing into the corresponding data table in the data lake in batches through the transaction control mode.

[0086] In one embodiment, the apparatus may further include:

[0087] The service shutdown module is used to shut down the data receiving service and release the used resources if all push messages stored in the Kafka cluster are pulled and saved in the data lake through the data receiving service.

[0088] It is understandable that the explanation, specific implementation, beneficial effects, examples, etc. of the relevant contents in the device provided by the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0089] In a third aspect, an embodiment of the present invention provides a computer-readable medium having computer instructions stored thereon. When the computer instructions are executed by a processor, the processor executes the method provided in the first aspect.

[0090] Specifically, a system or device equipped with a storage medium can be provided, on which software program codes that implement the functions of any of the above-mentioned embodiments are stored, and a computer (or CPU or MPU) of the system or device can be enabled to read and execute the program codes stored in the storage medium.

[0091] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute part of the present invention.

[0092] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0093] In addition, it should be clear that the functions of any of the above embodiments can be achieved not only by executing the program code read by the computer, but also by enabling the operating system operating on the computer to complete part or all of the actual operations based on the instructions of the program code.

[0094] In addition, it can be understood that the program code read from the storage medium is written into a memory provided in an expansion board inserted into the computer or into a memory provided in an expansion module connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or expansion module is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.

[0095] It is understandable that the explanation, specific implementation methods, beneficial effects, examples, etc. of the relevant contents in the computer-readable medium provided in the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0096] In a fourth aspect, an embodiment of this specification provides a computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method in any one of the embodiments in the specification.

[0097] It is understandable that the explanation, specific implementation, beneficial effects, examples, etc. of the relevant contents in the computing device provided by the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0098] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0099] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention may be implemented using hardware, software, widgets, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0100] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A distributed data backhaul method, characterized in that: include: Collecting model data generated by the large model through a data acquisition service, and saving the model data in a database connected to the data acquisition service; wherein the database is a local database of the distributed cloud center; Generate a data push task through the data push service according to pre-configured scheduled task execution information, add the data push task to the thread queue of the thread pool, and add the data table information of the model data in the database to the blocking queue; execute at least one data push task in the thread queue in parallel through at least one thread in the thread pool, and each thread is responsible for retrieving the corresponding data table information in the blocking queue; Establishing a connection channel with the database through the data push service, obtaining the corresponding data table in the database through the connection channel according to the data table information retrieved by each thread, and closing the connection channel; Determine the incremental data in the data table through the data push service, encapsulate the incremental data to obtain a push message, and send the push message to the Kafka cluster; Pull the push message from the Kafka cluster by subscribing to the data receiving service of the corresponding topic, save the incremental data in the pulled push message into the data lake, and submit its own pull offset to the Kafka cluster.

2. The method according to claim 1, characterized in that Before collecting the model data generated by the large model through the data collection service, the method further includes: A data table is created in the database, and the table name of the data table created in the database is the same as the table name of the corresponding data table in the data lake, and the structured table fields of the data table created in the database are the same as the structured table fields of the corresponding data table in the data lake.

3. The method according to claim 1, characterized in that Before generating the data push task by the data push service according to the pre-configured scheduled task execution information, the method further includes: Configure the first relevant information corresponding to the data push service; wherein the first relevant information includes information about the data table created in the database, information about the Kafka cluster to be pushed to, published topic information, the number of threads in the thread pool, and scheduled task execution information; Starting the data push service; Call the initialization method so that the initialization method creates the blocking queue, creates a thread pool according to the number of threads in the first relevant information, creates the thread queue in the thread pool, and publishes the topic information in the first relevant information; wherein, the thread queue is used to store the data push task.

4. The method according to claim 1, wherein The method further comprises: Configure the second relevant information corresponding to the data receiving service; wherein the second relevant information includes information about the data lake, information about the Kafka cluster to be consumed, and information about the subscribed topic; The data receiving service is started, and the subject information in the second related information is subscribed.

5. The method according to claim 1, wherein The determining the incremental data in the data table by using the data push service includes: The timestamp in the last push information is obtained through the data push service; based on the timestamp, the data generated in the data table after the last push operation is used as the incremental data, and the timestamp of this push operation is recorded in this push information.

6. The method according to claim 1, characterized in that The process of saving the incremental data in the pulled push message into the data lake by subscribing to the data receiving service of the corresponding topic includes: The push message is structurally parsed through the data receiving service; if the parsing is normal, the incremental data obtained by the parsing is written in batches into the corresponding data table in the data lake through the transaction control mode.

7. The method according to claim 1, characterized in that Also includes: If all push messages stored in the Kafka cluster are pulled and saved to the data lake through the data receiving service, the data receiving service is closed to release the used resources.

8. A distributed data return device, characterized in that: include: A data acquisition module, configured to acquire model data generated by a large model through a data acquisition service, and save the model data to a database connected to the data acquisition service; wherein the database is a local database of a distributed cloud center; The data push module is used to generate a data push task according to pre-configured scheduled task execution information through the data push service, add the data push task to the thread queue of the thread pool, and add the data table information of the model data in the database to the blocking queue; execute at least one data push task in the thread queue in parallel through at least one thread in the thread pool, and each thread is responsible for retrieving the corresponding data table information in the blocking queue; establish a connection channel with the database through the data push service, obtain the corresponding data table in the database through the connection channel according to the data table information retrieved by each thread, and close the connection channel; determine the incremental data in the data table through the data push service, encapsulate the incremental data, obtain a push message, and send the push message to the Kafka cluster; The data receiving module is used to pull the push message from the Kafka cluster by subscribing to the data receiving service of the corresponding topic, save the incremental data in the pulled push message to the data lake, and submit its own pull offset to the Kafka cluster.

9. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

10. A computing device, characterized in that The method comprises a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 7 is implemented.

Images