Distributed data return method and device, medium and equipment

By using multithreading technology and Kafka cluster in distributed cloud centers, model data is pushed from the local database to the Kafka cluster and saved to the data lake, solving the problem of inefficient data backhaul and centralized storage, and achieving efficient and reliable data Transmission and storage provide strong support for big data analysis.

CN119094605AActive Publication Date: 2024-12-06SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411195781.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-12-06
Estimated Expiration
2044-08-29

AI Technical Summary

Technical Problem

The prior art is difficult to efficiently return and centrally store model data generated by distributed cloud centers, resulting in low data analysis and processing efficiency and insufficient data transmission reliability.

Method used

The model data is saved to the local database through the data acquisition service, and data push and reception are realized using multi-threading technology and Kafka cluster. The data push service encapsulates incremental data into messages and sends it to the Kafka cluster. The data receiving service pulls and receives it from Kafka and Save to a data lake and use Kafka's high availability and fault tolerance to improve the reliability of data transmission.

Benefits of technology

It significantly improves the speed of data collection and push, reduces data latency, realizes efficient, reliable transmission and centralized storage of data, facilitates data analysis and long-term storage, and provides strong support for big data analysis and artificial intelligence applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119094605A_ABST
    Figure CN119094605A_ABST
Patent Text Reader

Abstract

The invention provides a distributed data return method and device, a medium and equipment. The method comprises the steps of collecting model data through a data collection service and storing the model data in a database; generating a data pushing task according to the timed task execution information through the data pushing service, adding the data pushing task into a thread queue, and adding the data table information into a blocking queue; executing at least one data pushing task in the thread queue in parallel through at least one thread in the thread pool; obtaining a corresponding data table in a database according to the data table information extracted from each thread; determining incremental data through the data push service, packaging the incremental data to obtain a push message, and sending the push message to a Kafka cluster; and the push message is pulled from the Kafka cluster through the data receiving service, incremental data is stored in a data lake, and the pull offset is submitted to the Kafka cluster. The embodiment of the invention provides an efficient, reliable and safe model data return scheme.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data storage technology, and in particular to a distributed data return method and device, medium, and equipment. Background Art

[0002] The model data generated by each distributed cloud center needs to be stored centrally to facilitate the use of model data from all distributed cloud centers for related analysis. Therefore, it is necessary to provide a solution for transmitting distributed model data back to meet this demand. Summary of the invention

[0003] In order to solve at least one of the above technical problems, an embodiment of the present invention provides a distributed data backhaul method, device, medium, and equipment.

[0004] According to a first aspect, an embodiment of the present invention provides a distributed data backhaul method comprising:

[0005] Collecting model data generated by the large model through a data collection service, and saving the model data in a database connected to the data collection service; wherein the database is a local database of a distributed cloud center;

[0006] Generate a data push task according to pre-configured scheduled task execution information through the data push service, add the data push task to the thread queue of the thread pool, and add the data table information of the model data in the database to the blocking queue; execute at least one data push task in the thread queue in parallel through at least one thread in the thread pool, and each thread is responsible for taking out the corresponding data table information in the blocking queue;

[0007] Establishing a connection channel with the database through the data push service, obtaining the corresponding data table in the database through the connection channel according to the data table information retrieved by each thread, and closing the connection channel;

[0008] Determine the incremental data in the data table through the data push service, encapsulate the incremental data, obtain a push message, and send the push message to the Kafka cluster;

[0009] The push message is pulled from the Kafka cluster by subscribing to the data receiving service of the corresponding topic, the incremental data in the pulled push message is saved in the data lake, and the pull offset is submitted to the Kafka cluster.

[0010] In one embodiment, before collecting the model data generated by the large model through the data collection service, the method further includes:

[0011] A data table is created in the database, and the table name of the data table created in the database is the same as the table name of the corresponding data table in the data lake, and the structured table fields of the data table created in the database are the same as the structured table fields of the corresponding data table in the data lake.

[0012] In one embodiment, before generating the data push task by the data push service according to the pre-configured scheduled task execution information, the method further includes:

[0013] Configure the first relevant information corresponding to the data push service; wherein the first relevant information includes information about the data table created in the database, information about the Kafka cluster to be pushed, published topic information, the number of threads in the thread pool, and scheduled task execution information;

[0014] Initiate the data push service;

[0015] Call the initialization method so that the initialization method creates the blocking queue, creates a thread pool according to the number of threads in the first relevant information, creates the thread queue in the thread pool, and publishes the subject information in the first relevant information; wherein the thread queue is used to store the data push task.

[0016] In one embodiment, the method further comprises:

[0017] Configure the second related information corresponding to the data receiving service; wherein the second related information includes information about the data lake, information about the Kafka cluster to be consumed, and subscribed topic information;

[0018] The data receiving service is started, and the subject information in the second related information is subscribed.

[0019] In one embodiment, determining the incremental data in the data table through the data push service includes:

[0020] The timestamp in the last push information is obtained through the data push service; according to the timestamp, the data generated in the data table after the last push operation is used as the incremental data, and the timestamp of this push operation is recorded in this push information.

[0021] In one embodiment, the process of saving the incremental data in the pushed message pulled to the data lake by subscribing to the data receiving service of the corresponding topic includes:

[0022] The push message is structurally parsed through the data receiving service; if the parsing is normal, the incremental data obtained by the parsing is batch-written into the corresponding data table in the data lake through the transaction control mode.

[0023] In one embodiment, the method further comprises:

[0024] If all push messages stored in the Kafka cluster are pulled and saved in the data lake through the data receiving service, the data receiving service is closed to release the used resources.

[0025] According to the second aspect, the distributed data backhaul device provided by the embodiment of the present invention includes:

[0026] A data acquisition module, used to acquire model data generated by a large model through a data acquisition service, and save the model data into a database connected to the data acquisition service; wherein the database is a local database of a distributed cloud center;

[0027] A data push module is used to generate a data push task according to pre-configured scheduled task execution information through a data push service, add the data push task to the thread queue of a thread pool, and add the data table information of the model data in the database to a blocking queue; execute at least one data push task in the thread queue in parallel through at least one thread in the thread pool, and each thread is responsible for taking out the corresponding data table information in the blocking queue; establish a connection channel with the database through the data push service, obtain the corresponding data table in the database through the connection channel according to the data table information taken out by each thread, and close the connection channel; determine the incremental data in the data table through the data push service, encapsulate the incremental data, obtain a push message, and send the push message to the Kafka cluster;

[0028] The data receiving module is used to pull the push message from the Kafka cluster by subscribing to the data receiving service of the corresponding topic, save the incremental data in the pulled push message into the data lake, and submit its own pull offset to the Kafka cluster.

[0029] According to a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, enables the computer to execute the method provided in the first aspect.

[0030] According to a fourth aspect, an embodiment of the present invention provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method provided in the first aspect is implemented.

[0031] The distributed data transmission method, device, medium, and equipment provided by the embodiment of the present invention, the data collection service collects the model data and stores it persistently in the local database of the cloud center, and then the data push service starts multi-threading to read the local database. After the data push service reads the data, it sends the model data to the Kafka cluster through the opened network link as a message producer, and the data receiving service reads the message from the Kafka cluster as a consumer. After reading the message, the model data is persisted in the data lake. It can be seen that the embodiment of the present invention can process a large amount of data in parallel by using Kafka and multi-threading technology, significantly improving the speed of data collection and push, and reducing data delay. The high availability and fault tolerance of the Kafka cluster are used to improve the reliability of data transmission. By storing data in the data lake, the present invention makes data analysis and processing more convenient, and also facilitates the long-term preservation and archiving of data, and realizes the centralized storage and management of data. In general, the embodiment of the present invention provides an efficient, reliable, and secure large model data collection and push solution, which provides strong support for big data analysis and artificial intelligence applications, and has significant technical advantages and practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of a distributed data transmission method according to an embodiment of the present invention;

[0033] Figure 2 FIG. 4 is a structural block diagram of a distributed data return device in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In a first aspect, an embodiment of the present invention provides a distributed data return method, which involves a data collection service, a data push service, a Kafka cluster, a data receiving service, and a data lake.

[0035] See also Figure 1 The method comprises the following steps S110 to S150:

[0036] S110, collecting model data generated by the large model through a data collection service, and saving the model data in a database connected to the data collection service; wherein the database is a local database of a distributed cloud center;

[0037] Specifically, after startup, the data collection service will connect to the local database of the distributed cloud center, such as the local MySQL database. The data collection service will record the time when the large model generates model data, the API involved, and the specific data generated by the large model, and save the time, API, and specific data as model data in the local database.

[0038] In one embodiment, before collecting the model data generated by the large model through the data collection service in S110, the method may further include:

[0039] A data table is created in the database, and the table name of the data table created in the database is the same as the table name of the corresponding data table in the data lake, and the structured table fields of the data table created in the database are the same as the structured table fields of the corresponding data table in the data lake.

[0040] It can be seen that before collecting data, a data table identical to that in the data lake is created on the database. At this time, the data tables created in the database and the data lake are empty tables. After collecting model data, the data collection service saves the collected model data to the data table in the database, and then transfers the data in the data table to the same data table in the database through subsequent steps. This is to ensure the consistency and integrity of the data and avoid failure of data transmission.

[0041] S120, generating a data push task according to pre-configured scheduled task execution information through the data push service, adding the data push task to the thread queue of the thread pool, and adding the data table information of the model data in the database to the blocking queue; executing at least one data push task in the thread queue in parallel through at least one thread in the thread pool, and each thread is responsible for taking out the corresponding data table information in the blocking queue;

[0042] In one embodiment, before generating the data push task by the data push service according to the pre-configured scheduled task execution information in S120, the method may further include:

[0043] 1. Configure the first relevant information corresponding to the data push service; wherein the first relevant information includes information about the data table created in the database, information about the Kafka cluster to be pushed, published topic information, the number of threads in the thread pool, and scheduled task execution information;

[0044] The scheduled task execution information includes, for example, an execution time interval of 5 minutes.

[0045] 2. Start the data push service;

[0046] 3. Call the initialization method to enable the initialization method to create the blocking queue, create a thread pool according to the number of threads in the first relevant information, create the thread queue in the thread pool, and publish the subject information in the first relevant information; wherein, the thread queue is used to store the data push task.

[0047] Specifically, after the data push service is started, it will be instantiated into at least one data push unit, and the data push unit will execute the above step 3. In step 3, the instantiated data push unit calls the initialization method. The initialization method will create a blocking queue; create a thread pool according to the number of threads in the first relevant information, place threads with the number of threads in the thread pool, and create the thread queue in the thread pool, which can be used to place data push tasks; and publish the topic information in the first relevant information so that the data receiving service can subscribe to the topic.

[0048] After executing the above three steps, the data push service is specifically that the data push unit can generate a data push task every 5 minutes, and then place the data push task in the thread queue, and add the information of the data table created in the database to the blocking queue. When there is at least one data push task in the thread queue, at least one thread will be started, one thread is responsible for one data push task, and these threads execute these data push tasks in parallel. The process of each thread executing a data push task includes: taking out the corresponding data table information in the blocking queue, so as to know which data table needs to obtain data from the database, so as to facilitate the subsequent S130.

[0049] The thread can use the take method to retrieve the data table information from the blocking queue. If the blocking queue is empty, the thread will be blocked in the take method and keep monitoring the blocking queue. After the next scheduled task, the data table information will be added to the blocking queue.

[0050] S130, establishing a connection channel with the database through the data push service, obtaining the corresponding data table in the database through the connection channel according to the data table information retrieved by each thread, and closing the connection channel;

[0051] Specifically, the data push unit can use the connection pool to obtain a database connection, connect to the database of the distributed cloud center, and then pull the corresponding data table from the database according to the data table information taken out from the blocking queue by each thread. After the pull is completed, the connection can be closed to release resources.

[0052] S140, determining incremental data in the data table through the data push service, encapsulating the incremental data to obtain a push message, and sending the push message to the Kafka cluster;

[0053] In one embodiment, determining the incremental data in the data table through the data push service in S140 may include: obtaining the timestamp in the last push information through the data push service; based on the timestamp, using the data in the data table generated after the last push operation as the incremental data, and recording the timestamp of this push operation in this push information.

[0054] That is, after each data push, the push information including the timestamp will be recorded. Therefore, in S140, which data in the data table is newly generated data can be determined according to the timestamp in the last push information, and the newly generated data is used as incremental data. At the same time, the timestamp of this push operation is also recorded, so as to facilitate the determination of incremental data in the next push.

[0055] It can be seen that after the incremental data is determined, the incremental data is encapsulated to obtain a push message, which is sent to the Kafka cluster. The Kafka cluster stores it in the local log file for consumers to consume, thus completing the push work. Among them, message retry and dead letter queue mechanisms can be added to ensure reliable message delivery.

[0056] S150. Pull the push message from the Kafka cluster by subscribing to the data receiving service of the corresponding topic, save the incremental data in the pulled push message into the data lake, and submit its own pull offset to the Kafka cluster.

[0057] In one embodiment, before S150, the method further includes:

[0058] 1. Configure the second related information corresponding to the data receiving service; wherein the second related information includes information about the data lake, information about the Kafka cluster to be consumed, and subscribed topic information;

[0059] 2. Start the data receiving service and subscribe to the subject information in the second relevant information.

[0060] It can be seen that before using the data receiving service, first perform relevant configuration of the data receiving service, then start the data receiving service, and then subscribe to relevant topics.

[0061] In one embodiment, the process of saving the incremental data pulled from the push message into the data lake by subscribing to the data receiving service of the corresponding topic in S150 may include: performing structured parsing on the push message by the data receiving service; if the parsing is normal, writing the incremental data obtained by the parsing into the corresponding data table in the data lake in batches through the transaction control mode.

[0062] Furthermore, the method may also include: if all push messages stored in the Kafka cluster are pulled and saved in the data lake through the data receiving service, then the data receiving service is closed to release the used resources.

[0063] As can be seen, the data receiving service pulls push messages from the Kafka cluster through the pull method. The pull method blocks until new messages are available or the specified timeout is reached. And through the message confirmation mechanism, it ensures that the message is processed at least once to prevent data loss. After pulling the push message, the data receiving service will perform structured parsing on the push message. If the parsing is abnormal, the message will not be processed later. If there is no problem with the parsing, the parsed data will be saved to the data lake using batch writing and transaction control. After the data is saved, the data receiving service will submit its own pull offset to the Kafka cluster through the submit method so that it can continue to consume from the correct position when the data receiving service is restarted or fails. After the data receiving service consumes all the push messages of the subscribed topic, it closes the consumer to release resources and ensure that the offset is committed.

[0064] There are many professional terms involved in the above text, which are explained as follows:

[0065] (1) Kafka cluster: a distributed stream processing platform and data storage system. It allows applications to exchange messages through decoupling between producers and consumers. Producers publish messages to Kafka topics, and consumers read messages from topics. It is used to process large-scale data, achieve decoupling between systems, and provide high availability and fault tolerance. A Kafka cluster consists of one or more Brokers, which are server nodes in a Kafka cluster. Each Broker is responsible for storing data and serving client requests. These Brokers work together to provide high throughput, scalability, and fault tolerance.

[0066] (2) Kafka producer: refers to the entity that publishes messages to the Kafka cluster. The producer is responsible for creating messages and sending them to the Kafka cluster.

[0067] (3) Kafka consumer: refers to the subject that reads messages from the Kafka cluster. Consumers subscribe to one or more topics and process the messages received from these topics.

[0068] (4) Topic: A category or channel used to store messages. Producers publish messages to topics, and consumers read messages from topics.

[0069] (5) Thread pool: Create several threads in the system in advance and put them into a pool. When a task needs to be executed, the thread pool will take a thread from the pool to execute the task. After the execution is completed, the thread will not be destroyed, but put into the pool again for subsequent tasks. This can significantly improve the system performance and resource utilization.

[0070] (6) Bidirectional blocking queue: A thread-safe double-ended queue that supports insertion and deletion operations at both ends of the queue. It is usually used in the producer-consumer model that requires high concurrency processing. The consumer thread can wait when there are no elements in the queue until the producer thread adds new elements.

[0071] (7) Data Lake: A data storage architecture used to store large amounts of raw data, including structured, semi-structured, and unstructured data; capable of storing data in its original format without first converting or structuring it. All data can be centrally managed in a central repository to facilitate data analysis and processing.

[0072] In actual scenarios, before executing the method provided by the embodiment of the present invention, it is necessary to deploy a Kafka cluster. The specific process is: first download the latest binary package on each target server, then unzip it to the appropriate directory, modify the server property configuration, refer to the ID of the server node, and specify the listening port of the listener. Finally, start the cluster service on each server. In addition to deploying the Kafka cluster, it is also necessary to open up the network link. Since the data acquisition service, Kafka cluster, and data receiving service of the distributed cloud center are not in the same network area, it is necessary to open them up in advance to ensure the network connectivity of each node.

[0073] The embodiment of the present invention provides a method for collecting and pushing large model data based on Kafka and multi-threading technology, aiming to realize the transmission of model data of distributed cloud centers back to the data lake. The embodiment of the present invention relates to data collection services, data push services, Kafka clusters, data receiving services, data lakes, etc. The method includes the data collection service collecting model data and storing it persistently in the local database of the cloud center, and then the data push service starts multi-threading to read the local database. After the data push service reads the data, it sends the model data to the Kafka cluster as a message producer through the opened network link. The data receiving service reads the message from the Kafka cluster as a consumer, and after reading the message, the model data is persisted in the data lake. It can be seen that the embodiment of the present invention solves the problems of collection, transmission and centralized storage of model data generated by distributed cloud centers, so that the generated model data can be regularly transmitted back to the data lake, and finally the data of all distributed cloud centers are analyzed and operated.

[0074] It can be seen that the embodiment of the present invention can process a large amount of data in parallel by using Kafka and multi-threading technology, which significantly improves the speed of data collection and push and reduces data latency. The high availability and fault tolerance of the Kafka cluster are utilized to improve the reliability of data transmission. By storing data in a data lake, the present invention makes data analysis and processing more convenient, while also facilitating the long-term preservation and archiving of data, and realizing centralized storage and management of data. In general, the embodiment of the present invention provides an efficient, reliable, and secure large-model data collection and push solution through a series of technical means, which provides strong support for big data analysis and artificial intelligence applications, and has significant technical advantages and practical value.

[0075] In a second aspect, an embodiment of the present invention provides a distributed data return device, see Figure 2 , the device 100 comprises:

[0076] The data acquisition module 110 is used to acquire the model data generated by the large model through the data acquisition service, and save the model data to a database connected to the data acquisition service; wherein the database is a local database of the distributed cloud center;

[0077] The data push module 120 is used to generate a data push task according to pre-configured scheduled task execution information through a data push service, add the data push task to a thread queue of a thread pool, and add the data table information of the model data in the database to a blocking queue; execute at least one data push task in the thread queue in parallel through at least one thread in the thread pool, and each thread is responsible for taking out the corresponding data table information in the blocking queue; establish a connection channel with the database through the data push service, obtain the corresponding data table in the database through the connection channel according to the data table information taken out by each thread, and close the connection channel; determine the incremental data in the data table through the data push service, encapsulate the incremental data, obtain a push message, and send the push message to the Kafka cluster;

[0078] The data receiving module 130 is used to pull the push message from the Kafka cluster by subscribing to the data receiving service of the corresponding topic, save the incremental data in the pulled push message into the data lake, and submit its own pull offset to the Kafka cluster.

[0079] In one embodiment, before the data acquisition module acquires the model data generated by the large model through the data acquisition service, it is also used to: create a data table in the database, and the table name of the data table created in the database is the same as the table name of the corresponding data table in the data lake, and the structured table fields of the data table created in the database are the same as the structured table fields of the corresponding data table in the data lake.

[0080] In one embodiment, the apparatus further comprises:

[0081] The first configuration module is used to configure the first relevant information corresponding to the data push service before the data push module generates a data push task through the data push service according to the pre-configured scheduled task execution information; wherein the first relevant information includes the information of the data table created in the database, the information of the Kafka cluster to be pushed, the published topic information, the number of threads in the thread pool and the scheduled task execution information; start the data push service; call the initialization method so that the initialization method creates the blocking queue, creates a thread pool according to the number of threads in the first relevant information, creates the thread queue in the thread pool, and publishes the topic information in the first relevant information; wherein the thread queue is used to store the data push task.

[0082] In one embodiment, the device may further include:

[0083] The second configuration module is used to configure the second related information corresponding to the data receiving service; wherein the second related information includes information about the data lake, information about the Kafka cluster to be consumed, and subscribed topic information; start the data receiving service and subscribe to the topic information in the second related information.

[0084] In one embodiment, the data push module determines the incremental data in the data table through the data push service, including: obtaining the timestamp in the last push information through the data push service; based on the timestamp, using the data generated in the data table after the last push operation as the incremental data, and recording the timestamp of this push operation in this push information.

[0085] In one embodiment, the process of saving the incremental data pulled from the push message into the data lake by subscribing to the data receiving service of the corresponding topic in the data receiving module includes: performing structured parsing on the push message through the data receiving service; if the parsing is normal, writing the incremental data obtained by the parsing into the corresponding data table in the data lake in batches through the transaction control mode.

[0086] In one embodiment, the device may further include:

[0087] The service shutdown module is used to shut down the data receiving service and release the used resources if all the push messages stored in the Kafka cluster are pulled and saved in the data lake through the data receiving service.

[0088] It is understandable that the explanation, specific implementation, beneficial effects, examples, etc. of the relevant contents in the device provided in the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0089] In a third aspect, an embodiment of the present invention provides a computer-readable medium, on which computer instructions are stored. When the computer instructions are executed by a processor, the processor executes the method provided in the first aspect.

[0090] Specifically, a system or device equipped with a storage medium can be provided, on which software program code that implements the functions of any of the above-mentioned embodiments is stored, and a computer (or CPU or MPU) of the system or device can be enabled to read and execute the program code stored in the storage medium.

[0091] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0092] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0093] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0094] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion module connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or expansion module is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0095] It is understandable that the explanation, specific implementation methods, beneficial effects, examples, etc. of the relevant contents in the computer-readable medium provided in the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0096] In a fourth aspect, an embodiment of the present specification provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method in any embodiment of the specification.

[0097] It is understandable that the explanation, specific implementation, beneficial effects, examples, etc. of the relevant contents in the computing device provided in the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0098] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0099] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by hardware, software, widgets, or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0100] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A distributed data return method, It is characterized in that include: Collecting model data generated by the large model through a data collection service, and saving the model data in a database connected to the data collection service; wherein the database is a local database of a distributed cloud center; Generate a data push task according to pre-configured scheduled task execution information through the data push service, add the data push task to the thread queue of the thread pool, and add the data table information of the model data in the database to the blocking queue; execute at least one data push task in the thread queue in parallel through at least one thread in the thread pool, and each thread is responsible for taking out the corresponding data table information in the blocking queue; Establishing a connection channel with the database through the data push service, obtaining the corresponding data table in the database through the connection channel according to the data table information retrieved by each thread, and closing the connection channel; Determine the incremental data in the data table through the data push service, encapsulate the incremental data, obtain a push message, and send the push message to the Kafka cluster; The push message is pulled from the Kafka cluster by subscribing to the data receiving service of the corresponding topic, the incremental data in the pulled push message is saved in the data lake, and the pull offset is submitted to the Kafka cluster.

2. The method according to claim 1, It is characterized in that Before collecting the model data generated by the large model through the data collection service, the method further includes: A data table is created in the database, and the table name of the data table created in the database is the same as the table name of the corresponding data table in the data lake, and the structured table fields of the data table created in the database are the same as the structured table fields of the corresponding data table in the data lake.

3. The method according to claim 1, It is characterized in that Before the data push task is generated by the data push service according to the pre-configured scheduled task execution information, the method further includes: Configure the first relevant information corresponding to the data push service; wherein the first relevant information includes information about the data table created in the database, information about the Kafka cluster to be pushed, published topic information, the number of threads in the thread pool, and scheduled task execution information; Initiate the data push service; Call the initialization method so that the initialization method creates the blocking queue, creates a thread pool according to the number of threads in the first relevant information, creates the thread queue in the thread pool, and publishes the subject information in the first relevant information; wherein the thread queue is used to store the data push task.

4. The method according to claim 1, It is characterized in that The method further comprises: Configure the second related information corresponding to the data receiving service; wherein the second related information includes information about the data lake, information about the Kafka cluster to be consumed, and subscribed topic information; The data receiving service is started, and the subject information in the second related information is subscribed.

5. The method according to claim 1, It is characterized in that The determining the incremental data in the data table through the data push service includes: The timestamp in the last push information is obtained through the data push service; according to the timestamp, the data generated in the data table after the last push operation is used as the incremental data, and the timestamp of this push operation is recorded in this push information.

6. The method according to claim 1, It is characterized in that The process of saving the incremental data in the pushed message pulled into the data lake by subscribing to the data receiving service of the corresponding topic includes: The push message is structurally parsed through the data receiving service; if the parsing is normal, the incremental data obtained by the parsing is batch-written into the corresponding data table in the data lake through the transaction control mode.

7. The method according to claim 1, It is characterized in that Also includes: If all push messages stored in the Kafka cluster are pulled and saved in the data lake through the data receiving service, the data receiving service is closed to release the used resources.

8. A distributed data return device, It is characterized in that include: A data acquisition module, used to acquire model data generated by a large model through a data acquisition service, and save the model data into a database connected to the data acquisition service; wherein the database is a local database of a distributed cloud center; A data push module is used to generate a data push task according to pre-configured scheduled task execution information through a data push service, add the data push task to the thread queue of a thread pool, and add the data table information of the model data in the database to a blocking queue; execute at least one data push task in the thread queue in parallel through at least one thread in the thread pool, and each thread is responsible for taking out the corresponding data table information in the blocking queue; establish a connection channel with the database through the data push service, obtain the corresponding data table in the database through the connection channel according to the data table information taken out by each thread, and close the connection channel; determine the incremental data in the data table through the data push service, encapsulate the incremental data, obtain a push message, and send the push message to the Kafka cluster; The data receiving module is used to pull the push message from the Kafka cluster by subscribing to the data receiving service of the corresponding topic, save the incremental data in the pulled push message into the data lake, and submit its own pull offset to the Kafka cluster.

9. A computer-readable storage medium, It is characterized in that A computer program is stored thereon, and when the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

10. A computing device, It is characterized in that The method comprises a memory and a processor, wherein the memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Batch data ingestion in database systems

    CN111480154A

  • Data return method and device, equipment and medium

    CN118014732A

  • Database system with database engine and separate distributed storage service

    US20140279929A1

  • Distributed database cluster system, data synchronization method and storage medium

    WO2019154394A1

  • Method, system and device for active pushing of distributed system, and medium

    WO2021238287A1