A data collection method and system of a cross-business theme data platform

Abstract

The application discloses a data collection method and system of a cross-service theme data platform, wherein a service end sends a data request to middleware through a request interface, the middleware establishes a data task according to the type of the request interface corresponding to the data request, updates a data task table according to the data task, triggers a data platform to download to-be-collected data based on the data task table, stores the to-be-collected data downloaded to the platform database, returns a download result to the middleware, and updates the data task table according to the download result, and triggers a collector to collect the to-be-collected data from the platform database based on the updated data task table and returns the to-be-collected data to the service end, so that the middleware is used to simplify the data interaction process between the service end and different service theme data platforms, the downloaded data is first stored in the data platform, and then the collector is triggered by the middleware to collect the data to the service end, the database burden is relieved, the data download efficiency is improved, and the data interaction efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of data acquisition technology, and in particular to a data acquisition method and system for a cross-business subject data platform. Background Technology

[0002] In existing technologies, when business personnel need to analyze data across business themes, they need to make corresponding data requests on the business side. The requests will be forwarded to the data platform, which will then retrieve a matching data synchronization table from the data platform database. This synchronization table stores the data snapshot information of the current business user's related businesses. Based on the data synchronization strategy between the current business user's business and related businesses, the data snapshot information is then returned to the current business user.

[0003] In the above method, when users submit requests for the same data range at different times, the data platform will also generate two identical synchronization records. A large amount of work in the process is mechanical and repetitive, resulting in low data access efficiency. In addition, for some data platforms with large market data volumes, the data return efficiency is low. Furthermore, user request data is stored in the data platform database, which increases the database burden and affects query efficiency. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a data acquisition method and system for cross-business theme data platforms, which can improve data interaction efficiency.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A data acquisition method for a cross-business-themed data platform includes the following steps: The business side sends a data request to the middleware through a request interface. The data request includes the data to be collected and the data platform corresponding to the data to be collected. The middleware establishes a data task based on the type of the request interface corresponding to the data request. The data task includes task creation time, task update time, task status, and request parameters, and updates the data task table based on the data task. The middleware triggers the data platform to download the data to be collected based on the data task table; The data platform stores the downloaded data to be collected into the platform database and returns the download result of the data to be collected to the middleware. The middleware updates the task update time and task status of the data task in the data task table according to the download result, and triggers the collector to collect the data to be collected from the platform database and return it to the business terminal based on the updated data task table.

[0006] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows: A data acquisition system for a cross-business theme data platform includes a business terminal, middleware, and a data platform. The business terminal includes a first memory, a first processor, and a first computer program stored in the first memory and executable on the first processor. The middleware includes a second memory, a second processor, and a second computer program stored in the second memory and executable on the second processor. The data platform includes a third memory, a third processor, and a third computer program stored in the third memory and executable on the third processor. When the first processor executes the first computer program, it performs the following steps. A data request is sent to the middleware via a request interface. The data request includes the data to be collected and the data platform corresponding to the data to be collected. When the second processor executes the second computer program, it performs the following steps; A data task is created based on the type of the request interface corresponding to the data request. The data task includes the task creation time, task update time, task status, and request parameters. The data task table is then updated based on the data task. Based on the data task table, the data platform is triggered to download the data to be collected; Update the task update time and task status of the data task in the data task table according to the download results, and trigger the collector to collect the data to be collected from the platform database and return it to the business end based on the updated data task table; When the third processor executes the third computer program, it performs the following steps; The downloaded data to be collected is stored in the platform database, and the download result of the data to be collected is returned to the middleware.

[0007] The beneficial effects of this invention are as follows: The business end sends a data request to the middleware through a request interface. The middleware establishes a data task according to the type of the request interface corresponding to the data request, and updates the data task table according to the data task. Based on the data task table, the middleware triggers the data platform to download the data to be collected. The data platform stores the downloaded data to be collected in the platform database and returns the download result to the middleware. The middleware updates the task update time and task status of the data task in the data task table according to the download result, and triggers the collector to collect the data to be collected from the platform database and return it to the business end based on the updated data task table. In this way, the middleware simplifies the data interaction process between the business end and different business-themed data platforms. Since the market data volume is large, the downloaded data is first stored in the data platform instead of being directly stored in the business end. The middleware then triggers the collector to collect the data to the business end, which alleviates the database burden and improves the data download efficiency, thereby improving the data interaction efficiency. Attached Figure Description

[0008] Figure 1 This is a flowchart illustrating the steps of a data acquisition method for a cross-business-themed data platform according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of a data acquisition system for a cross-business theme data platform according to an embodiment of the present invention; Figure 3 This is a schematic diagram of data acquisition in the data acquisition method of the cross-business theme data platform according to an embodiment of the present invention. Detailed Implementation

[0009] Please refer to Figure 1 A data acquisition method for cross-business subject data platforms, comprising the following steps: The business side sends a data request to the middleware through a request interface. The data request includes the data to be collected and the data platform corresponding to the data to be collected. The middleware establishes a data task based on the type of the request interface corresponding to the data request. The data task includes task creation time, task update time, task status, and request parameters, and updates the data task table based on the data task. The middleware triggers the data platform to download the data to be collected based on the data task table; The data platform stores the downloaded data to be collected into the platform database and returns the download result of the data to be collected to the middleware. The middleware updates the task update time and task status of the data task in the data task table according to the download result, and triggers the collector to collect the data to be collected from the platform database and return it to the business terminal based on the updated data task table.

[0010] As can be seen from the above description, the beneficial effects of the present invention are as follows: the business end sends a data request to the middleware through a request interface. The middleware establishes a data task according to the type of the request interface corresponding to the data request, updates the data task table according to the data task, triggers the data platform to download the data to be collected based on the data task table, stores the downloaded data to be collected in the platform database, and returns the download result to the middleware. The middleware updates the task update time and task status of the data task in the data task table according to the download result, and triggers the collector to collect the data to be collected from the platform database and return it to the business end based on the updated data task table. In this way, the middleware simplifies the data interaction process between the business end and different business-themed data platforms. Since the market data volume is large, the downloaded data is first stored in the data platform instead of being directly stored in the business end. Subsequently, the middleware triggers the collector to collect the data to the business end, which alleviates the database burden and improves the data download efficiency, thereby improving the data interaction efficiency.

[0011] Furthermore, the middleware establishes a data task based on the type of the request interface corresponding to the data request, including: The middleware determines whether the request parameters corresponding to the data request exist in the request parameters of the incomplete data task in the data task table. If they exist, the data request is not processed. If they do not exist, a data task is created according to the type of the request interface corresponding to the data request. The request parameters include start time, end time, site, data type, and report type.

[0012] As described above, if the request parameters for an incomplete data task in the data task table contain the request parameters for a data request, it means that the business user submitted a request for the same data range at different times. The middleware will not process the data request, thereby effectively avoiding duplicate data downloads and affecting data interaction efficiency.

[0013] Furthermore, the middleware triggering the data platform to download the data to be collected based on the data task table includes: The middleware triggers the data platform to download the data to be collected based on the task sorting in the data task table.

[0014] As described above, the middleware triggers the data platform to download the data to be collected based on the task order in the data task table, ensuring the orderly download of data.

[0015] Furthermore, the middleware updates the task update time and task status of the data task in the data task table based on the download result, including: If the download result is successful, the middleware updates the task status of the data task corresponding to the data request in the data task table to "completed" and updates the task update time of the data task. If the download result is a download failure, the middleware updates the task status of the data task corresponding to the data request in the data task table to incomplete, updates the task update time of the data task, and returns to the step of the middleware triggering the data platform to download the data to be collected based on the data task table.

[0016] As described above, the task status and update time of the data tasks in the data task table are updated according to the download results. If a data task fails to process, it waits for the next round of processing to ensure the effectiveness of data interaction.

[0017] Furthermore, the step of triggering the collector to collect the data to be collected from the platform database and return it to the business terminal based on the updated data task table includes: The middleware determines all completed data tasks from the updated data task table and triggers the collector to collect the data to be collected from the platform database at regular intervals according to the task order. The middleware extracts, cleans, and transforms the data to be collected to obtain processed data. The middleware determines incremental data from the processed data based on the data in the business database, and stores the incremental data in the business database of the business.

[0018] As described above, after collecting the data to be collected from the platform database, the middleware extracts, cleans, and transforms the data to obtain processed data. It also identifies incremental data from the processed data and stores the incremental data in the business database, thus avoiding duplicate data collection. Furthermore, through data extraction, cleaning, and transformation, the data that finally reaches the business end not only meets business requirements but also improves data query efficiency, thereby improving data interaction efficiency.

[0019] Please refer to Figure 2Another embodiment of the present invention provides a data acquisition system for a cross-business theme data platform, including a business terminal, middleware, and a data platform. The business terminal includes a first memory, a first processor, and a first computer program stored in the first memory and executable on the first processor. The middleware includes a second memory, a second processor, and a second computer program stored in the second memory and executable on the second processor. The data platform includes a third memory, a third processor, and a third computer program stored in the third memory and executable on the third processor. When the first processor executes the first computer program, it performs the following steps. A data request is sent to the middleware via a request interface. The data request includes the data to be collected and the data platform corresponding to the data to be collected. When the second processor executes the second computer program, it performs the following steps; A data task is created based on the type of the request interface corresponding to the data request. The data task includes the task creation time, task update time, task status, and request parameters. The data task table is then updated based on the data task. Based on the data task table, the data platform is triggered to download the data to be collected; Update the task update time and task status of the data task in the data task table according to the download results, and trigger the collector to collect the data to be collected from the platform database and return it to the business end based on the updated data task table; When the third processor executes the third computer program, it performs the following steps; The downloaded data to be collected is stored in the platform database, and the download result of the data to be collected is returned to the middleware.

[0020] As can be seen from the above description, the beneficial effects of the present invention are as follows: the business end sends a data request to the middleware through a request interface. The middleware establishes a data task according to the type of the request interface corresponding to the data request, updates the data task table according to the data task, triggers the data platform to download the data to be collected based on the data task table, stores the downloaded data to be collected in the platform database, and returns the download result to the middleware. The middleware updates the task update time and task status of the data task in the data task table according to the download result, and triggers the collector to collect the data to be collected from the platform database and return it to the business end based on the updated data task table. In this way, the middleware simplifies the data interaction process between the business end and different business-themed data platforms. Since the market data volume is large, the downloaded data is first stored in the data platform instead of being directly stored in the business end. Subsequently, the middleware triggers the collector to collect the data to the business end, which alleviates the database burden and improves the data download efficiency, thereby improving the data interaction efficiency.

[0021] Furthermore, when the second processor executes the second computer program, it also performs the following steps; Determine whether the request parameters corresponding to the data request exist in the request parameters of the incomplete data task in the data task table. If they exist, the data request is not processed. If they do not exist, a data task is created according to the type of the request interface corresponding to the data request. The request parameters include start time, end time, site, data type, and report type.

[0022] As described above, if the request parameters for an incomplete data task in the data task table contain the request parameters for a data request, it means that the business user submitted a request for the same data range at different times. The middleware will not process the data request, thereby effectively avoiding duplicate data downloads and affecting data interaction efficiency.

[0023] Furthermore, when the second processor executes the second computer program, it also performs the following steps; The data platform is triggered to download the data to be collected based on the task sorting in the data task table.

[0024] As described above, the middleware triggers the data platform to download the data to be collected based on the task order in the data task table, ensuring the orderly download of data.

[0025] Furthermore, when the second processor executes the second computer program, it also performs the following steps; If the download result is successful, then the task status of the data task corresponding to the data request in the data task table is updated to completed, and the task update time of the data task is updated. If the download result is a download failure, then the task status of the data task corresponding to the data request in the data task table is updated to incomplete, the task update time of the data task is updated, and the process returns to the step of triggering the data platform to download the data to be collected based on the data task table.

[0026] As described above, the task status and update time of the data tasks in the data task table are updated according to the download results. If a data task fails to process, it waits for the next round of processing to ensure the effectiveness of data interaction.

[0027] Furthermore, when the second processor executes the second computer program, it also performs the following steps; All completed data tasks are identified from the updated data task table, and the collector is triggered periodically to collect the data to be collected from the platform database according to the task order. The data to be collected is extracted, cleaned, and transformed to obtain processed data; Incremental data is determined from the processed data based on the data in the business database, and the incremental data is stored in the business database of the business.

[0028] As described above, after collecting the data to be collected from the platform database, the middleware extracts, cleans, and transforms the data to obtain processed data. It also identifies incremental data from the processed data and stores the incremental data in the business database, thus avoiding duplicate data collection. Furthermore, through data extraction, cleaning, and transformation, the data that finally reaches the business end not only meets business requirements but also improves data query efficiency, thereby improving data interaction efficiency.

[0029] The data acquisition method and terminal for the cross-business theme data platform described above are applicable to cross-business theme data analysis and integration scenarios. The following detailed embodiments illustrate these methods: Please refer to Figure 1 and Figure 3 Embodiment 1 of the present invention is as follows: A data acquisition method for a cross-business-themed data platform includes the following steps: S1. The business end sends a data request to the middleware through the request interface API. The data request includes the data to be collected and the data platform corresponding to the data to be collected.

[0030] Because there are multiple business-themed data platforms, including sales domain data platforms, advertising domain data platforms, and event domain data platforms, when users request data through the business end, they need to specify the data to be collected and its corresponding data platform so that the middleware can collect data from the corresponding data platform.

[0031] S2. The middleware establishes a data task based on the type of the request interface corresponding to the data request, and updates the data task table according to the data task, such as... Figure 3 As shown, specifically including S21-S22: S21. The middleware determines whether the request parameters corresponding to the data request exist in the request parameters of the unfinished data task in the data task table. If they exist, the data request is not processed. If they do not exist, a data task is created according to the type of the request interface corresponding to the data request.

[0032] The data task includes task creation time, task update time, task status, and request parameters. The request parameters include start_date (start time), end_date (end time), account_station_id (site), record_type (data type), and ad_type (report type), which are determined by the user on the business side. The request interface type includes activity type, product type, or order type, etc. An incomplete data task is considered to exist only if all request parameters corresponding to it are consistent with all request parameters corresponding to the data request. Data for different request interface types is stored in different task tables, and the corresponding task table is updated according to the report type requested by the user. S22: Update the data task table according to the data task.

[0033] S3. The middleware triggers the data platform to download the data to be collected based on the data task table.

[0034] Specifically, the middleware triggers the data platform to download the data to be collected based on the task sorting in the data task table.

[0035] S4. The data platform stores the downloaded data to be collected into the platform database and returns the download result of the data to be collected to the middleware, such as... Figure 3 As shown.

[0036] S5. The middleware updates the task update time and task status of the data task in the data task table according to the download result, and triggers the collector to collect the data to be collected from the platform database and return it to the business terminal based on the updated data task table, such as... Figure 3 As shown, specifically including S51-S55: S51. If the download result is successful, the middleware updates the task status of the data task corresponding to the data request in the data task table to "completed" and updates the task update time of the data task.

[0037] S52. If the download result is a download failure, the middleware updates the task status of the data task corresponding to the data request in the data task table to incomplete, updates the task update time of the data task, and returns to execute S3.

[0038] S53. The middleware determines all completed data tasks from the updated data task table and triggers the collector to collect the data to be collected from the platform database according to the task order at regular intervals.

[0039] The step of periodically triggering the collector to collect the data to be collected from the platform database according to the task order includes: The collector is triggered periodically according to the task order, and collects the data to be collected from the platform database based on the common dimensions of the data platform corresponding to the data to be collected.

[0040] For example, for data to be collected from the advertising domain data platform, data is collected from the advertising domain data platform according to its corresponding general dimension, i.e., date + site; for data to be collected from the sales domain data platform, data is collected from the sales domain data platform according to its corresponding general dimension, i.e., date + data type + site. For data platforms with the same business theme, data is collected in batches at regular intervals using the corresponding general dimensions. The advantage of this approach is that it reduces development costs and ensures data integrity.

[0041] S54. The middleware extracts, cleans, and transforms the data to be collected to obtain processed data.

[0042] S55. The middleware determines incremental data from the processed data based on the data in the business database, and stores the incremental data in the business database of the business.

[0043] In one optional implementation, the method further includes: comparing the data in the business database with the platform database to obtain a comparison result; determining missing data and / or misaligned data based on the comparison result; issuing an alarm for the missing data and / or the misaligned data; or repairing the missing data and / or the misaligned data, thereby realizing the data reconciliation function and ensuring data integrity.

[0044] In one optional implementation, the method further includes: performing layered processing on the incremental data to obtain raw data layer data, processed data layer data, and application data layer data. Specifically, raw data layer data is obtained from the incremental data; the raw data layer data is integrated, filtered, and / or processed to obtain processed data layer data; and the processed data layer data is summarized to obtain application data layer data. This layered processing of the data facilitates the analysis and utilization of data from different business themes by the business side.

[0045] Please refer to Figure 2 Embodiment two of the present invention is as follows: A data acquisition system for a cross-business theme data platform includes a business terminal, middleware, and a data platform. The business terminal includes a first memory, a first processor, and a first computer program stored in the first memory and executable on the first processor. The middleware includes a second memory, a second processor, and a second computer program stored in the second memory and executable on the second processor. The data platform includes a third memory, a third processor, and a third computer program stored in the third memory and executable on the third processor. When the first processor executes the first computer program, it performs the following steps. A data request is sent to the middleware via a request interface. The data request includes the data to be collected and the data platform corresponding to the data to be collected. When the second processor executes the second computer program, it performs the following steps; A data task is created based on the type of the request interface corresponding to the data request. The data task includes the task creation time, task update time, task status, and request parameters. The data task table is then updated based on the data task. Based on the data task table, the data platform is triggered to download the data to be collected; Update the task update time and task status of the data task in the data task table according to the download results, and trigger the collector to collect the data to be collected from the platform database and return it to the business end based on the updated data task table; When the third processor executes the third computer program, it performs the following steps; The downloaded data to be collected is stored in the platform database, and the download result of the data to be collected is returned to the middleware.

[0046] In one optional implementation, the second processor further performs the following steps when executing the second computer program; Determine whether the request parameters corresponding to the data request exist in the request parameters of the incomplete data task in the data task table. If they exist, the data request is not processed. If they do not exist, a data task is created according to the type of the request interface corresponding to the data request. The request parameters include start time, end time, site, data type, and report type.

[0047] In one optional implementation, the second processor further performs the following steps when executing the second computer program; The data platform is triggered to download the data to be collected based on the task sorting in the data task table.

[0048] In one optional implementation, the second processor further performs the following steps when executing the second computer program; If the download result is successful, then the task status of the data task corresponding to the data request in the data task table is updated to completed, and the task update time of the data task is updated. If the download result is a download failure, then the task status of the data task corresponding to the data request in the data task table is updated to incomplete, the task update time of the data task is updated, and the process returns to the step of triggering the data platform to download the data to be collected based on the data task table.

[0049] In one optional implementation, the second processor further performs the following steps when executing the second computer program; All completed data tasks are identified from the updated data task table, and the collector is triggered periodically to collect the data to be collected from the platform database according to the task order. The data to be collected is extracted, cleaned, and transformed to obtain processed data; Incremental data is determined from the processed data based on the data in the business database, and the incremental data is stored in the business database of the business.

[0050] In summary, this invention provides a data acquisition method and system for cross-business-themed data platforms. The business end sends data requests to middleware via a request interface. The middleware creates data tasks based on the type of the request interface corresponding to the data request and updates the data task table accordingly. Based on the data task table, the middleware triggers the data platform to download the data to be collected. The data platform stores the downloaded data in its platform database and returns the download results to the middleware. The middleware updates the task update time and task status of the data tasks in the data task table based on the download results and triggers the collector to collect the data from the platform database and return it to the business end. This simplifies the data interaction process between the business end and different business-themed data platforms using middleware. Due to the large volume of market data, the downloaded data is first stored in the data platform, rather than being directly stored in the middleware. On the business side, the middleware then triggers the collector to collect data, alleviating the database burden and improving data download efficiency, thereby enhancing data interaction efficiency. Furthermore, if the request parameters for an incomplete data task in the data task table contain the same request parameters as a data request, it indicates that users on the business side submitted requests for the same data range at different times. In this case, the middleware does not process the data request, effectively preventing duplicate data downloads and improving data interaction efficiency. Moreover, after collecting the data to be collected from the platform database, the middleware extracts, cleans, and transforms the data to obtain processed data. It also identifies incremental data and stores it in the business-side database, avoiding duplicate data collection. Through data extraction, cleaning, and transformation, the data ultimately reaching the business side not only meets business requirements but also improves data query efficiency.

[0051] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A data acquisition method for a cross-business-themed data platform, characterized in that, Including the following steps: The business side sends a data request to the middleware through a request interface. The data request includes the data to be collected and the data platform corresponding to the data to be collected. The middleware establishes a data task based on the type of the request interface corresponding to the data request. The data task includes task creation time, task update time, task status, and request parameters, and updates the data task table based on the data task. The middleware triggers the data platform to download the data to be collected based on the data task table; The data platform stores the downloaded data to be collected into the platform database and returns the download result of the data to be collected to the middleware. The middleware updates the task update time and task status of the data task in the data task table according to the download result, and triggers the collector to collect the data to be collected from the platform database and return it to the business terminal based on the updated data task table; The middleware establishes a data task based on the type of the request interface corresponding to the data request, including: The middleware determines whether the request parameters corresponding to the data request exist in the request parameters of the incomplete data task in the data task table. If they exist, the data request is not processed. If they do not exist, a data task is created according to the type of the request interface corresponding to the data request. The request parameters include start time, end time, site, data type, and report type; The middleware updates the task update time and task status of the data task in the data task table based on the download result, including: If the download result is successful, the middleware updates the task status of the data task corresponding to the data request in the data task table to "completed" and updates the task update time of the data task. If the download result is a download failure, the middleware updates the task status of the data task corresponding to the data request in the data task table to incomplete, updates the task update time of the data task, and returns to the step of the middleware triggering the data platform to download the data to be collected based on the data task table. The step of triggering the collector to collect the data to be collected from the platform database and return it to the business terminal based on the updated data task table includes: The middleware determines all completed data tasks from the updated data task table and triggers the collector periodically according to the task order to collect the data to be collected from the platform database based on the common dimensions of the data platform corresponding to the data to be collected. The middleware extracts, cleans, and transforms the data to be collected to obtain processed data. The middleware determines incremental data from the processed data based on the data in the business database, and stores the incremental data in the business database of the business.

2. The data acquisition method for a cross-business theme data platform according to claim 1, characterized in that, The middleware triggers the data platform to download the data to be collected based on the data task table, including: The middleware triggers the data platform to download the data to be collected based on the task sorting in the data task table.

3. A data acquisition system for a cross-business theme data platform, comprising a business terminal, middleware, and a data platform, wherein the business terminal includes a first memory, a first processor, and a first computer program stored in the first memory and executable on the first processor; the middleware includes a second memory, a second processor, and a second computer program stored in the second memory and executable on the second processor; and the data platform includes a third memory, a third processor, and a third computer program stored in the third memory and executable on the third processor, characterized in that... When the first processor executes the first computer program, it performs the following steps; A data request is sent to the middleware via a request interface. The data request includes the data to be collected and the data platform corresponding to the data to be collected. When the second processor executes the second computer program, it performs the following steps; A data task is created based on the type of the request interface corresponding to the data request. The data task includes the task creation time, task update time, task status, and request parameters. The data task table is then updated based on the data task. Based on the data task table, the data platform is triggered to download the data to be collected; Update the task update time and task status of the data task in the data task table according to the download results, and trigger the collector to collect the data to be collected from the platform database and return it to the business end based on the updated data task table; When the third processor executes the third computer program, it performs the following steps; The downloaded data to be collected is stored in the platform database, and the download result of the data to be collected is returned to the middleware; When the second processor executes the second computer program, it also performs the following steps; Determine whether the request parameters corresponding to the data request exist in the request parameters of the incomplete data task in the data task table. If they exist, the data request is not processed. If they do not exist, a data task is created according to the type of the request interface corresponding to the data request. The request parameters include start time, end time, site, data type, and report type; When the second processor executes the second computer program, it also performs the following steps; If the download result is successful, then the task status of the data task corresponding to the data request in the data task table is updated to completed, and the task update time of the data task is updated. If the download result is a download failure, then the task status of the data task corresponding to the data request in the data task table is updated to incomplete, the task update time of the data task is updated, and the process of triggering the data platform to download the data to be collected based on the data task table is returned to be executed. The step of triggering the collector to collect the data to be collected from the platform database and return it to the business terminal based on the updated data task table includes: The middleware determines all completed data tasks from the updated data task table and triggers the collector periodically according to the task order to collect the data to be collected from the platform database based on the common dimensions of the data platform corresponding to the data to be collected. The middleware extracts, cleans, and transforms the data to be collected to obtain processed data. The middleware determines incremental data from the processed data based on the data in the business database, and stores the incremental data in the business database of the business.

4. The data acquisition system for a cross-business theme data platform according to claim 3, characterized in that, When the second processor executes the second computer program, it also performs the following steps; The data platform is triggered to download the data to be collected based on the task sorting in the data task table.

Images