Data acquisition method and device, electronic equipment and storage medium

Abstract

This application relates to the field of big data technology and discloses a data acquisition method, apparatus, electronic device, and storage medium. The method includes: if the structure of the tenant database to be acquired in the business system changes, acquiring the change information of the tenant database, the change information indicating the source data table to be acquired after the change and the first field in the source data table to be acquired; generating a destination table corresponding to the source data table based on the first field in the source data table and the field type of the first field in the source data table, wherein the field type of the first field in the source data table is the same as the field type of the first field in the destination table; acquiring task allocation information; controlling multiple nodes to execute their corresponding acquisition tasks in parallel to acquire data from multiple source data tables in parallel, and writing the acquired data into the destination table corresponding to the source data table. This application can improve data acquisition efficiency.

Description

Technical Field

[0001] This application relates to the field of big data technology, and more specifically, to a data acquisition method, apparatus, electronic device, and storage medium. Background Technology

[0002] Big data refers to data sets that cannot be captured, managed, and processed within a certain timeframe using conventional software tools. It represents massive, rapidly growing, and diverse information assets that require new processing models to achieve stronger decision-making, insightful discovery, and process optimization capabilities. With the advent of the cloud era, big data has attracted increasing attention. Big data requires specialized technologies to effectively process massive amounts of data.

[0003] The first step in big data analytics is to collect data from business systems, followed by big data analysis. However, the structure of the database within a business system may change. To ensure the accuracy of the data used for analysis, after data collection, it's necessary to check its accuracy against these changes in the database structure. This can lead to significant time commitment for data verification, especially with large datasets, resulting in low overall data collection efficiency. Summary of the Invention

[0004] In view of the above problems, this application proposes a data acquisition method, apparatus, electronic device and storage medium to solve the problem of low data acquisition efficiency in related technologies.

[0005] According to one aspect of the embodiments of this application, a data acquisition method is provided, applied to a control node in a data analysis system, wherein the data analysis system acquires data from a tenant database of a tenant in a business system for analysis; the method includes: if the structure of the tenant database to be acquired in the business system changes, acquiring change information of the tenant database, the change information indicating the source data table to be acquired after the change and a first field in the source data table to be acquired; generating a destination table corresponding to the source data table based on the first field in the source data table and the field type of the first field in the source data table, wherein the field type of the first field in the source data table is the same as the field type of the first field in the destination table; acquiring task allocation information, the task allocation information indicating multiple nodes allocated in the data analysis system for data acquisition and the acquisition task of each node, the acquisition task indicating the source data table to be acquired by each node and the first field in the corresponding source data table to be acquired; controlling the multiple nodes to execute the corresponding acquisition tasks in parallel to acquire data from multiple source data tables in parallel, and writing the acquired data into the destination table corresponding to the source data table.

[0006] According to one aspect of the embodiments of this application, a data acquisition device is provided, applied to a control node in a data analysis system, wherein the data analysis system acquires data from a tenant database of a tenant in a business system for analysis; the device includes: a change information acquisition module, configured to acquire change information of the tenant database if the structure of the tenant database to be acquired in the business system changes, the change information indicating the source data table to be acquired after the change and a first field in the source data table to be acquired; and a destination table generation module, configured to generate a destination table based on the first field in the source data table and the field type of the first field in the source data table. The source data table corresponds to the destination table, where the field type of the first field in the source data table is the same as the field type of the first field in the destination table; a task allocation information acquisition module is used to acquire task allocation information, which indicates multiple nodes allocated in the data analysis system for data collection and the collection tasks of each of the multiple nodes, wherein the collection tasks indicate the source data table to which each node is to collect data; a parallel acquisition module is used to control the multiple nodes to execute their corresponding acquisition tasks in parallel, so as to collect data from multiple source data tables in parallel and write the collected data into the destination table corresponding to the source data table.

[0007] According to one aspect of the embodiments of this application, an electronic device is provided, including: a processor; a memory, the memory storing computer-readable instructions, which, when executed by the processor, implement the data acquisition method as described above.

[0008] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, on which computer-readable instructions are stored, which, when executed by a processor, implement the data acquisition method as described above.

[0009] According to one aspect of the embodiments of this application, a computer program product is provided, including computer instructions that, when executed by a processor, implement the data acquisition method described above.

[0010] In this application, when the structure of the tenant database in the business system changes, the change information of the tenant database indicates the source data table that needs to be collected and the first field in the source data table that needs to be collected. A destination table corresponding to the source data table is generated. The field type of the first field in the source data table is the same as the field type of the first field in the destination table. In this way, during the data collection process, the data collected from the newly added source data table can be saved to the destination table corresponding to the source data table, ensuring the accuracy of the data in the destination table.

[0011] Since the target table, corresponding to the structure of the source data table, is generated based on the tenant database change information before data collection, there is no need for secondary processing of the data in the target table after data collection and before analysis to adapt to the tenant database structure changes. Compared to related technologies that perform secondary analysis and processing on the collected data based on changes in the tenant database structure after data collection, this solution improves data collection efficiency, especially when the data collection volume is large. Therefore, adopting this solution can effectively ensure the accuracy and efficiency of data collection even when the tenant database structure changes. Moreover, in this application, assigning corresponding collection tasks to multiple nodes and executing these tasks in parallel on multiple nodes allows for parallel data collection from multiple source data tables, further improving data collection efficiency. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0013] Figure 1 This is a schematic diagram illustrating an application scenario of the present solution according to an embodiment of this application.

[0014] Figure 2 This is a flowchart illustrating a data acquisition method according to an embodiment of this application.

[0015] Figure 3 This is a schematic diagram of a configuration interface shown according to an embodiment of this application.

[0016] Figure 4 This is illustrated according to an embodiment of the present application. Figure 2 The flowchart of step 220 in the corresponding embodiment.

[0017] Figure 5 This is a schematic diagram illustrating data collection from a tenant database for a single tenant, according to an embodiment of this application.

[0018] Figure 6 This is a schematic diagram illustrating data collection from a tenant database for multiple tenants, according to an embodiment of this application.

[0019] Figure 7 This is illustrated according to an embodiment of the present application. Figure 2 The flowchart of step 230 in the corresponding embodiment.

[0020] Figure 8 This is a schematic diagram illustrating the adjustment of data acquisition tasks based on a scaling strategy, according to an embodiment of this application.

[0021] Figure 9 This is a block diagram of a data analysis system according to an embodiment of this application.

[0022] Figure 10 This is a timing diagram illustrating data acquisition in a DDL change scenario according to an embodiment of this application.

[0023] Figure 11 This is a timing diagram illustrating data collection in a tenant change scenario according to an embodiment of this application.

[0024] Figure 12 This is a block diagram of a data acquisition device according to an embodiment of this application.

[0025] Figure 13 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0026] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0027] It should be noted that "multiple" in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0028] Before providing a detailed description of this application, the terms used in this application are explained as follows:

[0029] Structured Query Language (SQL) is a database query and programming language used to access, query, update, and manage relational database systems. SQL statements are divided into two categories: data definition language and data manipulation language. Data manipulation language is used to implement basic operations on the database.

[0030] Data Definition Language (DDL): Used to describe databases, define database structure, create database objects, and create sub-objects of tables. It is a language used to describe real-world entities to be stored in the database. Data Definition Language is mainly generated through four keywords: create, alter, drop, and truncate.

[0031] Load balancing distributes network requests or other forms of load evenly across different machines. This prevents some machines in a cluster from being overloaded while others are relatively idle. Through load balancing, each machine receives a load appropriate to its processing capacity. While offloading high-load machines, it also avoids resource waste.

[0032] SaaS (Software-as-a-Service) is an application model that provides software services over the internet. All the initial implementation and subsequent maintenance of the network infrastructure, software, and hardware operating platforms are handled by the SaaS platform. Users simply rent the software service and can use it by hosting, deploying, and accessing it via the internet.

[0033] Multi-tenant architecture refers to a software architecture where a single software instance can provide services to multiple different user groups. Software as a Service (SaaS) is a type of multi-tenant architecture. In contrast, single-tenant architecture refers to a software instance or computer system used by only one end-user or user group.

[0034] Figure 1 This is a schematic diagram illustrating an application scenario of the present solution according to an embodiment of this application, such as... Figure 1 As shown, this application scenario includes a business system 110 and a data analysis system 120. The business system 110 can provide business services 111 to tenants, such as facial recognition, speech recognition, and text classification services. The business system 110 also provides a tenant database for each tenant. Figure 1 The example illustrates a tenant database f1 and a tenant database f2. The data tables in this tenant database are used to store data generated during the process of a tenant's device calling business service 111. In some embodiments, the business service provided by the business system can be a SaaS service. The business system 110 can adopt a cloud computing model to provide business services to multiple tenants. Correspondingly, the business system 110 distributes the computing tasks corresponding to the business services across a resource pool composed of a large number of electronic devices, and deploys various types of virtual resources in the resource pool for external tenants to choose from. The cloud computing resource pool mainly includes: computing devices (virtualized machines containing operating systems), storage devices, network devices, etc.

[0035] The data analysis system 120 includes a terminal 121 that interacts with the user and a server 122 that communicates with the terminal 121. In this application, the server 122 in the data analysis system 120 is used to collect data from the tenant database and then further analyze and process the collected data. The server 122 can be a server cluster consisting of multiple nodes. Figure 1 The example illustrates server 122, which includes control node 123 and nodes A and B for data acquisition. The data analysis system can also employ a cloud computing model, utilizing virtual resources from a resource pool to perform big data analysis on data collected from tenant databases of the business system.

[0036] In this application, users of the data analysis system 120 can configure the source data table for data collection and the first field in the source data table to be collected in the configuration interface of the terminal 121 for tenant databases whose structure has changed, thereby obtaining the change information of the tenant database. Then, the control node 123 can obtain the change information of the tenant database and generate a destination table corresponding to the source data table based on the change information. The generated destination table is used to store the data extracted from the source data table in the tenant database. Based on this, the control node 123 can allocate tasks based on the source data table that needs to be collected, determine the collection tasks corresponding to multiple nodes, and control multiple control nodes to execute the corresponding collection tasks in parallel, so that the multiple nodes collect data in the tenant database in parallel and write the collected data into the corresponding destination table for subsequent analysis and processing.

[0037] The implementation details of the technical solution of this application are described below:

[0038] Figure 2 This is a flowchart illustrating a data acquisition method according to an embodiment of this application. The method is applied to a control node in a data analysis system, which may be an electronic device such as a server. The data analysis system acquires data from the tenant database of a tenant in a business system for analysis. Figure 2 As shown, the method includes at least steps 210 to 240, which are described in detail below:

[0039] Step 210: If the structure of the tenant database that needs to be collected in the business system changes, obtain the change information of the tenant database. The change information indicates the source data table that needs to be collected after the change and the first field in the source data table that needs to be collected.

[0040] The services provided by the business system to tenants, such as cloud computing services, facial recognition services, voice recognition services, SMS distribution services, and text classification services, are not specifically limited here.

[0041] In some embodiments, the business system can be a multi-tenant architecture, meaning a single software instance within the business system is provided to multiple tenants. Tenants can be companies, etc. Therefore, a tenant can be viewed as a user group of multiple users. For example, if the tenant is a company, multiple users belonging to that company are considered users in a user group. Since the business system provides services to multiple tenants, a tenant database is built for each tenant within the business system. The tenant database is used to store the tenant's data. Different tenant databases are isolated from each other to ensure the security of tenant data.

[0042] In other embodiments, the business system may be a single-tenant architecture, meaning that a single software instance in the business system is provided to a single tenant. In this case, a corresponding tenant database is built for the tenant in the business system.

[0043] The tenant database may include one or more data tables. In this application, the data table in the tenant database that needs to be collected is referred to as the source data table, which corresponds to the source of data collection. It can be understood that the source data table can be all the data tables in the tenant database or only some of the data tables. In some embodiments, when the business system is a multi-tenant architecture, in order to ensure the consistency of business logic, the data tables of the same type in different tenants have the same structure and are all copied from the "template table".

[0044] A change in the structure of a tenant database within a business system can be caused by changes to tables within the tenant database (e.g., adding or deleting tables), changes to fields in the source tables within the tenant database (e.g., adding or deleting fields), or changes to the field types of fields in the source tables. A change in the structure of a tenant database is referred to as a DDL change.

[0045] The data analysis system comprises multiple nodes, including a control node for scheduling the other nodes. All nodes in the data analysis system, except the control node, are communicatively connected to it. The data analysis system collects data from the tenant database of the business system and performs further analysis and processing on the collected data. In a specific embodiment, the data analysis system may be a ClickHouse system.

[0046] In this application, the field in the source data table that requires data collection is referred to as the first field. In other words, the first field refers to the field to which the data to be collected from the source data table belongs, and the data collected from the source data table can also be regarded as the content of the first field in the source data table. It can be understood that the field in the source data table that requires data collection can be some fields in the source data table or all fields in the source data table.

[0047] In some embodiments, step 210 includes: receiving change information sent by the terminal, wherein the change information is obtained by updating the data table configuration in the configuration interface displayed on the terminal.

[0048] In other words, if the structure of the tenant database is determined to have changed, the user can reconfigure the data collection according to the changed structure of the tenant database in the terminal's configuration interface. This allows the user to specify the source data table in the tenant database that needs to be collected, as well as the first field in the source data table that needs to be collected.

[0049] Figure 3 This is a schematic diagram of a configuration interface shown according to an embodiment of this application, such as... Figure 3 As shown, in the configuration interface, the "Source Table Name" option is used to specify the name of the source table from which data is to be collected; the "Destination Table Name" option is used to specify the name of the destination table to be generated from the source table; the "Tenant Database" option is used to specify the database identifier of the tenant database from which the source table for data collection originates; the "Collect Fields" option is used to specify the first field in the source table from which data is to be collected; and the "Start Collection" option is used to specify whether data collection has started for this source table.

[0050] In some embodiments, to facilitate timely data table updates and configurations by users on the terminal, when the structure of a tenant's data table changes in the business system, the business system (e.g., the control node in the business system) can send a change notification to the data analysis system (e.g., the administrator terminal of the data analysis system). This allows users acting as administrators to promptly update and configure the data tables in the terminal's configuration interface.

[0051] In some embodiments, if the data acquisition system is pre-configured to collect data from all tables and all fields in the tenant database, a structure change log for the tenant database can be obtained from the business system when the structure of the tenant database changes. This structure change log indicates the content of the structure change in the tenant database. Then, change information for the tenant database can be generated based on the tenant database's structure change log. For example, if a new field is added to a source table in the tenant database, the corresponding field is added to the initial data acquisition information corresponding to that tenant database according to the field indicated in the structure change log, thus obtaining the change information for that tenant database. The initial data acquisition information corresponding to the tenant database is generated based on the table name and fields included in the source table in the tenant database. It indicates that all tables in the tenant database need to be collected and that all fields in each table are used as the first field.

[0052] Step 220: Generate the target table corresponding to the source data table based on the first field in the source data table and the field type of the first field in the source data table. The field type of the first field in the source data table is the same as the field type of the first field in the target table.

[0053] The destination table, corresponding to the source data table, stores the data collected from the source data table. Since the first field in the source data table is the field from which data needs to be collected, the destination table must at least include a field representing the first field. Furthermore, the data type of the field representing the first field in the destination table must be the same as the data type of the first field in the source data table. This ensures the accuracy of the analysis results obtained from subsequent analysis based on the data in the destination table and avoids the impact of different data types for the same field between the source and destination tables on the analysis results.

[0054] In some embodiments, such as Figure 4 As shown, step 220 includes steps 410-430, which are detailed below:

[0055] Step 410: Determine the field type of the first field in the data analysis system based on the correspondence between the field types in the data analysis system and the field types in the business system, and the field type of the first field in the source data table.

[0056] Because data analysis systems and business data belong to different systems, and because of the differences in the programming languages ​​used by the data analysis system and the business system, the same field type may have different names in the data analysis system and in the business system. Therefore, in this case, a correspondence between the field types in the data analysis system and the field types in the business system is needed to determine the field type of the first field in the data analysis system.

[0057] The correspondence between field types in the data analysis system and field types in the business system indicates the field type name in the data analysis system and the field type name in the business system for the same field type. Therefore, by combining the correspondence between field types in the two systems and the field type of the first field in the source data table, the field type of each first field in the data analysis system can be determined.

[0058] For example, in the ClickHouse system, the field type "int8" represents signed integer data in the range of -128 to 127. In the MySQL system, the field type "Tinyint" represents signed integer data in the range of -128 to 127. Thus, if the data analysis system is a ClickHouse system and the business system is a MySQL system, and the field type of the first field in the business system is "Tinyint", then based on the correspondence between the field types in the data analysis system and the field types in the business system, it can be determined that the field type of the first field in the data analysis system is "int8".

[0059] It is understandable that, since the field types of each field in the source data table have been predefined during the construction process, the field type of the first field in the source data table can be obtained by querying the source data table.

[0060] Step 420: Generate a data table construction statement based on the first field in the source data table that needs to be collected and the field type of the first field in the data analysis system.

[0061] Step 430: Execute the data table construction statement to obtain the target table corresponding to the source data table.

[0062] The generated data table construction statement is the data definition language of the target table. The data table construction statement defines the fields included in the target table and the field types of the included fields. Correspondingly, executing the data table construction statement can obtain the target table corresponding to the source data table.

[0063] In some embodiments, if the table change information of the source data table also indicates the table name of the target table corresponding to the source data table, in step 420, the table construction statement is further generated by combining the table name of the target table indicated by the change information. That is, the table construction statement further defines the table name of the target table. Correspondingly, in step 430, the obtained table name of the target table corresponds to the table name indicated in the change information.

[0064] In other embodiments, the name of the source data table can also be used as the name of the corresponding destination table. Therefore, in step 420, the data table construction statement is generated by combining the name of the source data table. Correspondingly, the name of the obtained destination table is the same as the name of the corresponding source data table.

[0065] Please continue reading. Figure 2 Step 230: Obtain task allocation information. The task allocation information indicates the multiple nodes allocated in the data analysis system for data collection and the collection tasks of each node. The collection tasks indicate the source data table to be collected by each node and the first field to be collected from the corresponding source data table.

[0066] In some embodiments, the number of nodes for data collection can be preset by the user, and then a corresponding number of nodes can be selected from the available nodes in the data analysis system for data collection.

[0067] In some embodiments, data collection tasks can be allocated on a per-source-data-table basis, meaning that different nodes among multiple nodes used for data collection are used to collect data from different source-data-tables. In this case, before step 230, collection instruction information corresponding to each tenant database that needs to be collected is obtained. This collection instruction information indicates the source-data-tables in the corresponding tenant database that need to be collected and the first field to be collected in each source-data-table. Then, based on the collection instruction information corresponding to all tenant databases that need to be collected, all source-data-tables in the business system that need to be collected are determined. Subsequently, based on all source-data-tables in the business system that need to be collected and the number of nodes in the data analysis system used to perform data collection, tasks are allocated to determine the collection tasks corresponding to each node used to perform data collection. It is understandable that if the structure of a tenant database (let's say the first tenant database) that needs to collect data changes, the collection instruction information corresponding to the first tenant database will be the change information of the first tenant database. For example, if the first tenant database involves adding or deleting data tables, and if the added data table is the data table that needs to be collected, since the collection tasks of each node are determined on a table-by-table basis, the task allocation results will be affected.

[0068] In some embodiments, if data collection tasks are allocated on a per-source-data-table basis, before step 230, if the structural change in the tenant database involves adding a new source-data-table to be collected or deleting an existing source-data-table, the initial task allocation information is updated accordingly based on the change information of that tenant database to obtain the current task allocation information. Specifically, for example, if the newly added source-data-table indicated by the change information of the tenant database is allocated to a node performing data collection, the collection task of that node is updated accordingly. The initial task allocation information is determined based on the initial source-data-tables in all tenant databases requiring data collection and the nodes used to perform data collection in the data analysis system. It is understood that if the structural change in the tenant database requiring data collection involves changes to fields or field types in one or more source-data-tables within the tenant database, this structural change will not affect the task allocation of the nodes, and in this case, it is not necessary to update the task allocation information.

[0069] In other embodiments, data collection tasks can be allocated on a per-tenant or per-tenant-database basis, in which case different nodes among the multiple nodes used for data collection are used to collect data from different tenant databases.

[0070] In some embodiments, data collection can be performed on a tenant database of a single tenant, meaning that the object of data collection is the source data table in the same tenant database. In this case, collection tasks can be allocated on a per-source data table basis. For example, the source data tables to be collected can be allocated to multiple nodes for data collection based on the number of source data tables to be collected in the tenant database, and the difference in the number of source data tables allocated to any two nodes can be guaranteed not to exceed a first quantity threshold, so as to basically ensure that the collection tasks of multiple nodes are basically balanced.

[0071] In other embodiments, data collection can be performed on all tenant databases in the business system. In this case, collection tasks can be allocated on a per-tenant database (or per tenant). For example, based on the number of tenant databases (or the number of tenants) to be collected, the tenant databases to be collected can be allocated to multiple nodes, and the difference in the number of tenant databases allocated to any two nodes can be guaranteed not to exceed a second quantity threshold, so as to basically ensure that the collection tasks of multiple nodes are basically balanced.

[0072] In other embodiments, data collection tasks can be allocated on a per-source-data-table basis. In this case, the number of source-data-tables that need to be collected in all tenant databases is counted. Then, based on the number of source-data-tables and the number of nodes used for data collection, the source-data-tables that need to be collected are allocated equally to each node, ensuring that the difference in the number of source-data-tables allocated to any two nodes does not exceed a first quantity threshold.

[0073] Understandably, when the tenant database structure changes, the source data tables for data collection and the first field to be collected within those source tables will also change accordingly. Therefore, the data collection tasks for each node across multiple nodes are determined by allocating tasks based on the source data tables and the first field to be collected as indicated by the tenant database change information, and the source data tables and the first field to be collected in the unchanged tenant database. In other words, when the tenant database structure changes, the node's data collection tasks need to be updated accordingly.

[0074] Step 240: Control multiple nodes to execute their corresponding acquisition tasks in parallel, so as to acquire data from multiple source data tables in parallel, and write the acquired data into the destination table corresponding to the source data table.

[0075] In this application, since the collected data needs to be stored in the corresponding destination table, before step 240, the method further includes sending the destination table corresponding to the source data table indicated by the collection task of each node to the corresponding node according to the source data table that needs to be collected. In this way, it is ensured that each node can store the collected data in the corresponding destination table during the data collection process.

[0076] In some embodiments, a control node may issue a data acquisition command to a node for data acquisition, thereby enabling multiple nodes to respond to the data acquisition command and perform data acquisition in parallel.

[0077] In some embodiments, step 240 includes: obtaining acquisition configuration information, which indicates the data acquisition start time and data acquisition period; and periodically sending data acquisition instructions to multiple nodes according to the data acquisition start time and data acquisition period, so that the multiple nodes respond to the data acquisition instructions and execute the corresponding acquisition tasks in parallel.

[0078] The collection configuration information can be obtained by the user through the time configuration interface on the terminal according to actual needs. Then, the terminal sends the configured collection configuration information to the server of the data analysis system (such as the control node in the data analysis system).

[0079] The data acquisition start time refers to the time when data acquisition is first performed. Based on the data acquisition start time and the data acquisition cycle, the time for the next data acquisition can be determined. In this way, the acquisition time for each data acquisition cycle can be determined according to the data acquisition start time and the data acquisition cycle, and when the acquisition time is reached, data acquisition instructions are issued to multiple nodes.

[0080] The data collection start time and data collection period indicated in the data collection configuration information can be specified according to actual needs, and are not specifically limited here.

[0081] In other embodiments, the data acquisition configuration information can be distributed to multiple nodes used for data acquisition, so that each node automatically triggers the execution of the corresponding acquisition task when the corresponding acquisition time is reached.

[0082] Each node executes its corresponding data acquisition task by accessing the source data table indicated by the task, reading the data (or the content of the first field) from the first field to be acquired, and storing the read data in the corresponding destination table. Since each node needs to access the source data table indicated by its assigned task, the address information of the source data table must be pre-sent to the corresponding node.

[0083] In some embodiments, before step 240, the method further includes: obtaining database instance information of the tenant to which data needs to be collected, wherein the database instance information indicates the address of the source data table in the tenant database corresponding to the tenant; and, according to the collection tasks of each node among multiple nodes, sending the database instance information of the tenant to which the source data table to be collected belongs to each node, so that each node accesses the corresponding source data table according to the address indicated by the database instance information.

[0084] In this scenario, when data collection is performed on a tenant database targeting a single tenant, only one tenant needs to have their data collected. If data collection is performed on a tenant database targeting multiple tenants, multiple tenants need to have their data collected. In some embodiments, the tenants requiring data collection can be all tenants within the business system.

[0085] In some embodiments, prior to step 230, the method further includes receiving a data acquisition request sent by a terminal, the data acquisition request indicating a tenant for whom data acquisition is required.

[0086] Because the database instance information of the tenant to which the source data table belongs is pre-distributed to the corresponding node according to the node's collection task, the node is guaranteed to successfully access the corresponding tenant database according to the database instance information.

[0087] In a specific embodiment, the tenant's database instance information can indicate the IP address and port of the tenant's tenant database. The indicated IP address and port can be regarded as the address information of the source data table in the tenant database. Furthermore, the tenant's database instance information can also include the account and password authorized to access the tenant's tenant database. In this way, the node performing the data collection task can access the tenant's tenant database based on the account and password in the database instance information, which is equivalent to granting the node access permission to the tenant database, thus ensuring the data security of the tenant database.

[0088] In some embodiments, the step of obtaining the database instance information of the tenant for which data collection is required includes: if the tenant for which data collection is required in the business system changes, obtaining the updated database instance information of the tenant from the business system.

[0089] Tenant changes can include adding a new tenant, merging tenants, migrating tenants, and deleting tenants. Adding a new tenant refers to a tenant that newly rents a software instance from the business system. In this case, a tenant database needs to be configured in the business system for the new tenant.

[0090] Tenant deletion refers to the deletion of a tenant who no longer rents a software instance in the business system. In the case of tenant deletion, it is necessary to deregister the tenant database configured for that tenant in the business system.

[0091] Tenant consolidation refers to merging multiple tenants into one. In this case, the business system needs to merge the data in the tenant databases of the multiple tenants before the consolidation.

[0092] Tenant migration refers to a change in the resources allocated to a tenant in the business system due to changes in the tenant's needs. For example, if a tenant's daily active users (DAU) exceed the first DAU threshold, the tenant's database in the business system will be migrated to an availability zone with higher resource allocation. Alternatively, if a tenant's DAU remains below the second DAU threshold for a period of time, to avoid resource waste, the tenant's database in the business system will be migrated to an availability zone with lower resource allocation. Finally, if a tenant requests migration to an availability zone with higher resource allocation, the corresponding migration of the tenant's database in the business system will be performed.

[0093] Understandably, if the tenant for whom data collection is required changes in the business system, such as adding a new tenant, the tenant database of the new tenant cannot be accessed in step 240 without obtaining the database instance information of the tenant database corresponding to the new tenant from the business system. Similarly, if the tenant for whom data collection is required is merged, the address of the tenant database may change because the tenant database of the tenant in the business system has been merged with other tenant databases. If the updated database instance information of the tenant is not obtained from the business system, the tenant database of the tenant may be inaccessible in step 240.

[0094] Therefore, in this application, when the tenant for which data collection is required in the business system changes, the database instance information of the changed tenant is obtained from the business system. This ensures that the database instance information stored in the data analysis system is the latest database instance information of the tenant for which data collection is required. In this way, the latest database instance information of the tenant for which data collection is required is sent to the nodes used for data collection, thereby ensuring that the nodes can accurately collect data and guaranteeing the success rate and accuracy of data collection.

[0095] Understandably, if a new tenant is added to the business system that requires data collection, the user needs to configure the change information of the tenant database corresponding to that tenant in the configuration interface, and generate the destination table corresponding to each source data table in the tenant database corresponding to the newly added tenant according to the process in step 220.

[0096] In some embodiments, if a tenant in the business system changes, the business system can send a tenant change notification to the control node in the data analysis system. In response to the tenant change notification, the control node in the data analysis system can obtain the database instance information of the changed tenant from the business system.

[0097] In other embodiments, the control node in the data analysis system can be configured to periodically and proactively request tenant change logs from the business system. If the tenant change logs indicate a change in the business system, the control node proactively retrieves the changed tenant's database instance information from the business system and replaces the previous database instance information with the currently retrieved information. This ensures that the database instance information stored by the control node is the latest information from the business system. Subsequently, the control node can distribute its stored database instance information to the nodes used for data collection.

[0098] Understandably, if tasks are assigned to multiple nodes performing data collection based on a tenant database or a tenant as a unit, after obtaining the updated tenant database instance information, the number of tenant databases that need to be collected is recalculated based on the updated tenant database instance information, and tasks are then assigned in combination with the number of nodes used to perform data collection, thereby updating the task allocation information.

[0099] In some embodiments, after step 240, the method further includes merging the data in the destination table across multiple nodes.

[0100] As described above, the data collected by each node is written to the destination table corresponding to the source data table from which the data originated. Therefore, the data in the destination tables of multiple nodes can be merged to facilitate further analysis.

[0101] In some embodiments, the merging may be to generate a distributed table based on the destination table in multiple nodes. In some embodiments, if data is collected from multiple tenant databases and the source data tables in different tenant databases are of the same type, the merging may also be to merge the data in the destination tables corresponding to the source data tables of the same type to generate the corresponding distributed table.

[0102] Figure 5 This is a schematic diagram illustrating data collection from a tenant database for a single tenant, according to an embodiment of this application. Figure 5 As shown, the nodes used for data acquisition in the data analysis system include node A, node B, and node C. Assume that in... Figure 5In this embodiment, data collection is performed on a tenant database f for tenant F. The source data tables to be collected in tenant database f include source data tables D1-D9. Node A's collection task instructs node A to collect data from source data tables D1, D2, and D3; node B's collection task instructs node B to collect data from source data tables D4, D5, and D6; and node C's collection task instructs node C to collect data from source data tables D7, D8, and D9. In this case, nodes A, B, and C can execute their respective collection tasks in parallel. Afterward, the data collected by nodes A, B, and C can be merged to generate a corresponding distributed table.

[0103] Figure 6 This is a schematic diagram illustrating data collection from a tenant database for multiple tenants, according to an embodiment of this application. Figure 6 In this embodiment, data is collected from tenant databases f1 (for tenant F1), f2 (for tenant F2), f3 (for tenant F3), f4 (for tenant F4), f5 (for tenant F5), and f6 (for tenant F6). Node A's collection task instructs it to collect data from tenant databases f1 and f2; Node B's collection task instructs it to collect data from tenant databases f3 and f4; and Node C's collection task instructs it to collect data from tenant databases f5 and f6. In this scenario, nodes A, B, and C can execute their respective collection tasks in parallel. Afterward, the data collected by nodes A, B, and C can be merged to generate a distributed table.

[0104] In some embodiments, if the data analysis system periodically collects data from the business system, steps 210-220 can be executed during the non-data collection period of a cycle to ensure that the target table is up-to-date in the next data collection period, thereby ensuring the validity of the data in the target table.

[0105] In this application, when the structure of the tenant database in the business system changes, the change information of the tenant database indicates the source data table that needs data collection and the first field in the source data table that needs data collection. A destination table corresponding to the source data table is generated, and the field type of the first field in the source data table is the same as that in the destination table. Thus, during the data collection process, data collected from the newly added source data table can be saved to the corresponding destination table, ensuring the accuracy of the data in the destination table. Since the destination table corresponding to the structure of the source data table is generated based on the tenant database change information before data collection, there is no need for secondary processing of the data in the destination table to adapt to the tenant database structure change after data collection and before analysis. Compared to related technologies that perform secondary analysis and processing of the collected data based on the changes in the tenant database structure after data collection, this solution improves the efficiency of data collection, especially when the data collection volume is large. Therefore, adopting this solution can effectively ensure the accuracy and efficiency of data collection even when the tenant database structure changes. Furthermore, in this application, assigning corresponding collection tasks to multiple nodes and executing these tasks in parallel on multiple nodes allows for the parallel collection of data from multiple source data tables, which can further improve data collection efficiency.

[0106] In some embodiments, when periodically collecting data from the tenant database, steps 210-220 can be executed during non-data collection periods, and steps 230-240 can be executed during data collection periods, thereby ensuring that the target table update and data collection in the data analysis system are carried out in an orderly manner.

[0107] In some embodiments, such as Figure 7 As shown, step 230 includes steps 710-730, which are detailed below:

[0108] Step 710: Based on the number of nodes used for data acquisition, allocate a corresponding number of nodes in the data analysis system for data acquisition.

[0109] The number of nodes used for data collection can be preset, and the specific number can be set as needed, such as two or three. Then, the corresponding number of nodes are specified from the data analysis system for data collection.

[0110] Step 720: Based on the source data table in the tenant database that needs to be collected, determine the collection tasks for each node among multiple nodes.

[0111] In some embodiments, the number of source data tables to be collected for each node can be determined based on the total number of source data tables to be collected and the number of nodes used for data collection; then, a corresponding number of source data tables to be collected are allocated to each node according to the corresponding number. In a specific embodiment, the number of source data tables to be collected can be allocated to each node on an average basis. For example, if the total number of source data tables to be collected is K1 and the number of nodes used for data collection is K2, then the number of source data tables to be collected for each node is K1 / K2, or the difference between the source data tables allocated to any two nodes is guaranteed not to exceed a first quantity threshold.

[0112] In other embodiments, since different nodes have varying amounts of available processing resources, the number of source data tables for data collection by each node can be proportionally allocated according to the proportion of available processing resources of each node. For example, if the nodes used for data table collection include nodes A, B, and C, and the ratio of available processing resources L1 of node A, L2 of node B, and L3 of node C is L1:L2:L3, then based on the total number of source data tables requiring data collection, the source data tables are allocated according to the ratio of available processing resources, such that the ratio of the number of source data tables S1 for node A, S2 for node B, and S3 for node C is S1:S2:S3 = L1:L2:L3. The available processing resources of each node can be represented by the remaining available memory.

[0113] In other embodiments, the number of tenant databases to be collected from each node can be determined based on the total number of tenant databases requiring data collection and the number of nodes used for data collection; then, a corresponding number of tenant databases requiring data collection are allocated to each node according to the corresponding number. In specific embodiments, the number of tenant databases requiring data collection can be allocated to each node in a substantially even manner, for example, ensuring that the difference between the number of tenant databases allocated to any two nodes does not exceed a second quantity threshold.

[0114] Step 730: Generate task allocation information based on the node identifiers of the assigned multiple nodes and the data collection tasks of each node among the multiple nodes.

[0115] The generated task allocation information facilitates subsequent effective control of multiple nodes to execute their corresponding data collection tasks in parallel.

[0116] In some embodiments, the method further includes: obtaining load information of each node in the plurality of nodes; adjusting the number of nodes used for data acquisition in the data analysis system according to the load information of each node in the plurality of nodes, and adjusting the acquisition tasks of each node used for data acquisition; and updating task allocation information according to the adjusted number of nodes used for data acquisition and the adjusted acquisition tasks of each node used for data acquisition.

[0117] The number of nodes used for data collection is adjusted based on the load information of each node, and the collection tasks of the nodes are adjusted accordingly. This avoids the nodes being overloaded or underloaded due to the collection tasks, and ensures that the load of different nodes is basically balanced.

[0118] In some embodiments, the number of nodes and the data acquisition tasks of the nodes can be adjusted as follows: if there are nodes among the multiple nodes whose load information indicates a load exceeding a first load threshold, then new nodes are added for data acquisition, and the data acquisition tasks of each node are reallocated; if there are nodes among the multiple nodes whose load information indicates a load not exceeding a second load threshold, then the number of nodes used for data acquisition is reduced, and the data acquisition tasks of each node are reallocated; wherein, the first load threshold is greater than the second load threshold. In other words, when the node load is high, new nodes are added for data acquisition to share the load of the nodes with high loads; conversely, when the node load is low, nodes used for data acquisition are reduced to improve node utilization.

[0119] Figure 8 This is a schematic diagram illustrating the adjustment of data acquisition tasks based on a scaling strategy, according to an embodiment of this application. Figure 8 As shown, based on the preset scaling strategy and node load information, the number of nodes used for data collection is adjusted. The preset scaling strategy, for example, is to increase the number of nodes used for data collection when the node load is high, and to decrease the number of nodes used for data collection when the node load is low.

[0120] Furthermore, Figure 8The example illustrates a scenario where two nodes with IDs 0 and 1 need to collect data before adjustment. Node ID 0 collects data from tenant database f1 (tenant ID 123) and tenant database f4 (tenant ID 126); node ID 1 collects data from tenant database f2 (tenant ID 124) and tenant database f3 (tenant ID 125). Assuming node 1 has a high load in this case, according to the preset scaling strategy, a new node with ID 2 is added for data collection, thus scaling up the node. After scaling up, the data collection task of node ID 0 remains unchanged, node ID 1 collects data from tenant database f2 (tenant ID 124), and node ID 2 collects data from tenant database f3 (tenant ID 125). Figure 8 The example also illustrates the IP address and port corresponding to the tenant database, used to guide the assigned nodes to access the corresponding tenant database. After adjustment, in the next collection cycle, each node is controlled to collect data according to the adjusted collection task.

[0121] Figure 9 This is a block diagram of a data analysis system according to an embodiment of this application. Figure 9 As shown, the data analysis system 120 includes a DDL change module 910, a tenant change module 920, a data acquisition control module 930, a parameter configuration module 940, and a data acquisition execution module 950. The DDL change module 910 and the parameter configuration module 940 can be configured in the terminals of the data analysis system 120, the tenant change module 920 and the data acquisition control module 930 can be configured in the control nodes of the data analysis system 120, and the data acquisition execution module 950 can be configured in the nodes of the data analysis system 120 used for data acquisition.

[0122] The DDL change module 910 is used to handle database table structure changes. Specifically, in the event of a structural change in the tenant database in the business system, the DDL change module 910 can provide the tenant database change information to the acquisition control module 930. The tenant database change information is used to indicate the source data table in the tenant database that needs to be collected after the change and the first field in the source data table that needs to be collected.

[0123] The DDL change module 910 is used to handle tenant changes. Specifically, the tenant change module 920 can provide the database instance information of the changed tenant to the acquisition and control module 930 when a tenant changes in the business system. The tenant change module 920 can obtain the database instance information of the changed tenant from the business system based on the API (Application Programming Interface) provided by the business system.

[0124] The parameter configuration module 940 is used to provide the data acquisition start time, data acquisition cycle, and the number of nodes initially set for data acquisition to the acquisition control module.

[0125] The data acquisition and execution module 950 is responsible for executing specific commands, such as executing data table construction statements and performing data acquisition tasks.

[0126] The data acquisition control module 930 generates data acquisition commands based on the inputs from the DDL change module 910 and the parameter configuration module 940, and then sends these commands to the data acquisition execution module 950 for execution. The module 930 can also flexibly scale up or down based on the load of the data acquisition execution module 950 during the data acquisition process.

[0127] Figure 10 This is a timing diagram illustrating data acquisition in a DDL change scenario according to an embodiment of this application. Figure 10 The corresponding implementation consists of business services provided by the business system, and a tenant database within the business system. Figure 9 The DDL change module, acquisition control module, and acquisition execution module in the corresponding embodiment interact with each other. Figure 10 The corresponding embodiment illustrates the data collection process when a new source data table is added in a DDL change scenario. The data collection process for other DDL change scenarios (e.g., adding or deleting fields in an existing source data table) is similar. Figure 10 As shown, it includes:

[0128] Step 1.1: Develop new features. This is equivalent to updating business services in the business system.

[0129] Step 1.2: Create a new source data table. In this embodiment, after the business service in the business system is updated, the updated business service requests the creation of a new source data table in the tenant database for which data collection is required.

[0130] Step 1.3: Configure the source data table and the first field to be collected. That is, users can configure the first field to be collected for newly created source data tables in the terminal's configuration interface. Of course, if multiple source data tables are added, the first field to be collected should be configured for each new source data table.

[0131] Step 1.4: Save the source data table and the first field of the configuration.

[0132] Step 1.5, Interface Display. By displaying the configured source data table and the first field in the terminal interface, users can easily understand the configured changes to the tenant database.

[0133] Step 1.6: Read the source data table and the first field.

[0134] Step 1.7: Read the field type of the first field in the source data table. That is, query the field type of the first field in the source data table.

[0135] Step 1.8: Return the field type of the first field in the source data table.

[0136] Step 1.9: Generate data table construction statements. In this step, based on the field type of the first field in the source data table, the field type of the first field in the data analysis system is determined. Then, data table construction statements are generated based on the first field and its field type in the data analysis system.

[0137] Step 1.10: Execute the data table construction statement. By executing the data table construction statement, the target table corresponding to the source data table is obtained.

[0138] Step 1.11: Verify the source data table and the first field. In this step, the destination table and the corresponding source data table are verified, and the first field of the destination table is verified against the first field of the corresponding source data table. That is, on the one hand, it is verified whether the tenant database has a source data table corresponding to the destination table, and on the other hand, it is verified whether the first field of the destination table exists in the corresponding source data table.

[0139] Step 1.12: Return the verification result. Then, based on the verification result, further processing is performed on the target table. For example, if it is determined that a certain first field in the target table does not exist in the corresponding source data table, then that field in the target table is set to null; if, for a target table, it is determined that there is no corresponding source data table in the tenant database, then an instruction message is generated indicating that data collection from that source data table will not be performed.

[0140] Step 1.13: Generate Data Acquisition SQL. In this step, based on the source data table to be collected and the first field to be collected in the source data table as indicated by the acquisition task corresponding to each node, data acquisition SQL is generated specifically for each node.

[0141] Step 1.14: Execute the data acquisition SQL. Data under the first field in the corresponding source data table is collected by executing the data acquisition SQL. It is worth noting that before step 1.14, the acquisition control module distributes the database instance information of each tenant to the acquisition control module to access the corresponding tenant database according to the address indicated by the database instance information. In step 1.14, multiple nodes are controlled to execute the corresponding data acquisition SQL in parallel to collect data concurrently. It is also worth mentioning that in step 1.14, the acquisition control module also periodically controls the acquisition execution module to execute the data acquisition SQL according to the data acquisition start time and data acquisition period provided by the parameter configuration module 940.

[0142] Step 1.15: Write the collected data to the destination table. This means writing the collected data to the destination table corresponding to the source data table from which the data originated.

[0143] Step 1.16: Return the data collection results.

[0144] In this embodiment, when a new source data table requiring data collection is added, a corresponding destination table is generated based on the fields in the configured source data table that need data collection. This allows data collected from the newly added source data table to be saved to the corresponding destination table during the data collection process. If changes to the tenant database structure are addressed only after data collection, the amount of data processing required to address these changes would be substantial, especially with large datasets. Therefore, the solution in this embodiment effectively ensures both accuracy and efficiency in handling tenant database structure changes.

[0145] Figure 11 This is a timing diagram illustrating data collection in a tenant change scenario according to an embodiment of this application. Figure 11 The corresponding implementation examples consist of business systems and Figure 9 The DDL change module, acquisition control module, and acquisition execution module in the corresponding embodiment interact with each other. Figure 11 In the corresponding embodiment, steps 2.1-2.4 are the processing steps involved when a new tenant is added to the business system; specifically,

[0146] Step 2.1: Create a new tenant.

[0147] Step 2.2: Request tenant creation. After the business system requests the creation of a tenant, a tenant database is built for the newly created tenant.

[0148] Step 2.3: Save the tenant's database instance information. In this step, the tenant change module can obtain the database instance information of the newly created tenant from the business system through the application programming interface provided by the business system.

[0149] Step 2.4: Return the creation result.

[0150] exist Figure 11 In the corresponding embodiment, steps 3.1-3.4 are the processing procedures involved in the case of tenant merging or tenant migration in the business system;

[0151] Step 3.1, Tenant Merging / Tenant Migration.

[0152] Step 3.2: Request an update of the tenant.

[0153] Step 3.3: Update the tenant's database instance information. In this step, the tenant change module can obtain the updated tenant's database instance information from the business system through the application programming interface provided by the business system.

[0154] Step 3.4: Return the updated results.

[0155] exist Figure 11 In the corresponding embodiment, steps 4.1-4.7 constitute the process of data collection after a tenant change in the business system. In this embodiment, the data analysis system collects data from all tenant databases within the business system. Wherein:

[0156] Step 4.1: Read the database instance information of all tenants.

[0157] Step 4.2: Return the database instance information for all tenants.

[0158] Step 4.3: Data Acquisition Task Allocation. In this step, based on the source data table requiring data acquisition and the pre-defined number of nodes for data acquisition, tasks are allocated to determine the nodes used for data acquisition and the corresponding acquisition tasks for each node. Of course, if the tenant database requiring data acquisition undergoes structural changes, a corresponding destination table needs to be generated based on the tenant database change information. The specific process is described above and will not be repeated here.

[0159] Step 4.4: Traverse the database instance information of tenants under each collection task.

[0160] Step 4.5: Issue data acquisition commands; In this step, the acquisition control module, in conjunction with the data acquisition start time and data acquisition cycle provided by the parameter configuration module 940, periodically issues data acquisition commands.

[0161] Step 4.6: Execute the data acquisition task in parallel.

[0162] Step 4.7: Return the data collection results.

[0163] In this embodiment, the database instance information of the changed tenant is obtained before data collection, and data collection is performed using this information, rather than addressing tenant changes only after data collection. If tenant changes are addressed only after data collection, on the one hand, if the data collection volume is large, the workload and data volume for post-processing become significant, leading to time-consuming, labor-intensive processes and a higher risk of errors; on the other hand, the collected data may be incomplete or inaccurate, requiring secondary data collection. Therefore, in this embodiment, the database instance information of the changed tenant in the business system is updated to the data analysis system before data collection is executed. Thus, the data analysis system collects data based on the updated database instance information of the tenant during data collection, eliminating the need for post-collection processing. This improves the efficiency of data collection in tenant change scenarios, thereby ensuring the efficiency of subsequent data analysis.

[0164] In one specific embodiment, Figures 9-10 The data analysis system in this embodiment can be the ClickHouse system. In related technologies, data acquisition is generally performed using the Sqoop system. Practical comparisons show that if 100 million data entries need to be collected from a single table with 10 fields, the ClickHouse system takes less than 10 minutes, while the Sqoop system takes over 100 minutes. If 20 million data entries need to be collected from 36 data tables with 150 fields, the ClickHouse system takes less than 20 minutes, while the Sqoop system takes over 180 minutes. This demonstrates that using the ClickHouse system can effectively shorten data acquisition time. Further analysis reveals that because the Sqoop system supports serial acquisition while the ClickHouse system supports parallel data acquisition, using the ClickHouse system for parallel data acquisition can significantly shorten acquisition time and improve efficiency.

[0165] The following describes an apparatus embodiment of this application, which can be used to perform the methods described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments described in the above embodiments of this application.

[0166] Figure 12 This is a block diagram of a data acquisition device according to an embodiment of this application. The data acquisition device is applied to a control node in a data analysis system, which acquires data from the tenant database of a tenant in a business system for analysis. Figure 12 As shown, the data acquisition device includes: a change information acquisition module 1210, used to acquire change information of the tenant database if the structure of the tenant database to be acquired in the business system changes, the change information indicating the source data table to be acquired after the change and the first field in the source data table to be acquired; a destination table generation module 1220, used to generate a destination table corresponding to the source data table according to the first field in the source data table and the field type of the first field in the source data table, the field type of the first field in the source data table is the same as the field type of the first field in the destination table; a task allocation information acquisition module 1230, used to acquire task allocation information, the task allocation information indicating multiple nodes allocated in the data analysis system for data acquisition and the acquisition tasks of each node, the acquisition tasks indicating the source data table to be acquired by each node; and a parallel acquisition module 1240, used to control multiple nodes to execute their corresponding acquisition tasks in parallel, so as to acquire data from multiple source data tables in parallel and write the acquired data into the destination table corresponding to the source data table.

[0167] In some embodiments, the destination table generation module 1220 includes: a field type determination unit, used to determine the field type of the first field in the data analysis system based on the correspondence between field types in the data analysis system and field types in the business system, and the field type of the first field in the source data table; a data table construction statement generation unit, used to generate a data table construction statement based on the first field in the source data table that needs to be collected and the field type of the first field in the data analysis system; and an execution unit, used to execute the data table construction statement to obtain the destination table corresponding to the source data table.

[0168] In some embodiments, the data acquisition device further includes: a database instance information acquisition module, used to acquire database instance information of the tenant to be acquired, wherein the database instance information indicates the address of the source data table in the tenant database corresponding to the tenant; and a distribution module, used to distribute the database instance information of the tenant to which the source data table to be acquired belongs to each node according to the acquisition task of each node among multiple nodes, so that each node can access the corresponding source data table according to the address indicated by the database instance information.

[0169] In some embodiments, the database instance information acquisition module is further configured to: if the tenant for which data collection is required in the business system changes, obtain the updated database instance information of the tenant from the business system.

[0170] In some embodiments, the task allocation information acquisition module 1230 includes: an allocation unit, configured to allocate a corresponding number of nodes in the data analysis system for data acquisition based on the number of nodes used for data acquisition; a acquisition task determination unit, configured to determine the acquisition task of each node among the multiple nodes based on the source data table in the tenant database that needs to be acquired; and a task allocation information generation unit, configured to generate task allocation information based on the node identifiers of the allocated multiple nodes and the acquisition tasks of each node among the multiple nodes.

[0171] In some embodiments, the data acquisition device further includes: a load information acquisition unit, configured to acquire load information of each node among multiple nodes; an adjustment unit, configured to adjust the number of nodes used for data acquisition in the data analysis system according to the load information of each node among multiple nodes, and adjust the acquisition tasks of each node used for data acquisition; and an update unit, configured to update task allocation information according to the adjusted number of nodes used for data acquisition and the adjusted acquisition tasks of each node used for data acquisition.

[0172] In some embodiments, the updating unit is further configured to: if there is a node among the multiple nodes whose load, as indicated by the load information, exceeds a first load threshold, then add a node for data acquisition and reallocate the acquisition tasks of each node; if there is a node among the multiple nodes whose load, as indicated by the load information, does not exceed a second load threshold, then reduce the number of nodes used for data acquisition and reallocate the acquisition tasks of each node; wherein the first load threshold is greater than the second load threshold. In some embodiments, the parallel acquisition module 1240 includes: an acquisition configuration information acquisition unit, configured to acquire acquisition configuration information, which indicates the data acquisition start time and data acquisition period; and an instruction issuing module, configured to periodically issue data acquisition instructions to multiple nodes according to the data acquisition start time and data acquisition period, so that multiple nodes respond to the data acquisition instructions and execute the corresponding acquisition tasks in parallel.

[0173] In some embodiments, the change information acquisition module 1210 is further configured to receive change information sent by the terminal, wherein the change information is obtained by updating the data table configuration in the configuration interface displayed on the terminal.

[0174] In some embodiments, the data acquisition device further includes merging data in a destination table across multiple nodes.

[0175] Figure 13 A schematic diagram of a computer system suitable for implementing the electronic device of the embodiments of this application is shown. This electronic device can be a control node in a data analysis system, and it is used to execute the data acquisition method provided in this application. It should be noted that... Figure 13The computer system 1300 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0176] like Figure 13 As shown, the computer system 1300 includes a Central Processing Unit (CPU) 1301, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 1302 or programs loaded from storage portion 1308 into Random Access Memory (RAM) 1303. The RAM 1303 also stores various programs and data required for system operation. The CPU 1301, ROM 1302, and RAM 1303 are interconnected via a bus 1304. An Input / Output (I / O) interface 1305 is also connected to the bus 1304.

[0177] The following components are connected to I / O interface 1305: an input section 1306 including a keyboard, mouse, etc.; an output section 1307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1308 including a hard disk, etc.; and a communication section 1309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to I / O interface 1305 as needed. Removable media 1311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1310 as needed so that computer programs read from them can be installed into storage section 1308 as needed.

[0178] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1309, and / or installed from removable medium 1311. When the computer program is executed by central processing unit (CPU) 1301, it performs various functions defined in the system of this application.

[0179] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0180] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0181] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0182] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries computer-readable instructions that, when executed by a processor, implement the methods in any of the above embodiments.

[0183] According to one aspect of the embodiments of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the methods of any of the above embodiments.

[0184] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0185] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0186] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0187] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data acquisition method, characterized in that, A control node applied in a data analysis system, wherein the data analysis system collects data from the tenant database of a tenant in a business system for analysis; the method includes: If the structure of the tenant database that needs to be collected in the business system changes, obtain the change information of the tenant database. The change information indicates the source data table that needs to be collected after the change and the first field in the source data table that needs to be collected. Based on the first field in the source data table and the field type of the first field in the source data table, a destination table corresponding to the source data table is generated, wherein the field type of the first field in the source data table is the same as the field type of the first field in the destination table. Obtain task allocation information, which indicates multiple nodes allocated in the data analysis system for data collection and the data collection tasks of each node. The data collection tasks indicate the source data table to be collected by each node and the first field to be collected from the corresponding source data table. The system controls the multiple nodes to execute their respective data acquisition tasks in parallel, thereby acquiring data from multiple source data tables in parallel and writing the acquired data into the destination table corresponding to the source data table.

2. The method according to claim 1, characterized in that, The step of generating a destination table corresponding to the source data table based on the first field in the source data table and the field type of the first field in the source data table includes: Based on the correspondence between field types in the data analysis system and field types in the business system, and the field type of the first field in the source data table, the field type of the first field in the data analysis system is determined. Based on the first field in the source data table that needs to be collected and the field type of the first field in the data analysis system, generate a data table construction statement; Execute the data table construction statement to obtain the target table corresponding to the source data table.

3. The method according to claim 1, characterized in that, Before controlling the multiple nodes to execute their corresponding acquisition tasks in parallel to acquire data from the multiple source data tables in parallel, and before writing the data acquired by the multiple nodes into the destination table corresponding to the source data table, the method further includes: Obtain the database instance information of the tenant for which data collection is required. The database instance information indicates the address of the source data table in the tenant database corresponding to the tenant. Based on the collection tasks of each of the multiple nodes, the database instance information of the tenant to which the source data table to be collected belongs is sent to each node, so that each node can access the corresponding source data table according to the address indicated by the database instance information.

4. The method according to claim 3, characterized in that, The step of obtaining the database instance information of the tenant for whom data collection is required includes: If the tenant for whom data collection is required in the business system changes, the updated database instance information of the tenant is obtained from the business system.

5. The method according to claim 1, characterized in that, The process of obtaining task allocation information includes: Based on the number of nodes used for data acquisition, a corresponding number of nodes in the data analysis system are allocated for data acquisition. Based on the source data table in the tenant database that needs to be collected, determine the collection task for each of the multiple nodes; The task allocation information is generated based on the node identifiers of the assigned multiple nodes and the data acquisition tasks of each of the multiple nodes.

6. The method according to claim 1 or 5, characterized in that, The method further includes: Obtain the load information of each of the multiple nodes; The number of nodes used for data acquisition in the data analysis system is adjusted according to the load information of each node among the multiple nodes, and the acquisition tasks of each node used for data acquisition are also adjusted. The task allocation information is updated based on the adjusted number of nodes used for data acquisition and the adjusted acquisition tasks of each node.

7. The method according to claim 6, characterized in that, The step of adjusting the number of nodes used for data acquisition in the data analysis system based on the load information of each of the plurality of nodes, and adjusting the acquisition tasks of each node used for data acquisition, includes: If any of the multiple nodes has a load exceeding the first load threshold as indicated by the load information, then a new node for data acquisition is added, and the acquisition tasks of each node are reallocated. If any of the plurality of nodes has a load that does not exceed the second load threshold as indicated by the load information, the number of nodes used for data acquisition is reduced and the acquisition tasks of each node are reallocated; wherein, the first load threshold is greater than the second load threshold.

8. The method according to claim 1, characterized in that, The control of the multiple nodes to execute their corresponding acquisition tasks in parallel, to acquire data from multiple source data tables in parallel, and to write the acquired data into a destination table corresponding to the source data table, includes: Obtain the data acquisition configuration information, which indicates the data acquisition start time and data acquisition period; Based on the data acquisition start time and the data acquisition period, data acquisition instructions are periodically sent to the multiple nodes so that the multiple nodes respond to the data acquisition instructions and execute the corresponding acquisition tasks in parallel.

9. The method according to claim 1, characterized in that, The step of obtaining the change information of the tenant database includes: The terminal receives change information, which is obtained by updating the data table configuration in the configuration interface displayed on the terminal.

10. The method according to claim 1, characterized in that, After controlling the multiple nodes to execute their corresponding data acquisition tasks in parallel to acquire data from multiple source data tables and writing the acquired data into a destination table corresponding to the source data table, the method further includes: The data in the destination table of the multiple nodes are merged.

11. A data acquisition device, characterized in that, A control node applied in a data analysis system, wherein the data analysis system collects data from the tenant database of a tenant in a business system for analysis; the device includes: The change information acquisition module is used to acquire the change information of the tenant database if the structure of the tenant database to be collected in the business system changes. The change information indicates the source data table to be collected after the change and the first field in the source data table to be collected. The destination table generation module is used to generate a destination table corresponding to the source data table based on the first field in the source data table and the field type of the first field in the source data table, wherein the field type of the first field in the source data table is the same as the field type of the first field in the destination table. The task allocation information acquisition module is used to acquire task allocation information, which indicates multiple nodes allocated in the data analysis system for data collection and the data collection tasks of each node. The data collection tasks indicate the source data table for data collection by each node. The parallel acquisition module is used to control the multiple nodes to execute their corresponding acquisition tasks in parallel, so as to acquire data from multiple source data tables in parallel and write the acquired data into the destination table corresponding to the source data table.

12. An electronic device, characterized in that, include: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1-10.

13. A computer-readable storage medium storing computer-readable instructions thereon, characterized in that, When the computer-readable instructions are executed by a processor, the method as described in any one of claims 1-12 is implemented.

14. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method of any one of claims 1-10.

Images