Tenant-level resource isolation data write channel management method and system

Abstract

This invention provides a method and system for managing tenant-level resource isolation data write channels, including: Step 1: Deploying a load balancing cluster; Step 2: Performing multi-tenant isolation; Step 3: After data is written through the REST gateway, it is sent to the data cache layer, then consumed by the Spark Streaming program and written to HDFS, using the combination of the data cache layer and the Spark Streaming program as the data write channel; Step 4: Setting up consumer groups and consumer instances to change the data consumption speed; Step 5: Deploying a file merging service to monitor the number and size of files, automatically triggering small file merging when the number or size of files reaches a preset threshold. This invention allows for on-demand configuration of computing resources, improving data throughput and write efficiency, and supports tenant-level resource isolation configuration.

Description

Technical Field

[0001] This invention relates to the field of data management technology, and more specifically, to a method and system for managing data write channels with tenant-level resource isolation. Background Technology

[0002] With the advancement of industrial informatization and intelligentization, the amount of data generated in industrial production, as well as the value of this production data, are increasing daily. For existing Hadoop-based Hive data warehouses, efficiently writing data from different sources to their corresponding storage locations while avoiding small file issues remains a significant challenge, especially in high-frequency, near real-time data writing scenarios.

[0003] In the prior art, Chinese patent application number 202310745568.5 discloses "a method and apparatus for writing data to a big data warehouse". Although it can write data to HDFS, the timeliness of data writing to disk is uncontrollable and the applicable scenarios are limited. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for managing data write channels with tenant-level resource isolation.

[0005] The data write channel management method for tenant-level resource isolation provided by the present invention includes:

[0006] Step 1: Deploy a load balancing cluster;

[0007] Step 2: Implement multi-tenant isolation;

[0008] Step 3: After the data is written through the REST gateway, it is sent to the data cache layer Kafka, and then consumed and written to HDFS by the Spark Streaming program. The combination of the data cache layer and the Spark Streaming program is used as the data writing channel.

[0009] Step 4: Configure consumer groups and consumer instances to change the data consumption rate;

[0010] Step 5: Deploy a file merging service to monitor the number and size of files. When the number or size of files reaches a preset threshold, small file merging will be automatically triggered.

[0011] Preferably, data is written using a RESTful API, the REST service is deployed in a cluster mode to support horizontal scaling, and Nginx is used as a load balancer to provide a unified access address to the outside world.

[0012] Preferably, the data files are stored on the Hadoop Distributed File System (HDFS). The data files stored on HDFS provide query services to the outside world through the Impala query engine. Data tables are first authorized on Impala before access is granted. Different tenants can only see the tables and data under their own tenants. When writing data, it is possible to configure the write channel at the tenant level. The data writes of different tenants do not affect each other. Tenants adjust the resources used by the channel according to their own data write volume and configure it on demand.

[0013] Preferably, the STS provides a web management page to manage data write channels and configure write policies. The configuration relationship between tenants and data write channels is one-to-one or many-to-one. After the REST gateway receives a user's request to write data, it queries the corresponding write policy according to the user's tenant and sends the data to the corresponding write channel according to the write policy until it is finally written to disk. The channels are isolated from each other and the whole process does not affect each other.

[0014] Preferably, when data is written to the Kafka caching layer, it is stored in different topics, consumed by Spark Streaming programs and written to HDFS. A group of Spark Streaming programs is called a consumer group. Each consumer group is responsible for consuming data from one or more topics. The consumer group is configured with a corresponding number of consumer instances according to the actual amount of data consumed. Within a preset range, the consumption speed increases linearly as the number of consumer instances increases.

[0015] A consumer group contains one or more Spark Streaming instances. Each consumer group subscribes to data from one or more topics in Kafka. Spark Streaming instances periodically pull data from the subscribed topics in Kafka and write it to HDFS. If data backlog occurs, the number of consumer instances can be increased on the consumer group management page to improve data consumption speed.

[0016] The tenant-level resource isolation data write channel management system provided by the present invention includes:

[0017] Module M1: Deploy a load balancing cluster;

[0018] Module M2: Performs multi-tenant isolation;

[0019] Module M3: After data is written through the REST gateway, it is sent to the data cache layer Kafka, and then consumed and written to HDFS by the Spark Streaming program. The combination of the data cache layer and the Spark Streaming program is used as the data writing channel.

[0020] Module M4: Configure consumer groups and consumer instances to change the data consumption rate;

[0021] Module M5: Deploys the file merging service, which is responsible for monitoring the number and size of files. When the number or size of files reaches a preset threshold, it automatically triggers the merging of small files.

[0022] Preferably, data is written using a RESTful API, the REST service is deployed in a cluster mode to support horizontal scaling, and Nginx is used as a load balancer to provide a unified access address to the outside world.

[0023] Preferably, the data files are stored on the Hadoop Distributed File System (HDFS). The data files stored on HDFS provide query services to the outside world through the Impala query engine. Data tables are first authorized on Impala before access is granted. Different tenants can only see the tables and data under their own tenants. When writing data, it is possible to configure the write channel at the tenant level. The data writes of different tenants do not affect each other. Tenants adjust the resources used by the channel according to their own data write volume and configure it on demand.

[0024] Preferably, the STS provides a web management page to manage data write channels and configure write policies. The configuration relationship between tenants and data write channels is one-to-one or many-to-one. After the REST gateway receives a user's request to write data, it queries the corresponding write policy according to the user's tenant and sends the data to the corresponding write channel according to the write policy until it is finally written to disk. The channels are isolated from each other and the whole process does not affect each other.

[0025] Preferably, when data is written to the Kafka caching layer, it is stored in different topics, consumed by Spark Streaming programs and written to HDFS. A group of Spark Streaming programs is called a consumer group. Each consumer group is responsible for consuming data from one or more topics. The consumer group is configured with a corresponding number of consumer instances according to the actual amount of data consumed. Within a preset range, the consumption speed increases linearly as the number of consumer instances increases.

[0026] A consumer group contains one or more Spark Streaming instances. Each consumer group subscribes to data from one or more topics in Kafka. Spark Streaming instances periodically pull data from the subscribed topics in Kafka and write it to HDFS. If data backlog occurs, the number of consumer instances can be increased on the consumer group management page to improve data consumption speed.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] This invention provides a scalable data writing channel that allows for on-demand configuration of computing resources, improving data throughput and writing efficiency; it supports tenant-level resource isolation configuration; after data is written to disk, a background service monitors and automatically merges small files, and the merging process is seamless for the user. Attached Figure Description

[0029] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0030] Figure 1 This is a diagram illustrating the data writing process.

[0031] Figure 2 Write data to the strategy management page for xIn3Plat structured data services. Detailed Implementation

[0032] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0033] Example 1

[0034] This invention provides a method for managing tenant-level resource isolation data write channels, such as... Figure 1 Users under different tenants write data to the STS Rest cluster proxied by Nginx through the Rest interface. The data is sent to different topics in Kafka for caching according to the user's tenant, and then consumed by the corresponding STS Streaming instance group and written to the Hadoop Distributed File System. The small file merging service is responsible for monitoring and merging small files.

[0035] The specific implementation process includes the following steps:

[0036] 1. Load balancing cluster

[0037] Data is written using a RESTful API. The REST service is deployed in a cluster mode, supporting horizontal scaling, and uses Nginx as a load balancer, providing a unified access address. The cluster mode can effectively improve the throughput of written data and meet the needs of high-frequency data writing.

[0038] 2. Multi-tenant isolation

[0039] Multi-tenant isolation is implemented. Data files are stored on the Hadoop Distributed File System (HDFS). Data files stored on HDFS are queried through the Impala query engine. Access to data tables requires prior authorization on Impala. Different tenants can only see the tables and data within their own tenant's domain. During data writing, write channels can be configured at the tenant level, ensuring that data writes by different tenants do not affect each other. Tenants can adjust the resources used by the channels based on their individual data write volumes, configuring resources as needed.

[0040] 3. Data write channel

[0041] After data is written through the REST gateway, it is sent to the data cache layer (Kafka), then consumed by the Spark Streaming program and written to HDFS. The combination of the data cache layer and the Spark Streaming program is called the data write channel. STS provides a web management page to manage data write channels and configure write policies. The configuration relationship between tenants and data write channels can be one-to-one or many-to-one. When the REST gateway receives a user's request to write data, it queries the corresponding write policy based on the user's tenant and sends the data to the corresponding write channel according to this policy until it is finally written to disk. Channels are resource-isolated, and the entire process is unaffected by each other.

[0042] 4. Consumer groups and consumer instances

[0043] When data is written to the caching layer (Kafka), it is stored in different topics, and then consumed and written to HDFS using Spark Streaming programs. We call a group of Spark Streaming programs a consumer group. Each consumer group is responsible for consuming data from one or more topics. The number of consumer instances in a consumer group can be configured according to the actual amount of data consumed. Within a certain range, the consumption speed can be linearly increased as the number of consumer instances increases.

[0044] A consumer group contains one or more Spark Streaming instances. Each consumer group subscribes to data from one or more topics in Kafka. Spark Streaming instances periodically pull data from the subscribed topics in Kafka and write it to HDFS. If the data write volume is large, the consumer group may not be able to process it in time, resulting in data backlog. Small amounts of data backlog can be left untreated. However, if the backlog is severe, the number of consumer instances can be increased in the consumer group management page to improve data consumption speed.

[0045] 5. File merging service

[0046] When data is written to HDFS by the consumer instance, it is mostly small files. The file merging service is responsible for monitoring the number of files. When the number or size of files reaches a threshold (defined by the configuration file), it automatically triggers the merging of small files.

[0047] Example 2

[0048] Structured data services in the self-developed industrial internet platform xIn3Plat.

[0049] Specific implementation method:

[0050] 1.xIn3Plat structured data service provides a web-based management page for visually configuring write strategies and managing data write channels. See the example image for the web-based management page. Figure 2 .

[0051] 2.xIn3Plat structured data service implements a RESTful interface for writing data. It provides high-performance data writing capabilities through a REST gateway.

[0052] 3.xIn3Plat structured data service uses the Spark Streaming framework to implement the function of writing data from the cache layer (Kafka) to the file storage system (HDFS), which can ensure that the data is stored in the database securely and in a timely manner for users to query.

[0053] 4.xIn3Plat's Merge component of the structured data service can monitor file changes on the HDFS file storage system in real time and automatically merge small files.

[0054] Example 3

[0055] The present invention also provides a tenant-level resource isolation data write channel management system. The tenant-level resource isolation data write channel management system can be implemented by executing the process steps of the tenant-level resource isolation data write channel management method. That is, those skilled in the art can understand the tenant-level resource isolation data write channel management method as a preferred embodiment of the tenant-level resource isolation data write channel management system.

[0056] The tenant-level resource isolation data write channel management system provided by the present invention includes: module M1: deploying a load balancing cluster; module M2: performing multi-tenant isolation; module M3: after data is written through the REST gateway, it is sent to the data cache layer Kafka, and then consumed and written to HDFS by the Spark Streaming program, using the combination of the data cache layer and the Spark Streaming program as the data write channel; module M4: setting up consumer groups and consumer instances to change the data consumption speed; module M5: deploying a file merging service, responsible for monitoring the number and size of files, and automatically triggering small file merging when the number or size of files reaches a preset threshold.

[0057] Data is written using a RESTful API. The REST service is deployed in a cluster mode, supports horizontal scaling, uses Nginx as a load balancer, and provides a unified access address to the outside world.

[0058] Data files are stored on the Hadoop Distributed File System (HDFS). Data files stored on HDFS provide query services to the outside world through the Impala query engine. Data tables are first authorized on Impala before access is granted. Different tenants can only see the tables and data under their own tenants. When writing data, it is possible to configure the write channel at the tenant level. Data writes of different tenants do not affect each other. Tenants adjust the resources used by the channel according to their own data write volume and configure it on demand.

[0059] STS provides a web management page to manage data write channels and configure write policies. The configuration relationship between tenants and data write channels is one-to-one or many-to-one. After the REST gateway receives a user's request to write data, it queries the corresponding write policy according to the user's tenant and sends the data to the corresponding write channel according to the write policy until it is finally written to disk. The channels are isolated from each other and the whole process does not affect each other.

[0060] When data is written to the Kafka caching layer, it is stored in different topics and consumed by Spark Streaming programs and written to HDFS. A group of Spark Streaming programs is called a consumer group. Each consumer group is responsible for consuming data from one or more topics. The number of consumer instances is configured according to the actual amount of data consumed. Within a preset range, the consumption speed increases linearly as the number of consumer instances increases.

[0061] A consumer group contains one or more Spark Streaming instances. Each consumer group subscribes to data from one or more topics in Kafka. Spark Streaming instances periodically pull data from the subscribed topics in Kafka and write it to HDFS. If data backlog occurs, the number of consumer instances can be increased on the consumer group management page to improve data consumption speed.

[0062] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0063] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for managing data write channels with tenant-level resource isolation, characterized in that, include: Step 1: Deploy a load balancing cluster; Step 2: Implement multi-tenant isolation; Step 3: After the data is written through the REST gateway, it is sent to the data cache layer Kafka, and then consumed and written to HDFS by the Spark Streaming program. The combination of the data cache layer and the Spark Streaming program is used as the data writing channel. Step 4: Configure consumer groups and consumer instances to change the data consumption rate; Step 5: Deploy a file merging service to monitor the number and size of files. When the number or size of files reaches a preset threshold, it will automatically trigger the merging of small files. STS provides a web management page to manage data write channels and configure write policies. The configuration relationship between tenants and data write channels can be one-to-one or many-to-one. After receiving a user's request to write data, the REST gateway queries the corresponding write policy based on the user's tenant, and sends the data to the corresponding write channel according to the write policy until it is finally written to disk. The channels are isolated from each other, and the whole process is unaffected by each other.

2. The data write channel management method for tenant-level resource isolation according to claim 1, characterized in that, Data is written using a RESTful API. The REST service is deployed in a cluster mode, supports horizontal scaling, uses Nginx as a load balancer, and provides a unified access address to the outside world.

3. The data write channel management method for tenant-level resource isolation according to claim 1, characterized in that, Data files are stored on the Hadoop Distributed File System (HDFS). Data files stored on HDFS provide query services to the outside world through the Impala query engine. Data tables are first authorized on Impala before access is granted. Different tenants can only see the tables and data under their own tenants. When writing data, it is possible to configure the write channel at the tenant level. Data writes of different tenants do not affect each other. Tenants adjust the resources used by the channel according to their own data write volume and configure it on demand.

4. The data write channel management method for tenant-level resource isolation according to claim 1, characterized in that, When data is written to the Kafka caching layer, it is stored in different topics and consumed by Spark Streaming programs and written to HDFS. A group of Spark Streaming programs is called a consumer group. Each consumer group is responsible for consuming data from one or more topics. The number of consumer instances is configured according to the actual amount of data consumed. Within a preset range, the consumption speed increases linearly as the number of consumer instances increases. A consumer group contains one or more Spark Streaming instances. Each consumer group subscribes to data from one or more topics in Kafka. Spark Streaming instances periodically pull data from the subscribed topics in Kafka and write it to HDFS. If data backlog occurs, the number of consumer instances can be increased on the consumer group management page to improve data consumption speed.

5. A tenant-level resource isolation data write channel management system, characterized in that, include: Module M1: Deploy a load balancing cluster; Module M2: Performs multi-tenant isolation; Module M3: After data is written through the REST gateway, it is sent to the data cache layer Kafka, and then consumed and written to HDFS by the Spark Streaming program. The combination of the data cache layer and the Spark Streaming program is used as the data writing channel. Module M4: Configure consumer groups and consumer instances to change the data consumption rate; Module M5: Deploys the file merging service, responsible for monitoring the number and size of files. When the number or size of files reaches a preset threshold, it automatically triggers the merging of small files. STS provides a web management page to manage data write channels and configure write policies. The configuration relationship between tenants and data write channels can be one-to-one or many-to-one. After receiving a user's request to write data, the REST gateway queries the corresponding write policy based on the user's tenant, and sends the data to the corresponding write channel according to the write policy until it is finally written to disk. The channels are isolated from each other, and the whole process is unaffected by each other.

6. The data writing channel management system with tenant-level resource isolation according to claim 5, characterized in that, Data is written using a RESTful API. The REST service is deployed in a cluster mode, supports horizontal scaling, uses Nginx as a load balancer, and provides a unified access address to the outside world.

7. The data writing channel management system with tenant-level resource isolation according to claim 5, characterized in that, Data files are stored on the Hadoop Distributed File System (HDFS). Data files stored on HDFS provide query services to the outside world through the Impala query engine. Data tables are first authorized on Impala before access is granted. Different tenants can only see the tables and data under their own tenants. When writing data, it is possible to configure the write channel at the tenant level. Data writes of different tenants do not affect each other. Tenants adjust the resources used by the channel according to their own data write volume and configure it on demand.

8. The data writing channel management system with tenant-level resource isolation according to claim 5, characterized in that, When data is written to the Kafka caching layer, it is stored in different topics and consumed by Spark Streaming programs and written to HDFS. A group of Spark Streaming programs is called a consumer group. Each consumer group is responsible for consuming data from one or more topics. The number of consumer instances is configured according to the actual amount of data consumed. Within a preset range, the consumption speed increases linearly as the number of consumer instances increases. A consumer group contains one or more Spark Streaming instances. Each consumer group subscribes to data from one or more topics in Kafka. Spark Streaming instances periodically pull data from the subscribed topics in Kafka and write it to HDFS. If data backlog occurs, the number of consumer instances can be increased on the consumer group management page to improve data consumption speed.

Images