Low-latency high-stability big data component data access method and system

Abstract

The invention discloses a low-delay high-stability big data component data access method and system, and relates to the technical field of big data processing. According to the invention, the authority management module establishes fine mapping between business software and authorized data resources; the data write-in management module sorts write-in requests according to priorities through task arrangement, ensures that high-value data is written in preferentially, and distributes tasks to nodes with light loads through write-in load balancing; the data query management module realizes low-delay query through data unvarnished transmission, stops abnormal or long-time query through slow query management to guarantee system availability, and directly routes a request to a node where data is located through query load balancing; the data life cycle management module regularly marks and cleans out expired redundant data; according to the invention, the problems of resource contention, performance fluctuation and poor stability caused by disordered access to the big data component by multiple service parties are solved, and efficient, stable and controllable access to the big data component is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data processing, and particularly relates to a low-latency high-stability big data component data access method and system. BACKGROUND

[0002] Big data technology has become the core infrastructure for various enterprises to process massive data and support business decisions. Common open-source big data components, such as HDFS, Hive, HBase, ElasticSearch, etc., are widely deployed due to their powerful storage and computing capabilities. In the traditional application mode, the access of business systems to big data components mainly exists in two typical architectures, which gradually expose significant defects in actual large-scale, multi-tenant application scenarios.

[0003] The first is a direct access mode. In this mode, each business software directly connects to and uses the big data component through a network address. Although this method is simple and direct, it has serious security and management vulnerabilities. Since most open-source big data components lack fine-grained permission control mechanisms, once their access endpoints are exposed, all users or programs that can connect to the endpoints can access all the data in them, which can easily lead to sensitive data leakage. In addition, this mode completely relies on the constraints of business software itself, and when multiple business parties concurrently perform large-scale data writing or complex queries without coordination, resource contention is easily triggered, leading to sharp fluctuations in component performance and even service unavailability.

[0004] The second is a unified portal authentication mode. To overcome the security problems of direct access, existing technologies introduce a unified access portal. The portal performs identity authentication and authorization for all users, and only business software that passes the authentication can access the rear big data platform. This mode solves the data exposure risk to some extent, realizes the convergence of access entrances and user management. However, the management granularity of this mode is still rough, and its focus of control is on "who can access the platform", rather than "how to access resources". In actual multi-business-party collaborative scenarios, even if all accessers are authorized, the data access behaviors of each business party are still in a disordered state. For example, low-value, high-throughput data writing tasks may long-term occupy limited writing connections and I / O resources, causing serious delay of high-value data writing requests; some business parties may launch super-large-scale aggregation queries or defective query statements, which may run for a long time and consume a large amount of memory and CPU resources, forming "slow query attacks" and causing timeout of normal query responses of other businesses; at the same time, the lack of effective data lifecycle management mechanisms leads to the continuous accumulation of expired and redundant data, which occupies a large amount of storage space and affects query efficiency. The existing portal mode lacks effective sensing, scheduling and intervention capabilities for such resource contention and abuse behaviors.

[0005] The prior art, especially the mainstream unified portal mode, has obvious deficiencies in ensuring low latency and high stability of big data component services. The core problem lies in the lack of a system layer between business software and big data components that can conduct fine and intelligent access behavior control.

[0006] Therefore, the industry urgently needs a technical solution that can prioritize data writing and load balancing, ensure data query efficiency and exception fusing, and actively manage the entire life cycle of data, to achieve stable, efficient and reliable big data component services in a multi-tenant, high-concurrency environment. SUMMARY

[0007] The purpose of the present application is to overcome the shortcomings of the prior art and provide a low-latency and high-stability big data component data access method and system. The purpose is to comprehensively manage the use of big data components from multiple dimensions such as permissions, writing, querying, and life cycle through a set of system access control strategies, effectively solving the performance instability, high latency, and poor reliability caused by resource contention and abuse in a multi-tenant environment, and ultimately achieving the comprehensive goal of low latency, high stability, and high availability of big data component services.

[0008] To achieve the above-mentioned purpose, the present application provides a low-latency and high-stability big data component data access system, comprising: A permission management module for managing access permissions of multiple business software and establishing a mapping relationship between each business software and authorized data resources; A data writing management module in communication connection with the permission management module, for receiving and processing authorized business software initiated data writing requests for big data components through task scheduling and writing load balancing; A data query management module in communication connection with the permission management module, for receiving and processing authorized business software initiated data query requests for big data components through data transmission, slow query management, and query load balancing; A data life cycle management module for marking and cleaning data stored in the big data component.

[0009] As a further solution, the data writing management module comprises: A task scheduling unit for receiving the data writing request, the data writing request containing a priority parameter; the task scheduling unit arranges high-priority data writing requests at the head of the to-be-written task queue and low-priority data writing requests at the tail of the to-be-written task queue according to the priority parameter, and executes the writing tasks in order.

[0010] As a further solution, the data write management module comprises: A write load balancing unit is configured to periodically acquire load information of a plurality of data nodes in the big data component, and distribute a current data write task to a data node with a relatively small load according to the load information.

[0011] As a further solution, the data query management module comprises: A data transparent transmission unit is configured to forward a data query request of a business software to the big data component, and directly return a query result returned by the big data component to the business software without processing; wherein the data transparent transmission unit limits a data size of a single query request.

[0012] As a further solution, the data query management module comprises: A slow query management unit is configured to monitor an execution state of a data query request, and perform an abortion process on an abnormally executed query, a query causing a deadlock, and a query with an execution time exceeding a preset threshold, and return corresponding abnormal information to a business software initiating the query.

[0013] As a further solution, the data query management module comprises: A query load balancing unit is configured to periodically acquire storage location information of each data resource in different data nodes in the big data component; and when a data query request is received, directly issue the query request to a data node storing a target data resource according to the storage location information.

[0014] As a further solution, the data lifecycle management module is specifically configured to periodically mark a data resource with a long time without access record and a data resource with a storage time exceeding a preset time length, and clean up expired and redundant data resources after confirmation.

[0015] On the other hand, the application further provides a low-latency and high-stability big data component data access method, which is applied to the low-latency and high-stability big data component data access system as described in any one of the above, and the method steps comprise: A permission management module is configured to manage access permissions of a plurality of business softwares, and establish a mapping relationship between each business software and an authorized data resource; A data write management module is configured to receive and process, through task scheduling and write load balancing, a data write request initiated by an authorized business software to the big data component; A data query management module is configured to receive and process, through data transparent transmission, slow query management and query load balancing, a data query request initiated by an authorized business software to the big data component; Mark and clean the data stored in the big data component through the data life cycle management module.

[0016] Compared with the related art, the low-latency high-stability big data component data access method and system provided by the application has the following advantages: 1. The application realizes more fine-grained permission control than the traditional portal authentication mode by establishing the precise mapping relationship between the business software and the specific data resources (such as libraries, tables, and paths). This not only effectively prevents unauthorized access and data leakage risks, but also lays the foundation for subsequent differentiated services based on data value.

[0017] 2. The application unifies the scheduling of write requests according to the data value priority through the task scheduling function, ensuring that high-value data can be written first and quickly. At the same time, this mechanism smooths the write traffic and avoids resource contention caused by instantaneous peak write of multiple business parties, thereby maintaining the stability of the big data component write performance and solving the business pain point of delayed write of high-value data due to queuing.

[0018] 3. The write load balancing function of the application balances the load of each node in the big data component by intelligently distributing write tasks based on real-time sensing of the load state of each data node. This avoids the performance bottleneck caused by the overload of a single node, improves the resource utilization and processing capacity of the entire cluster, and enhances the overall stability and service reliability of the system.

[0019] 4. The data transmission mechanism of the application avoids the additional processing of query data by the proxy layer, maximally reducing the query delay. Combined with the query load balancing function, the query request is directly routed to the data node, reducing the request forwarding overhead between nodes and further improving the query response speed.

[0020] 5. The slow query management function of the application can actively monitor, identify and terminate abnormal queries, deadlocked queries and long-running queries, effectively preventing the depletion of system resources caused by "brute force queries" or errors of individual businesses, which may cause damage to the overall service. This is equivalent to providing "fusing" protection for the big data component, significantly improving the robustness of the system and the high availability protection of other normal businesses.

[0021] 6. The data life cycle management module of the application can periodically and automatically identify and mark "cold data" that is expired, redundant and long-term unvisited, and support administrators for cleaning. This helps to continuously release valuable storage space, reduce storage costs, and avoid the drag of invalid data on query performance, so that the big data component can maintain a healthy and efficient running state for a long time. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the application and, together with the description, further serve to explain the principles of the application.

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the application or the related art, the accompanying drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0024] Figure 1 A low-latency high-stability big data component data access method step schematic diagram provided by the application; Figure 2 A service software and data resource mapping relationship diagram provided by the application; Figure 3 A task arrangement in which a new write task arrangement logic diagram is added provided by the application; Figure 4 A write load balancing logic schematic diagram provided by the application; Figure 5 A data transparent transmission logic schematic diagram provided by the application; Figure 6 A slow query management schematic diagram provided by the application; Figure 7 A data query load balancing logic schematic diagram provided by the application; Figure 8 A data life cycle management schematic diagram provided by the application; Figure 9 A low-latency high-stability big data component data access method step schematic diagram provided by the application; The purpose of the application, the functional characteristics and the advantages will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the embodiments of the application clearer, the technical solutions in the embodiments of the application will be described clearly and completely with reference to the accompanying drawings of the embodiments of the application. Obviously, the described embodiments are some embodiments of the application, not all embodiments of the application. The components of the embodiments of the application described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0026] Please refer to Figure 1 The embodiment provides a low-latency high-stability big data component data access system, which comprises: The permission management module is configured to manage access permissions of the plurality of business software and establish a mapping relationship between each business software and an authorized data resource. The data write management module is in communication connection with the permission management module and is configured to receive and process, through task scheduling and write load balancing, a data write request initiated by the authorized business software to the big data component. The data query management module is in communication connection with the permission management module and is configured to receive and process, through data transmission, slow query management and query load balancing, a data query request initiated by the authorized business software to the big data component. The data lifecycle management module is configured to mark and clean up data stored in the big data component.

[0027] It should be noted that the logical architecture of the embodiment is as follows Figure 1 As shown in the figure, the business software of each different manufacturer performs data write and data query operations on the components in the big data platform through the system software agent. The system software provides permission authentication, data write management, data query management and data lifecycle management modules. The permission management module is responsible for accessing permission information and establishing a corresponding relationship between the business software and the data resource. Different business software can access different data resources. The data write management module manages the write behavior of all business software, including write task scheduling and write task load balancing. The data query management module manages the data query behavior of the business software, including data transmission, slow query management and query load balancing. The data lifecycle management is mainly responsible for cleaning up expired data to prevent expired data from occupying too much disk space resources and affecting the write and query efficiency of the big data component.

[0028] Further, the permission management module manages the access permissions of the business software. The mapping relationship between the business software and the data resource is as shown in the figure Figure 2 As shown in the figure, the business software of each manufacturer is registered through a visual interface and a mapping relationship between the business software and the specified data resource is established. That is, the business software can only use the authorized data resource and cannot access the unauthorized data resource. The permission authentication module monitors the data access behavior of the business software to prevent unauthorized access.

[0029] Further, the data write management module includes: The task scheduling unit is configured to receive the data write request, wherein the data write request includes a priority parameter. The task scheduling unit arranges the data write request with high priority at the head of the write task queue and arranges the data write request with low priority at the tail of the write task queue according to the priority parameter, and executes the write task in sequence.

[0030] Specifically, the processing logic of the task arrangement function in the data writing management module is as follows Figure 3 As shown in the figure. The business software issues data writing tasks to the system through the system data writing interface. Different big data components have different data writing interfaces. Each data writing interface has a corresponding big data component waiting for writing task queue. There is a priority parameter in the data writing interface. The higher the priority, the higher the data value. Conversely, the lower the priority, the lower the data value. The default writing data priority is the lowest. When a new writing task is a low-value data writing task, the writing task is placed at the end of the waiting writing task queue. When a new writing task is a high-value data writing task, the writing task is placed at the beginning of the waiting writing task queue, and data writing is performed in priority.

[0031] The waiting writing task queue performs data writing in turn according to the task order. This data writing method can ensure relatively stable data writing of each big data component and prevent resource contention from reducing data writing efficiency. At the same time, the priority mechanism can ensure that high-value data is written in priority to meet the actual business needs of the business system. The task arrangement function not only arranges writing tasks, but also uniformly arranges data update tasks and data deletion tasks Further, the data writing management module comprises: The writing load balancing unit is configured to periodically acquire load information of a plurality of data nodes in the big data component, and distribute the current data writing task to a data node with relatively small load according to the load information.

[0032] Specifically, the writing load balancing logic in the data writing management module is as follows Figure 4 As shown in the figure, the data writing management module periodically acquires load information of each node of the big data component, and distributes the current data writing task to a data node with relatively small load according to the load information of each node. The load information of each node is balanced as much as possible. The load balancing of each node can effectively ensure that the query and writing efficiency of each node is basically consistent, which can greatly improve the stability of the use of the big data component. Preventing a single node from having too high load, which can cause the performance of the entire big data component to drop sharply.

[0033] Further, the data query management module comprises: The data transparent transmission unit is configured to forward the data query request of the business software to the big data component, and return the query result returned by the big data component to the business software without processing. Wherein, the data transparent transmission unit limits the data size of a single query request.

[0034] Specifically, the data transparent transmission logic in the data query management module is as follows Figure 5As shown, the business software initiates a big data component query request through the data query management module, and the data query management module returns the queried data directly to the business software without processing. The data transmission function limits the data size of single data access, and the single query cannot exceed 500,000 data. Through the data transmission mode, the low latency of data query response is ensured.

[0035] Further, the data query management module comprises: The slow query management unit is configured to monitor the execution state of the data query request, and to perform abort processing on the query with execution exception, the query causing deadlock, and the query with execution time exceeding the preset threshold, and to return the corresponding exception information to the business software initiating the query.

[0036] Specifically, as shown in Figure 6 The slow query management function in the data query management module is mainly responsible for processing three types of transactions, which are the processing of query exception, the processing of deadlock query, and the processing of long query time.

[0037] The processing of query exception is to encapsulate the key information in the error information and return it to the business software. The business software can determine whether the query logic is incorrect by using the error information.

[0038] The processing of deadlock query is to encapsulate the prompt information that multiple business softwares jointly occupy data resources to cause possible deadlock and return it to the business software when the business software occupies the data resources of a table and other business software accesses the same resources of the table.

[0039] The processing of long query time is to abort the corresponding query task and return the information of long query time to the business software for the query statement with query time exceeding ten minutes and unable to respond.

[0040] Further, the data query management module comprises: The query load balancing unit is configured to periodically acquire the storage location information of each data resource in different data nodes of the big data component, and to directly issue the query request to the data node storing the target data resource according to the storage location information when receiving the data query request.

[0041] Specifically, the load balancing function of the data query management module has the following processing logic Figure 7As shown, the data query management module periodically retrieves basic storage information for each database table across different nodes in the big data component. When business software issues a data query through the data query management module, the business software forwards the query request to the data node containing that data. Although sending the data query request to all nodes will yield the query result, data nodes without the data need to forward the request to other nodes to obtain the result, which reduces query efficiency. By sending the query request directly to the data node containing the data, the number of requests between data nodes is reduced, indirectly improving data query efficiency.

[0042] Furthermore, the data lifecycle management module is specifically used to: periodically mark data resources that have not been accessed for a long time and data resources that have been stored for a longer period of time, and clean up expired and redundant data resources after confirmation.

[0043] Specifically, such as Figure 8 As shown, the data lifecycle management module mainly has three functions: periodically marking data that has not been accessed by users for a long time, marking data that has been stored for an excessively long time, and cleaning up expired and redundant data. The data lifecycle management module periodically marks data resources and prompts the super administrator user with expired and redundant data information through a visual interface. After confirmation by the super administrator user, the expired and redundant data is cleaned up.

[0044] Please see Figure 9 The present invention also provides a low-latency, high-stability data access method for big data components, applicable to the low-latency, high-stability data access system for big data components as described in any of the preceding claims, wherein the method includes the following steps: The access control module manages access permissions for multiple business software applications and establishes a mapping relationship between each business software application and authorized data resources. The data write management module receives and processes data write requests for big data components initiated by authorized business software through task orchestration and write load balancing. The data query management module receives and processes data query requests for big data components initiated by authorized business software through data pass-through, slow query management, and query load balancing. The data lifecycle management module is used to mark and clean the data stored in the big data component.

[0045] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A low-latency, high-stability big data component data access system, characterized in that, include: The access control module is used to manage access permissions for multiple business software applications and establish a mapping relationship between each business software application and authorized data resources. The data writing management module is communicatively connected to the permission management module and is used to receive and process data writing requests for big data components initiated by authorized business software through task orchestration and write load balancing. The data query management module is communicatively connected to the permission management module and is used to receive and process data query requests for big data components initiated by authorized business software through data pass-through, slow query management, and query load balancing. The data lifecycle management module is used to mark and clean up the data stored in the big data component.

2. The low-latency, high-stability big data component data access system according to claim 1, characterized in that, The data writing management module includes: A task orchestration unit is used to receive the data write request, the data write request including a priority parameter; the task orchestration unit arranges high-priority data write requests at the head of the queue of tasks to be written and low-priority data write requests at the tail of the queue of tasks to be written according to the priority parameter, and executes the write tasks in sequence.

3. The low-latency, high-stability big data component data access system according to claim 1, characterized in that, The data writing management module includes: A write load balancing unit is used to periodically obtain the load information of multiple data nodes in the big data component, and distribute the current data write task to the data nodes with relatively low load based on the load information.

4. The low-latency, high-stability big data component data access system according to claim 1, characterized in that, The data query management module includes: The data pass-through unit is used to forward data query requests from the business software to the big data component, and to return the query results returned by the big data component directly to the business software without processing; wherein, the data pass-through unit limits the data size of a single query request.

5. The low-latency, high-stability big data component data access system according to claim 1, characterized in that, The data query management module includes: The slow query management unit monitors the execution status of data query requests and terminates queries that are abnormal, cause deadlocks, or have execution times exceeding a preset threshold, and returns the corresponding exception information to the business software that initiated the query.

6. The low-latency, high-stability big data component data access system according to claim 1, characterized in that, The data query management module includes: The load balancing unit is used to periodically obtain the storage location information of each data resource in different data nodes in the big data component; when a data query request is received, the query request is directly sent to the data node storing the target data resource according to the storage location information.

7. The low-latency, high-stability big data component data access system according to claim 1, characterized in that, The data lifecycle management module is specifically used to: periodically mark data resources that have not been accessed for a long time and data resources that have been stored for a longer period of time, and clean up expired and redundant data resources after confirmation.

8. A low-latency, high-stability data access method for big data components, applied in the low-latency, high-stability data access system for big data components as described in any one of claims 1 to 7, characterized in that, The method steps include: The access control module manages access permissions for multiple business software applications and establishes a mapping relationship between each business software application and authorized data resources. The data write management module receives and processes data write requests for big data components initiated by authorized business software through task orchestration and write load balancing. The data query management module receives and processes data query requests for big data components initiated by authorized business software through data pass-through, slow query management, and query load balancing. The data lifecycle management module is used to mark and clean the data stored in the big data component.

Images