Cross-ecological cluster big data management system and method

Abstract

The invention relates to the technical field of big data, and provides a cross-ecological cluster big data management system and method. The system comprises a service unit, an adaptation unit and a driving unit, the service unit is used for receiving a user request and sending the user request to the adaptation unit; the adaptation unit is used for determining a corresponding cluster identifier and a drive type according to a user request, obtaining a drive instance port corresponding to the cluster identifier and the drive type from a preset routing table through matching, and carrying out routing addressing according to the drive instance port so as to forward the user request to a target drive instance in the drive unit; and the driving unit pre-establishes a connection relationship with the plurality of clusters by using a preset driving configuration mechanism, and is used for obtaining a cluster port corresponding to the user request from the target driving instance in a matching manner, performing addressing according to the cluster port, and forwarding the user request from the target driving instance to the target cluster. According to the embodiment of the invention, cross-ecological high compatibility and diversified cluster efficient management can be realized.

Description

Technical Field

[0001] This specification relates to the field of big data technology, and in particular to a cross-ecological cluster big data management system and method. Background Art

[0002] With the development of big data technology, big data products with diversified ecosystems and technology stacks have emerged. When the amount of data is small and the business scenarios are simple, a single cluster is often used to support big data application scenarios. However, for the financial industry and large Internet companies, which have large data volumes and complex internal business scenarios, it is often necessary to build multiple clusters and use multi-ecology collaboration to support big data scenario requirements. For cluster big data management issues, existing methods include single cluster management, same-ecology cluster management, and cross-ecology cluster resource monitoring.

[0003] For single cluster management, one management tool is responsible for only one cluster. This solution can meet the management, operation, maintenance, and usage functions of a single cluster and is often used in native components and early versions of cluster management tools.

[0004] For the management of clusters in the same ecosystem, the component structures of clusters in the same ecosystem are basically the same, and the cluster access methods and data specifications are also consistent, which is often used for the unified management of multiple clusters in the same ecosystem.

[0005] For cross-ecological cluster resource monitoring, this solution provides unified monitoring and operation and maintenance for multi-cluster spaces, and can perform spatial operations on multiple clusters at the same time to meet the unified management of multi-cluster resources.

[0006] However, a single cluster management method can only meet the management of a single cluster, but cannot meet the unified management of multiple clusters; the same-ecological cluster management solution can only meet the cluster management of the same ecology or the same product type. For example, a product can support the operation and maintenance management of multiple clusters corresponding to the product, but cannot support the management of clusters in other ecology; the cross-ecological cluster resource monitoring solution currently only provides monitoring and early warning, and cannot meet the user's diversified management needs for cluster resource allocation, tenant management, and online operation and maintenance. This single-function technical solution cannot achieve truly unified and diversified management of cross-ecological clusters. Therefore, there is an urgent need for a cross-ecological cluster big data management system with high cross-ecological compatibility to achieve all-round and diversified cluster efficient management. Summary of the invention

[0007] In view of the present situation, the present solution is proposed to overcome the above problems or at least partially solve the above problems.

[0008] On the one hand, the purpose of some embodiments of this specification is to provide a cross-ecological cluster big data management system, the system including a service unit, an adaptation unit and a driving unit;

[0009] Wherein, the service unit is used to receive a user request and send the user request to the adaptation unit;

[0010] The adapter unit is used to determine the corresponding cluster identifier and driver type according to the user request, match the driver instance port corresponding to the cluster identifier and driver type from a preset routing table, and perform routing addressing according to the driver instance port, so as to forward the user request to the target driver instance in the driver unit;

[0011] The driver unit pre-establishes a connection relationship with a plurality of clusters using a preset driver configuration mechanism, and is used to match a cluster port corresponding to the user request from the target driver instance, and performs addressing according to the cluster port, and forwards the user request from the target driver instance to the target cluster;

[0012] The preset driver configuration mechanism includes separate deployment of driver instances, driver programs and clusters.

[0013] Furthermore, the preset drive configuration mechanism also includes:

[0014] According to the driving scenario requirements, the association relationship between the driving instance, the driver program and the cluster is configured so that one driver program corresponds to at least one driving instance, and one driver instance corresponds to at least one cluster, wherein the driver program corresponds to the big data software product one-to-one.

[0015] Furthermore, the driving unit further includes:

[0016] The driver fuse module is used to interrupt the access service when the failure ratio of user requests reaches a preset threshold;

[0017] The driver heartbeat monitoring module is used to monitor abnormal conditions of driver instances.

[0018] Furthermore, it also includes:

[0019] The cache unit is used to cache responses corresponding to historical user requests and reuse the responses for user requests that meet preset response requirements.

[0020] Furthermore, it also includes:

[0021] The data processing unit is used to process the user request according to the preset data processing requirements corresponding to the user request during the process of forwarding the user request, and return the data processing result to the service unit or the adapter unit.

[0022] Furthermore, the data processing unit also includes:

[0023] The screening module is used to screen the user requests to obtain user requests of a preset type, so that the data processing unit performs data processing on the user requests of the preset type.

[0024] On the other hand, some embodiments of this specification also provide a cross-ecological cluster big data management method, the method comprising:

[0025] The service unit receives the user request and sends the user request to the adaptation unit;

[0026] The adapter unit determines the corresponding cluster identifier and driver type according to the user request, matches the driver instance port corresponding to the cluster identifier and driver type from a preset routing table, and performs routing addressing according to the driver instance port, thereby forwarding the user request to the target driver instance in the driver unit;

[0027] The driver unit pre-establishes a connection relationship with a plurality of clusters using a preset driver configuration mechanism, and is used to match a cluster port corresponding to the user request from the target driver instance, and performs addressing according to the cluster port, and forwards the user request from the target driver instance to the target cluster;

[0028] The preset driver configuration mechanism includes separate deployment of driver instances, driver programs and clusters.

[0029] On the other hand, some embodiments of the present specification further provide a computer device, including a memory, a processor, and a computer program stored in the memory, wherein when the computer program is executed by the processor, the instructions of the above method are executed.

[0030] On the other hand, some embodiments of the present specification further provide a computer storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor of a computer device, the instructions of the above method are executed.

[0031] On the other hand, some embodiments of the present specification further provide a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor of a computer device, the instructions of the above method are executed.

[0032] One or more technical solutions provided by some embodiments of this specification have at least the following technical effects:

[0033] The service unit in the embodiment of the present specification automatically receives the user request, forwards the request according to the user request, and sends the user request to the adapter unit. Afterwards, since the driver instance, driver program and cluster are deployed separately, the adapter unit needs to determine the cluster identifier and driver type corresponding to the user request, match the driver instance port corresponding to the cluster identifier and driver type from the preset routing table, perform routing addressing according to the driver instance port, and forward the user request to the target driver instance in the driver unit. On the other hand, the driver unit establishes corresponding connection relationships with multiple clusters in advance through a preset driver configuration mechanism. After the user request reaches the target driver instance, the cluster port corresponding to the user request is matched from the target driver instance, and addressing is performed according to the cluster port, and the user request is forwarded from the target driver instance to the target cluster, thereby decoupling the cross-ecological user requests in a hierarchical manner and forwarding them to the target cluster efficiently and accurately, thereby achieving high compatibility, all-round and diversified cluster efficient management across the ecology.

[0034] The above description is only an overview of the technical solutions of some embodiments of this specification. In order to more clearly understand the technical means of some embodiments of this specification, they can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of some embodiments of this specification more obvious and easy to understand, the specific implementation methods of some embodiments of this specification are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate some embodiments of this specification or technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or prior art descriptions. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. In the drawings:

[0036] Figure 1 This is a schematic diagram of the structure of a cross-ecological cluster big data management system in some embodiments of this specification;

[0037] Figure 2 A schematic diagram of the association relationship between a driver program, a driver instance, and a cluster in some embodiments of this specification;

[0038] Figure 3 A schematic diagram of routing and forwarding in some embodiments of this specification;

[0039] Figure 4 This is a schematic diagram of the structure of a driving unit in some embodiments of this specification;

[0040] Figure 5A schematic diagram of the connection relationship of the cache units in some embodiments of this specification;

[0041] Figure 6 A schematic diagram of the connection relationship of data processing units in some embodiments of this specification;

[0042] Figure 7 This is a schematic diagram of the structure of a data processing unit in some embodiments of this specification;

[0043] Figure 8 A flowchart of a big data management method across ecological clusters in some embodiments of this specification is shown;

[0044] Fig. 9 This is a schematic diagram of the computer device structure provided in some embodiments of this specification.

[0045] [Description of Reference Numerals]

[0046] 101. Service unit;

[0047] 102. Adaptation unit;

[0048] 103. Drive unit;

[0049] 1031, drive fuse module;

[0050] 1032. Drive the heartbeat monitoring module;

[0051] 104. Cache unit;

[0052] 105. Data processing unit;

[0053] 1051. Screening module;

[0054] 902. Computer equipment;

[0055] 904, processor;

[0056] 906. Memory;

[0057] 908, driving mechanism;

[0058] 910, input / output interface;

[0059] 912. Input device;

[0060] 914. Output device;

[0061] 916. Presentation equipment;

[0062] 918. Graphical user interface;

[0063] 920, network interface;

[0064] 922, communication link;

[0065] 924. Communication bus. DETAILED DESCRIPTION

[0066] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in some embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on some embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0067] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0068] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of relevant laws and regulations.

[0069] It should be noted that in the embodiments of this specification, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0070] like Figure 1 The following is a cross-ecological cluster big data management system provided by an embodiment of this specification. Figure 1The system may include a service unit 101, an adapter unit 102, a driver unit 103 and a target cluster. The service unit 101 is used to receive a user request and send the user request to the adapter unit 102; the adapter unit 102 is used to determine the corresponding cluster identifier and driver type according to the user request, match the driver instance port corresponding to the cluster identifier and driver type from the preset routing table, and perform routing addressing according to the driver instance port, so as to forward the user request to the target driver instance in the driver unit 103; the driver unit 103 uses a preset driver configuration mechanism to pre-establish a connection relationship with multiple clusters, and is used to match the cluster port corresponding to the user request from the target driver instance, and perform addressing according to the cluster port, so as to forward the user request from the target driver instance to the target cluster; the preset driver configuration mechanism includes separate deployment of the driver instance, the driver program and the cluster.

[0071] The service unit 101 in the embodiment of the present specification automatically receives user requests, which can be used to implement operation and maintenance, application services, monitoring warnings, scheduled tasks, etc., and forwards the request according to the user request, and sends the user request to the adapter unit 102. After that, since the driver instance, driver program and cluster are deployed separately, the adapter unit 102 needs to determine the cluster identifier and driver type corresponding to the user request, match the driver instance port corresponding to the cluster identifier and driver type from the preset routing table, and perform routing addressing according to the driver instance port, and forward the user request to the target driver instance in the driver unit 103. On the other hand, the driver unit 103 establishes corresponding connection relationships with multiple clusters in advance through a preset driver configuration mechanism. After the user request reaches the target driver instance, the cluster port corresponding to the user request is matched from the target driver instance, and the addressing is performed according to the cluster port, and the user request is forwarded from the target driver instance to the target cluster, thereby decoupling the cross-ecological user request in a hierarchical manner, and efficiently and accurately forwarding it to the target cluster, so as to achieve high compatibility, all-round and diversified cluster efficient management across the ecology.

[0072] Furthermore, in some embodiments, the preset drive configuration mechanism also includes:

[0073] According to the driving scenario requirements, the association relationship between the driving instance, the driver program and the cluster is configured so that one driver program corresponds to at least one driving instance, and one driver instance corresponds to at least one cluster, wherein the driver program corresponds to the big data software product one-to-one.

[0074] It can be understood that in some embodiments, big data software products are diverse, and their component composition, interaction mode, management mode, identity authentication, etc. are different, and the application scenarios are complex. In order to realize big data management across ecological clusters, it is necessary to pre-set the driver configuration mechanism. Figure 2 The schematic diagram of the relationship between the driver, driver instance and cluster shown in FIG. 1 shows a driver corresponding to a big data software product. Each driver can create one or more driver instances according to the driving scenario requirements. For example, Figure 2 In the example, for big data software product A, it can correspond to two driver instances, namely, product A driver instance 1 and product A driver instance 2. For big data software product B, it can correspond to two driver instances, namely, product B driver instance 1 and product B driver instance 2. For big data software product C, it corresponds to one driver instance, namely, product C driver instance 1. Each driver instance is connected to and accesses a single cluster. For example, in the attached Figure 2 In the example, product A driver instance 1 and product A driver instance 2 connect to and access product A cluster, product B driver instance 1 and product B driver instance 2 connect to and access product B cluster, and product C driver instance 1 connects to and accesses product C cluster. By deploying drivers, driver instances and clusters separately in different layers, it is convenient to quickly iterate the driver program. Each independent driver instance can use service registration and routing management services to achieve driver load balancing and high service availability, avoiding the problem of completing multiple cross-ecological cluster interactions in one driver instance, resulting in the program being too large and complex, and the problem that software package dependency conflicts may not be well resolved.

[0075] Further, in some embodiments, referring to the attached Figure 3 In the routing forwarding diagram shown, cluster information includes cluster identification (i.e., cluster ID) and cluster type, etc. When a user initiates a user request, the service unit 101 analyzes and feeds back the user request, and forwards the user request that needs to access the cluster to the adapter unit 102. The adapter unit 102 determines the corresponding cluster identification and driver type according to the user request, queries the corresponding cluster information and driver port from the preset routing table, matches the driver instance port corresponding to the cluster identification and driver type, and performs routing addressing according to the driver instance port, and forwards the user request to the target driver instance in the driver unit 103. The driver unit 103 determines the correspondence between the target driver instance and the cluster port according to the cluster information corresponding to the user request, so as to connect to the target cluster, obtain the data returned by the target cluster, and package the returned data into the style required by the unified interface, and returns it to the adapter unit 102, forwards it back to the service unit 101, and finally returns the user result through the service unit 101. Compared with traditional routing management, adding driver instance addressing and driver instance management can achieve accurate cluster matching, complete request forwarding and result feedback.

[0076] Refer to the attached Figure 4 The schematic diagram of the structure of the driving unit 103 is shown in FIG. 1 . In some embodiments, the driving unit 103 further includes:

[0077] A driver fuse module 1031 is used to interrupt access services when the failure ratio of user requests reaches a preset threshold;

[0078] The driver heartbeat monitoring module 1032 is used to monitor abnormal conditions of the driver instance.

[0079] It can be understood that in some embodiments, the driver unit 103 is a necessary condition for the user request to reach the cluster. In order to avoid the cluster load being too high or the driver continuously accessing when it is abnormal, the driver unit 103 also includes a driver fuse module 1031. By introducing a driver fuse mechanism, the cluster is prevented from being more seriously affected by driver access. At the same time, the driver unit 103 also includes a driver heartbeat monitoring module 1032, which provides driver heartbeat monitoring to grasp the driver instance in real time. When the driver instance is abnormal, the service is automatically pulled up to ensure high service availability. Further, in some embodiments, in order to meet the access requirements of multiple clusters and ensure the isolation of each cluster request, the method of multiple driver instances docking the cluster is adopted, which not only reduces the impact of inter-cluster requests, but also facilitates the independent start, stop and upgrade of each driver instance and driver program. Further, in order to ensure the continuous service of the driver instance, the driver is restarted and upgraded in a hot deployment manner. When the driver service is restarted, a new driver instance will be automatically enabled before the original driver instance is closed. After the original driver instance is restarted, the temporary driver instance will also be closed.

[0080] Refer to the attached Figure 5 The connection relationship diagram of the cache unit 104 shown in the figure may also include, in some embodiments:

[0081] The cache unit 104 is used to cache responses corresponding to historical user requests and reuse the responses for user requests that meet preset response requirements.

[0082] It can be understood that, in some embodiments, when a user request arrives at the service unit 101, since the cache unit 104 stores responses corresponding to historical user requests, when a new user request is consistent with the historical user request, that is, the new user request is a user request that meets the preset response requirements, then there is no need to forward the new user request to the adaptation unit 102, and the response corresponding to the new user request can be obtained from the cache unit 104, thereby quickly returning the response to the user.

[0083] Refer to the attached Figure 6 The connection relationship diagram of the data processing unit 105 shown in the figure, in some embodiments, further includes:

[0084] The data processing unit 105 is used to process the user request according to the preset data processing requirements corresponding to the user request during the process of forwarding the user request, and return the data processing result to the service unit 101 or the adaptation unit 102.

[0085] It can be understood that, in some embodiments, the service unit 101 and the adapter unit 102 can be used to process data for user requests. However, if the service unit 101 and the adapter unit 102 bear too heavy data processing tasks, it will affect the normal forwarding and processing of user requests. In addition, since each cluster has multiple components, the operation data involved will increase with the cluster load and usage frequency, and the more operation data, the longer the processing time and the more resources are consumed. Therefore, when multiple clusters are managed in a unified manner, the processing timeliness of the operation data often becomes the core issue of multi-cluster management. Therefore, an independent data processing unit 105 is specially set up to support resource isolation and independent deployment, and to undertake the data processing work of user requests of specified preset types, especially user requests with complex data processing processes and high processing costs, so as to decouple the data processing work of user requests according to the preset data processing requirements corresponding to the user requests, efficiently process the user requests, and quickly return the data processing results to the service unit 101 or the adapter unit 102, so that the service unit 101 or the adapter unit 102 forwards the processed user requests to the next unit for subsequent forwarding operations.

[0086] Further, in some embodiments, referring to the attached Figure 7 The data processing unit 105 is a schematic diagram of the structure of the data processing unit 105, and the data processing unit 105 also includes:

[0087] The screening module 1051 is used to screen the user requests to obtain user requests of a preset type, so that the data processing unit 105 performs data processing on the user requests of the preset type.

[0088] It can be understood that, in some embodiments, since not all user requests need to call the data processing unit 105 to complete, the filtering module 1051 can be used to filter the user requests to obtain user requests of a preset type. The preset user request type is a request with a complex data processing process and a high processing cost (for example, the time consumed exceeds a preset threshold), so that the data processing unit 105 processes the user requests of the preset type. For other user requests other than the preset user requests, if there is still a data processing requirement, it can be handed over to the adaptation unit 102 or the service unit 101 for processing, thereby providing unified data processing and conversion services for each level to achieve efficient data processing.

[0089] It should be noted that, although the operations of the method of the present invention are described in a specific order in the above embodiments and the accompanying drawings, this does not require or imply that the operations must be performed in the specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0090] Corresponding to the above-mentioned cross-ecological cluster big data management system, some embodiments of this specification also provide a cross-ecological cluster big data management method. Figure 8 It is a flowchart of a cross-ecological cluster big data management method provided by an embodiment of the present invention. This specification provides method operation steps as described in the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only order of execution. When the actual system or device product is executed, it can be executed in the order or in parallel according to the method shown in the embodiment or the accompanying drawings. Specifically, Figure 8 As shown, applied to the above-mentioned server side, the method may include:

[0091] S801: The service unit receives a user request and sends the user request to the adaptation unit;

[0092] S802: The adapter unit determines the corresponding cluster identifier and driver type according to the user request, matches the driver instance port corresponding to the cluster identifier and driver type from a preset routing table, and performs routing addressing according to the driver instance port, thereby forwarding the user request to the target driver instance in the driver unit;

[0093] S803: The driver unit pre-establishes connection relationships with multiple clusters using a preset driver configuration mechanism, so as to match a cluster port corresponding to the user request from the target driver instance, and performs addressing according to the cluster port to forward the user request from the target driver instance to the target cluster;

[0094] The preset driver configuration mechanism includes separate deployment of driver instances, driver programs and clusters.

[0095] The service unit in the embodiment of the present specification automatically receives the user request, forwards the request according to the user request, and sends the user request to the adapter unit. Afterwards, since the driver instance, driver program and cluster are deployed separately, the adapter unit needs to determine the cluster identifier and driver type corresponding to the user request, match the driver instance port corresponding to the cluster identifier and driver type from the preset routing table, perform routing addressing according to the driver instance port, and forward the user request to the target driver instance in the driver unit. On the other hand, the driver unit establishes corresponding connection relationships with multiple clusters in advance through a preset driver configuration mechanism. After the user request reaches the target driver instance, the cluster port corresponding to the user request is matched from the target driver instance, and addressing is performed according to the cluster port, and the user request is forwarded from the target driver instance to the target cluster, thereby decoupling the cross-ecological user requests in a hierarchical manner and forwarding them to the target cluster efficiently and accurately, thereby achieving high compatibility, all-round and diversified cluster efficient management across the ecology.

[0096] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0097] It should be noted that in the embodiments of this specification, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user and fully authorized by all parties.

[0098] It should be noted that the computer program product described in this specification is a software product that mainly implements the method described in this specification through a computer program.

[0099] The embodiments of this specification also provide a computer device. Fig. 9As shown, in some embodiments of the present specification, the computer device 902 may include one or more processors 904, such as one or more central processing units (CPUs) or graphics processing units (GPUs), and each processing unit may implement one or more hardware threads. The computer device 902 may also include any memory 906, which is used to store any kind of information such as code, settings, data, etc. In a specific embodiment, the computer program on the memory 906 and can be run on the processor 904, when the computer program is run by the processor 904, the instructions of the method described in any of the above embodiments may be executed. Non-limiting, for example, the memory 906 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory may use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the computer device 902. In one case, when the processor 904 executes an associated instruction stored in any memory or a combination of memories, the computer device 902 may perform any operation of the associated instruction. The computer device 902 also includes one or more drive mechanisms 908 for interacting with any storage, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0100] The computer device 902 may also include an input / output interface 910 (I / O) for receiving various inputs (via input devices 912) and for providing various outputs (via output devices 914). A specific output mechanism may include a presentation device 916 and an associated graphical user interface 918 (GUI). In other embodiments, the input / output interface 910 (I / O), input device 912, and output device 914 may not be included, and the computer device 902 may be used as a computer device in a network. The computer device 902 may also include one or more network interfaces 920 for exchanging data with other devices via one or more communication links 922. One or more communication buses 924 couple the components described above together.

[0101] The communication link 922 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 922 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.

[0102] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), computer-readable storage media, and computer program products of some embodiments of the present specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processor to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processor generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0103] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processor to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processor so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0105] In a typical configuration, a computer device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0106] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0107] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computer device. As defined in this specification, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0108] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0109] The present specification embodiments may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present specification embodiments may also be practiced in distributed computing environments where tasks are performed by remote processors connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0110] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0111] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0112] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiments of this specification. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0113] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A cross-ecological cluster big data management system, characterized in that: The system comprises a service unit, an adaption unit and a drive unit; Wherein, the service unit is used to receive a user request and send the user request to the adaptation unit; The adapter unit is used to determine the corresponding cluster identifier and driver type according to the user request, match the driver instance port corresponding to the cluster identifier and driver type from a preset routing table, and perform routing addressing according to the driver instance port, so as to forward the user request to the target driver instance in the driver unit; The driver unit pre-establishes a connection relationship with a plurality of clusters using a preset driver configuration mechanism, and is used to match a cluster port corresponding to the user request from the target driver instance, and performs addressing according to the cluster port, and forwards the user request from the target driver instance to the target cluster; The preset driver configuration mechanism includes separate deployment of driver instances, driver programs and clusters.

2. The cross-ecological cluster big data management system according to claim 1 is characterized in that: The preset drive configuration mechanism also includes: According to the driving scenario requirements, the association relationship between the driving instance, the driver program and the cluster is configured so that one driver program corresponds to at least one driving instance, and one driver instance corresponds to at least one cluster, wherein the driver program corresponds to the big data software product one-to-one.

3. The cross-ecological cluster big data management system according to claim 2 is characterized in that: The driving unit further comprises: The driver fuse module is used to interrupt the access service when the failure ratio of user requests reaches a preset threshold; The driver heartbeat monitoring module is used to monitor abnormal conditions of driver instances.

4. The cross-ecological cluster big data management system according to claim 1, characterized in that: Also includes: The cache unit is used to cache responses corresponding to historical user requests and reuse the responses for user requests that meet preset response requirements.

5. The cross-ecological cluster big data management system according to claim 1, characterized in that: Also includes: The data processing unit is used to process the user request according to the preset data processing requirements corresponding to the user request during the process of forwarding the user request, and return the data processing result to the service unit or the adapter unit.

6. The cross-ecological cluster big data management system according to claim 5 is characterized in that: The data processing unit also includes: The screening module is used to screen the user requests to obtain user requests of a preset type, so that the data processing unit performs data processing on the user requests of the preset type.

7. A cross-ecological cluster big data management method, characterized in that: The method comprises: The service unit receives the user request and sends the user request to the adaptation unit; The adapter unit determines the corresponding cluster identifier and driver type according to the user request, matches the driver instance port corresponding to the cluster identifier and driver type from a preset routing table, and performs routing addressing according to the driver instance port, thereby forwarding the user request to the target driver instance in the driver unit; The driver unit pre-establishes a connection relationship with a plurality of clusters using a preset driver configuration mechanism, and is used to match a cluster port corresponding to the user request from the target driver instance, and performs addressing according to the cluster port, and forwards the user request from the target driver instance to the target cluster; The preset driver configuration mechanism includes separate deployment of driver instances, driver programs and clusters.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: When the computer program is executed by the processor, the instructions of the method according to claim 7 are executed.

9. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor of a computer device, the computer program executes the instructions of the method according to claim 7.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, executes instructions of the method according to claim 7.

Images