Database interaction system and method

Abstract

The invention discloses a database interaction system and method, and relates to the technical field of data management, and the system comprises an index collection module which carries out the data collection of each level node when a data request instruction of a user is detected, and obtains index data; the data processing module performs data structure conversion on user behavior data in the index data to obtain a database operation request; the data source management module loads a pre-configured cluster configuration file into an HBase database for link initialization; and the database interaction module calls the client to perform data interaction with the initialized HBase database according to the database operation request. Due to the fact that all the modules are introduced, when the user operates the HBase database, the user behavior data can be collected and flexibly converted according to reality, the workload can be reduced even in the face of scenes with complex service logic, various use scenes can be met from the aspects of management and use, and management and control over HBase interaction are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data management, and particularly relates to a database interaction system and method. BACKGROUND

[0002] HBase (Hadoop Database, distributed column storage database) is a distributed column-oriented database with high reliability and high scalability, and can automatically complete horizontal partitioning and concurrent access of data. For data interaction of HBase, a HBase interaction component is generally used. Users can communicate with HBase and perform various operations by using the HBase interaction component, for example, allowing users to input commands to create tables, insert data, query data, delete data, and the like.

[0003] However, the traditional HBase interaction component is difficult for users to use when facing a complex business logic scenario, and the workload is large, and it is difficult to control the HBase interaction. SUMMARY

[0004] The main purpose of the present application is to provide a database interaction system and method, which aims to solve the technical problem that the traditional HBase interaction component is difficult for users to use when facing a complex business logic scenario, and the workload is large, and it is difficult to control the HBase interaction.

[0005] To achieve the above purpose, the present application provides a database interaction system, which is applied to an HBase database, and the system comprises an index collection module, a data processing module, a database interaction module and a data source management module.

[0006] The index collection module is configured to collect data of each level node of the system when detecting a data request instruction sent by a user, and obtain index data.

[0007] The data processing module is configured to convert user behavior data in the index data into a data structure, and obtain a database operation request corresponding to the HBase database.

[0008] The data source management module is configured to load a cluster configuration file configured by the user in advance into the HBase database for link initialization.

[0009] The database interaction module is configured to call a client to interact with the initialized HBase database according to the database operation request.

[0010] In an embodiment, the database interaction module is further configured to define underlying behaviors for interacting with the HBase database, the underlying behaviors including metadata operations, data query operations, data change operations, and data buffering operations, respectively;

[0011] The database interaction module is further configured to parse the database operation request to obtain a database native operation corresponding to the underlying behavior;

[0012] The database interaction module is further configured to perform data interaction with the initialized HBase database according to the database native operation.

[0013] In an embodiment, the data processing module is further configured to extract user behavior data and a user-defined data structure from the index data;

[0014] The data processing module is further configured to determine a data type of the user behavior data, the data type including entity mapping, custom logic, and condition constructor;

[0015] The data processing module is further configured to convert the user behavior data from the user-defined data structure to a data structure for operating the HBase database based on the data type, to obtain a corresponding database operation request.

[0016] In an embodiment, the data processing module is further configured to extract original interaction data from the HBase database according to an interaction result;

[0017] The data processing module is further configured to determine a data structure of the HBase database;

[0018] The data processing module is further configured to convert the original interaction data from the data structure of the HBase database to the user-defined data structure, to obtain target interaction data.

[0019] In an embodiment, the system further comprises a behavior template module;

[0020] The behavior template module is configured to define action templates of the user for operating the HBase database according to operation types, the operation types including adding, deleting, querying, and metadata;

[0021] The index collection module is further configured to collect data of each level node used by the user when detecting a data request instruction sent by the user, to obtain user behavior data, the user behavior data being limited by the action templates;

[0022] The index collection module is further configured to record performance indicators of the data processing module;

[0023] The index collection module is further configured to collect the user behavior data and the performance index as index data.

[0024] In an embodiment, the system further comprises an authentication module;

[0025] The authentication module is configured to configure an authentication mode according to the authentication request of the user when receiving the authentication request of the user, the authentication mode comprising a registration mode and a local mode;

[0026] The authentication module is further configured to provide a corresponding authentication interface to the user, and the user configures a cluster based on the authentication interface;

[0027] The authentication module is further configured to obtain a cluster configuration file according to the configuration result.

[0028] In an embodiment, the system further comprises a specification detection module;

[0029] The specification detection module is further configured to perform specification detection on the user behavior data according to the authentication mode, and obtain a detection result, wherein the preset detection items corresponding to the registration mode comprise operation permission, performance index and data specification, and the preset detection items corresponding to the registration mode and the local mode further comprise cell capacity, operation time consumption and batch operation record number.

[0030] The specification detection module is further configured to perform abnormal behavior warning on the user when there is an abnormal index in the corresponding preset detection item in the detection result.

[0031] In addition, to achieve the above-mentioned purpose, the present application further provides a database interaction method, which is applied to a database interaction system, the database interaction system is applied to an HBase database, and the system comprises an index collection module, a data processing module, a database interaction module and a data source management module; the method comprises:

[0032] The index collection module collects data of each level node of the system when detecting a data request instruction sent by a user, and obtains index data;

[0033] The data processing module performs data structure conversion on user behavior data in the index data, and obtains a database operation request corresponding to the HBase database;

[0034] The data source management module loads a cluster configuration file pre-configured by the user into the HBase database for link initialization;

[0035] The database interaction module calls the client to interact with the initialized HBase database according to the database operation request.

[0036] In an embodiment, the step of the database interaction module calling the client to interact with the initialized HBase database according to the database operation request comprises:

[0037] The database interaction module defines the underlying behaviors of interacting with the HBase database, and the underlying behaviors respectively include metadata operation, data query operation, data change operation and data buffering operation;

[0038] The database interaction module parses the database operation request to obtain the database native operation corresponding to the underlying behavior;

[0039] The database interaction module calls the client to interact with the initialized HBase database according to the database native operation.

[0040] In an embodiment, the step of the data processing module performing data structure conversion on the user behavior data in the index data to obtain the database operation request corresponding to the HBase database comprises:

[0041] The data processing module extracts the user behavior data and the user-defined data structure in the index data;

[0042] The data processing module determines the data type of the user behavior data, and the data type includes entity mapping, custom logic and condition constructor;

[0043] The data processing module converts the user behavior data from the user-defined data structure to the data structure when operating the HBase database based on the data type, to obtain the corresponding database operation request.

[0044] The one or more technical solutions provided in the application have at least the following technical effects: the database interaction system of the application is applied to an HBase database, and the system comprises an index collection module, a data processing module, a database interaction module, and a data source management module; the index collection module is configured to collect data of each level node of the system to obtain index data when detecting a data request instruction sent by a user; the data processing module is configured to perform data structure conversion on user behavior data in the index data to obtain a database operation request corresponding to the HBase database; the data source management module is configured to load a cluster configuration file configured by the user in advance into the HBase database for link initialization; and the database interaction module is configured to call a client to perform data interaction with the initialized HBase database according to the database operation request.

[0045] Due to the introduction of the index collection module, the data processing module, the database interaction module, and the data source management module, user behavior data can be collected and flexibly converted according to actual conditions when the user operates the HBase database, and the workload can be reduced even in a complex business logic scenario, so that various use scenarios can be met in management and use, and the control of HBase interaction is realized. BRIEF DESCRIPTION OF DRAWINGS

[0046] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the application.

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0048] Figure 1 The functional module diagram provided for the database interaction system embodiment one of the application;

[0049] Figure 2 The data processing logic diagram of query provided for the embodiment one of the application;

[0050] Figure 3 The data processing logic diagram of newly added and updated data provided for the embodiment one of the application;

[0051] Figure 4 The data processing logic diagram of deletion provided for the embodiment one of the application;

[0052] Figure 5 The flowchart of the authentication module provided for the embodiment one of the application;

[0053] Figure 6 The flow chart of the data source management module provided for Embodiment One of the present application;

[0054] Figure 7 The flow chart of the behavior template module provided for Embodiment Two of the present application;

[0055] Figure 8 The user's use journey map provided for Embodiment Two of the present application;

[0056] Figure 9 The flow diagram provided for Embodiment One of the database interaction method of the present application.

[0057] The implementation, functional features and advantages of the present application will be further described with reference to the accompanying drawings in conjunction with embodiments. DETAILED DESCRIPTION

[0058] It should be understood that the specific embodiments described herein are merely intended to explain the technical solutions of the present application, and are not intended to limit the present application.

[0059] In order to better understand the technical solutions of the present application, the following will be described in detail in conjunction with the drawings and specific embodiments of the specification.

[0060] It should be noted that the traditional HBase interaction components are mainly divided into four categories, including HBase interaction components based on no authentication, HBase interaction components based on authentication, HBase interaction components based on single cluster and HBase interaction components based on multi-cluster, and their respective defects are as follows:

[0061] 1. HBase interaction components based on no authentication, this type is a commonly used cluster interaction method, the common method is to introduce a client in the code and establish a database link pool for related operations, and the cluster is connected through a non-authentication form. First, it cannot be guaranteed in terms of security, and data leakage is easy to occur after link leakage. Secondly, many key configuration items of the cluster are easy to be missed when writing code, and the workload is very large.

[0062] 2. HBase interaction components based on authentication, this type is also a commonly used cluster interaction method, which needs to configure the required authentication file in advance to the specified directory, then introduce the file configuration in the code, and after logging in the authentication, use the original client to use HBase. This method is not very friendly to beginners, and it is difficult to start. It needs to have a certain understanding of the storage structure and syntax of HBase, and also needs to be familiar with the related API(Application Programming Interface, application programming interface) of the client, which has a certain initial learning cost.

[0063] 3. Single-cluster-based HBase interaction component, such an interaction mode is a cluster usage mode in most single-user scenarios, and common data exists in a single cluster, so there is no high requirement for data availability and cluster security, and it becomes difficult to expand once the operation behavior quality of data has higher requirements.

[0064] 4. Multi-cluster-based HBase interaction component, such an interaction mode is commonly used in scenarios with high requirements for HBase data availability, and has high and more requirements for performance and availability indicators such as latency, success rate, and service availability, so this scenario usually means complex business logic, difficult management of multiple clusters, heavy workload, and particular attention to details of the connection of clusters.

[0065] Therefore, to solve the above problems of the HBase interaction component, the present application provides a database interaction system, which introduces related configurations and components, calls related data interfaces for use of the HBase database, provides authentication approaches of registration mode and local mode, configuration and authentication file management strategies of local deployment and remote deployment, and flexibly configures according to actual conditions. Data processing is divided into native, entity mapping, and custom logic, thereby saving repeated data processing work. The cluster mode can be adjusted as needed, supports single cluster and mutual trust multi-cluster scenarios, and meets various use scenarios in management and use.

[0066] Based on this, the present application embodiment one is provided. The present application embodiment provides a database interaction system, which refers to Figure 1 , Figure 1 The function module diagram provided by the database interaction system embodiment one of the present application.

[0067] In this embodiment, the database interaction system is applied to an HBase database, and the database interaction system comprises an index collection module 10, a data processing module 20, a database interaction module 30, and a data source management module 40.

[0068] The index collection module 10 is configured to collect data of each level node of the system to obtain index data when detecting a data request instruction sent by a user.

[0069] It should be noted that the index collection module undertakes the task of data collection in the whole system, is configured to operate on each level node in the system, and can start a data collection process according to a data request instruction sent by a user.

[0070] It can be understood that the data request instruction can be an instruction sent by a user for interacting with certain data of the HBase database in the system.

[0071] The HBase database is a distributed column-oriented non-relational database. The HBase database is built on the Hadoop distributed file system and can use the high reliability and high fault tolerance of the HDFS to store massive data, and is suitable for processing random read-write operations of large-scale data.

[0072] It should be understood that the hierarchical node refers to the node where different modules in the database interaction system are located.

[0073] It should be noted that the index data is quantitative information for measuring the performance of various operations of the user in the process of using the system, including the performance index data of the key nodes when the user uses the module, user information and behavior data, and performance indicators including input data volume, input data capacity, output data volume, output data capacity, module call start time, module call end time, database call start time, database call end time, operation result, etc. User information and behavior data include instance name, service name, action type, namespace, table name, registration information, link log ID, etc.

[0074] The use of the index collection module runs through a complete request process. When the request enters the system, it starts recording, and the main recording nodes are distributed in the following three modules: the behavior template module, which records user basic information, namespace and table name, time node, action type, and part of the performance indicators such as operation result, time point, and influence record number. The data processing module mainly records performance indicators such as data input capacity and data output capacity. The HBase interaction module records database performance indicators such as the time node of initiating database interaction.

[0075] After the index data is collected, it is sent to the designated location by an asynchronous task through an interface. Subsequently, other modules can analyze user behavior, data structure compliance, query action, operation time consumption, and whether the success rate is normal.

[0076] In a specific implementation, the index collection module identifies a user data request instruction, sequentially locates each hierarchical node according to the system module structure, and obtains the index data involved by the user when using the system from each module.

[0077] The data processing module 20 is configured to perform data structure conversion on the user behavior data in the index data to obtain a database operation request corresponding to the HBase database.

[0078] It should be noted that the data processing module is a module for processing data, which involves operations such as query, addition, update, and deletion of data processing of the HBase database.

[0079] It can be understood that the user behavior data is data related to user operations, such as operation results, time points, affected record numbers, action types, and the like of the user in each module of the system.

[0080] It should be understood that the database operation request is a specific instruction to the HBase database for obtaining data. When data is to be obtained from the HBase database, the data processing module needs to convert the user-defined data structure into a data structure for operating the HBase database.

[0081] In a specific implementation, considering that different data storage and systems require different data structures, the original user behavior data can be a user-defined data structure, and the HBase database needs to store and query data in a specific key-value pair. Therefore, when the data processing module receives the user behavior data in the index data, it can convert the user-defined data structure into a data structure for operating the HBase database to obtain the corresponding database operation request of the HBase database.

[0082] In a possible implementation, the data processing module 20 of the embodiment is also used to extract the user behavior data and the user-defined data structure in the index data; the data processing module 20 is also used to determine the data type of the user behavior data, the data type including entity mapping, custom logic, and condition constructor; and the data processing module 20 is also used to convert the user behavior data from the user-defined data structure into a data structure for operating the HBase database based on the data type to obtain the corresponding database operation request.

[0083] It should be noted that the entity mapping as a data type is a way of establishing a corresponding relationship between entities in the real world and data objects in the database. For the structure mapping logic of the entity, it can be divided into: identification of namespace and table name, identification of Rowkey (row key), and identification of column family, which are used to specify the key metadata of the table.

[0084] It can be understood that the custom logic is a processing logic set by the user, and in the data processing process, the custom logic determines how the user behavior data is processed. For the mapping logic of the custom structure, the user needs to implement specific data processing logic, and subsequent processing logic will also depend on the user's custom processing logic to process the data into a custom form.

[0085] It should be understood that the condition constructor is a data type for constructing query conditions, which can be constructed according to the attributes or values in the business data generated by the user behavior to construct the conditions for querying the HBase database. For example, for the mapping logic of the condition constructor, the specified operation condition can be encapsulated into a condition constructor for specifying the user's use behavior, and the user configures the configuration items related to the operation as needed. When constructing the native condition, the corresponding configuration items are converted into the corresponding operation condition according to the user's configuration of the condition constructor.

[0086] For the data processing module, the logic of converting the user-defined data structure into the data structure for operating the HBase database mainly involves the operations of querying, adding, updating and deleting the HBase database. The various operations of converting the user-defined data structure into the database data structure are as follows:

[0087] 1. For the query operation, it is mainly divided into: mapping the user data entity into the database operation request, mapping the query condition constructed by the user into the data operation request, and mapping the other data structure customized by the user into the database operation request. It is similar to an m:n data conversion structure, m is used for user recognizable structure, and n is used for database recognizable structure.

[0088] For example, refer to Figure 2 , Figure 2 The data processing logic diagram provided by the first embodiment of the present application for querying. GET (get), GETS (Get Scanner, batch get) and SCAN (scan) are instructions for operating the HBase database, which are used to get a single record, multiple records and scan data. Entity (point to specific data object), custom (user-defined operation), condition constructor and native (directly use the native query method of HBase database), are different ways to handle HBase database requests. Through the above data processing module, the original user-defined data structure can be converted into a native database operation request (i.e. database operation request). Then the database operation request can be sent to the database interaction module (also known as HBase interaction module), which is responsible for handling the interaction with the HBase database to realize the response of the user and the native database query operation.

[0089] 2. For the operation of adding or updating, it is mainly needed to map the user-defined data structure into the database operation request, which is also an m:n data conversion structure. The conversion of the operation response data is from the request result state to the fixed type of the number of successful operation data.

[0090] For example, refer to Figure 3 , Figure 3The data processing logic diagram provided by the first embodiment of the application for adding and updating is shown in the following table. PUT (add) is an instruction for writing to the HBase database. Entity, custom and native are different ways of processing HBase database requests. Through the above data processing module, the original user-defined data structure can be converted into a native database operation request. Then the database operation request can be sent to the HBase interaction module, which is responsible for processing the interaction with the HBase database, realizing the response of the user and the native database addition or update operation, and obtaining the complete process of the number of affected records.

[0091] 3. For the delete operation, the user's delete condition is mainly constructed into a database data deletion request, which is an m: 1 data conversion structure. The conversion of the operation response data is also a mapping from the request result state to the data number of a fixed type of successful operation.

[0092] For example, refer to Figure 4 , Figure 4 The data processing logic diagram provided by the first embodiment of the application for deleting is shown in the following table. DELETE (delete) is an instruction for deleting data from the HBase database. The user can construct the condition of the delete operation through the condition constructor, for example, specify the row key or column family of the record to be deleted, so that the original user-defined data structure can be converted into a native database operation request. Then the database operation request is sent to the HBase interaction module, which is responsible for processing the interaction with the HBase database, realizing the response of the user and the native database delete operation, and obtaining the complete process of the number of affected records.

[0093] In this embodiment, through the above data processing module, the user behavior data can be accurately extracted and processed and converted accordingly, so that the data can accurately adapt to the operation requirements of the HBase database, and the accuracy and efficiency of the database query can be improved.

[0094] In a possible implementation, the data processing module 20 of the embodiment is also used to extract original interaction data from the HBase database according to the interaction result; the data processing module 20 is also used to determine the data structure of the HBase database; and the data processing module 20 is also used to convert the original interaction data from the data structure of the HBase database into the user-defined data structure to obtain target interaction data.

[0095] It should be noted that the interaction result is the output generated by the interaction behavior between the database interaction module and the HBase database. The data processing module can determine the original interaction data extracted from the HBase database according to the interaction result.

[0096] For the HBase database, there is a specific data structure, including tables, row keys, column families, columns, etc. The user-defined data structure is a data organization method determined according to specific business needs. Therefore, the data structure needs to be converted through the data processing module.

[0097] It can be understood that the original interaction data is the data stored in the HBase database without processing. For example, the user's order operation is stored in the HBase database. The format and structure of these data are defined by the data structure of the HBase database, which contains various detailed information in the interaction process, but only meets the data structure of the HBase database, and needs to be further processed and converted into the user-defined data structure to obtain the target interaction data.

[0098] In this embodiment, the data processing module can convert the original interaction data obtained from the HBase database into the user-specified data structure to obtain the target interaction data. Thus, the data can better adapt to the user's business needs and adapt to different business scenarios, ensuring that the data can be effectively utilized in different application scenarios, and the complexity in the subsequent data processing process can be reduced.

[0099] The data source management module 40 is configured to load the cluster configuration file pre-configured by the user into the HBase database for link initialization.

[0100] It should be noted that the data source management module is a functional component specially responsible for managing data source related operations, and undertakes the task of configuring and initializing the HBase database, etc. According to the subsequent registration mode, different strategies can be adopted to load the configuration file into the HBase database for link initialization.

[0101] It can be understood that the cluster configuration file contains various setting information related to the HBase database cluster, such as the number of nodes in the cluster, the configuration parameters of the nodes (such as memory size, storage path, etc.), network related settings (such as IP address range, port number, etc.), and some key parameters of the HBase database itself (such as data block size, number of copies, etc.).

[0102] In a possible implementation, the cluster configuration file can be pre-configured when the user registers to use the database interaction system. The system of the present embodiment further comprises an authentication module. The authentication module is configured to configure an authentication mode according to the authentication request of the user when receiving the authentication request of the user, the authentication mode including a registration mode and a local mode. The authentication module is further configured to provide a corresponding authentication interface to the user, and the user configures the cluster based on the authentication interface. The authentication module is further configured to obtain the cluster configuration file according to the configuration result.

[0103] As shown in Figure 5 , Figure 5 The flowchart of the authentication module provided by Embodiment One of the present application is shown in

[0104] For the registration mode, registration (e.g. registration of SDK information for subsequent permission management and authentication) can be performed on the external permission management platform. Then, the registration information is configured to ensure that the connection and communication settings between the application and the permission management platform are correct. Then, the required permissions are obtained remotely from the permission management platform, which can be assigned by an administrator. Then, the operation is performed using the permissions obtained from the permission management platform, and the registration information is configured in the configuration to generate a cluster configuration file.

[0105] For the local mode, the cluster configuration and authentication file need to be prepared by the user, and the authentication information is configured to ensure that the local authentication settings are correct. Then, the local configuration content is specified in the configuration, including the file location, user information, etc., for subsequent program operation. Finally, the authentication mode is completed.

[0106] In the present embodiment, the two authentication modes described above are suitable for different application scenarios, providing a flexible authentication and permission management method, and improving the convenience and flexibility of the user in using the system.

[0107] Further, the data source management module will also have certain differences according to the mode of the front-end authentication module. The main difference is in the loading logic of the specification and configuration. In addition to the configuration items in the cluster configuration file, the remote management center is allowed to add additional configurations, which are loaded when the link is initialized. The local mode is loaded according to the default specification and user-configured cluster configuration file, without additional processing.

[0108] As shown in Figure 6 , Figure 6 The flowchart of the data source management module provided by Embodiment One of the present application is shown in

[0109] The database interaction module 30 is configured to call the initialized HBase database to interact with the client according to the database operation request.

[0110] In a specific implementation, since the user has preconfigured a cluster configuration file when authenticating the module, the data source management module can load the cluster configuration file into the HBase database after obtaining the cluster configuration file, and complete the initialization of the database, including setting parameters, constructing data structures, and the like. When a database operation request is generated, the database interaction module receives the request, and then calls the initialized HBase database to perform data searching, extracting, or storing, and the like according to the request content.

[0111] In another possible implementation, the database interaction module 30 is further configured to define underlying behaviors for interacting with the HBase database, and the underlying behaviors include metadata operations, data query operations, data change operations, and data buffering operations. The database interaction module 30 is further configured to analyze the database operation request to obtain database native operations corresponding to the underlying behaviors. The database interaction module 30 is further configured to call the initialized HBase database to interact with the client according to the database native operations.

[0112] It should be noted that the HBase interaction module defines underlying behaviors for interacting with the HBase database, and the module can be divided into four parts, namely, Admin (metadata operation), Query (data query operation), Put (data change operation), and Mutator (data buffering operation).

[0113] For the metadata operation, the operation defines user operation behaviors on metadata, including obtaining information about namespaces, tables, column families, columns, and the like, and in addition, includes table structure change operations, and the like, which are all performed by interacting with the HBase database through the module.

[0114] For the data query operation, the operation defines user query operations on specific data in a table, and includes single Rowkey (row key) query, multiple Rowkey query, and SCAN query, and the like, which are all underlying data entrances for query operations on data in a table.

[0115] For the data change operation, the operation defines real-time change operations of users on data in a table, and includes adding, updating, and deleting data, and the like, which are all underlying data entrances for real-time change operations on data in a table.

[0116] For data buffering operation, this action defines the asynchronous buffering change operation of the user to the data in the table. It respectively contains the use of adding, updating and deleting data scenes. It is in the asynchronous scene or the scene with a large number of change requests. In order to save network overhead and improve the write performance of the underlying data entry.

[0117] In this embodiment, for all the upper-layer encapsulated user behavior data, the HBase interaction module is parsed into the native operation, and by explicitly distinguishing different underlying behaviors (metadata, query, change, buffering operation), the database interaction can be more modularized and standardized, so as to improve the accuracy and efficiency of the interaction.

[0118] In the technical scheme provided in the embodiment, the index collection module identifies a user data request instruction, sequentially locates each level node according to a system module structure, and obtains index data related to the user when using the system from each module. Considering that different data storage and systems need different data structures, the original user behavior data can be a user-defined data structure, and the HBase database needs to store and query data in a specific key-value pair. Therefore, when the data processing module receives the user behavior data in the index data, the user-defined data structure can be converted into a data structure for operating the HBase database, and a database operation request corresponding to the HBase database is obtained. Since the user has preconfigured a cluster configuration file in the authentication module, the data source management module can load the cluster configuration file into the HBase database after obtaining the cluster configuration file, and complete the link initialization work of the database, including setting parameters, building data structures, etc. When a database operation request is generated, the database interaction module can call the link of the initialized HBase database, and perform interactive operations such as data searching, extracting or storing according to the request content. Since the above modules are introduced in the embodiment, the user behavior data can be collected and flexibly converted according to the actual situation when the user operates the HBase database. Even in the scene with complex business logic, the workload can be reduced, and various use scenes can be met in management and use, and the control of the HBase interaction is realized.

[0119] Based on the above embodiment one of the application, the second embodiment of the application is proposed. In the second embodiment of the application, the same or similar contents as the above embodiment one can refer to the above introduction, and will not be described hereinafter.

[0120] The system also includes a behavior template module; the behavior template module is configured to define an action template of the user operating the HBase database according to an operation type, and the operation type includes adding, deleting, querying, and metadata; the index collection module is further configured to collect data of each level node of the user using the system to obtain user behavior data when detecting a data request instruction sent by the user, and the user behavior data is limited by the action template; the index collection module is further configured to record the performance index of the data processing module; and the index collection module is further configured to take the user behavior data and the performance index as index data.

[0121] It should be noted that the behavior template module defines an action template of the user operating the database, and the public behavior of the user can be divided into four operation types, i.e., adding (or updating), deleting, querying, and metadata according to the operation type, and a corresponding data interface can be provided according to a specific user scenario. Then the data form is:

[0122] For the adding (or updating) template, the adding or updating logic of the user to the database can be defined, and the change behavior to the data is limited within a specified range. According to different performance requirements of different scenarios, real-time change operations and change operations with a buffer are provided. The change operation with the buffer is more suitable for the scenario of high-frequency or large-batch data change, and is used to save network overhead and improve the performance of the change.

[0123] For the deleting template, the deleting logic of the user to the database data can be defined, and the deleting action of the user to the database data is limited within a specified range. At the same time, according to different performance and data forms, the deleting template also provides real-time and change operations with a buffer, and supports different data APIs with different input forms, thereby increasing the flexibility and applicability of the template.

[0124] For the querying template, the querying action of the user to the database data can be defined, and the querying action of the user to the database is limited within a limited range, and the querying scenario is mainly divided into single-point querying of single Rowkey, multi-point querying of multiple Rowkeys, range querying based on conditions, and the like.

[0125] For example, refer to Figure 7 , Figure 7 The flowchart of the behavior template module provided in Embodiment Two of the present application. The user initiates a database operation request through public behavior (such as adding or updating, deleting, querying, and the like), the request is processed through a data form, which may involve entities, custom, condition constructor, or native, and the like. The processed request can be converted into a data structure through a data processing module, and finally interact with the HBase database through an HBase interaction module.

[0126] In this embodiment, the action template defined by the behavior template module can make the user's operation on the HBase database more standardized and normalized, and reduce errors caused by non-standard operation. At the same time, it is convenient for the subsequent horizontal expansion of each template, and supports more flexible range and form of scenarios.

[0127] Further, the system further comprises a specification detection module; the specification detection module is further configured to perform specification detection on the user behavior data according to the preset detection items of the authentication mode, to obtain a detection result, wherein the preset detection items corresponding to the registration mode include operation permission, performance index and data specification, and the preset detection items corresponding to the local mode and the registration mode further include cell capacity, operation time consumption and batch operation record number; the specification detection module is further configured to perform abnormal behavior warning on the user when there is an abnormal index in the corresponding preset detection item in the detection result.

[0128] It should be noted that the HBase specification detection module defines the rationality constraint of the user's use of the HBase database. For a cluster applied by a certain business from a big data management platform, the use of the HBase data by the developer does not follow a unified specification as a constraint. The unreasonable use of data by novice personnel may cause the operation to occupy too many resources of the HBase data, consume the performance of the cluster, and even cause cluster problems and business production events.

[0129] Therefore, in order to constrain user behavior and ensure the high availability of the cluster, the behavior of the user use module is also detected. This part is in the form of an aspect, which is cut into the behavior template module and the data processing module, and the detection content is different according to the authentication mode.

[0130] In the local mode, the cell capacity, operation time consumption, batch operation record number and the like of the user are detected and warned. In the registration mode, the detection rules include permission verification, performance index, data specification detection, in addition to the data specification and performance index specification detection as in the local mode, additional permission verification and the like can also be added.

[0131] In this embodiment, through the specification detection module, the abnormal behavior of the user can be found in time, and the user knows that there is a problem in the operation, which helps to improve the compliance of the user's operation and reduce the damage caused by improper operation to the system.

[0132] By way of example, in order to help understand the implementation process of the database interaction system obtained after combining the above-mentioned embodiment one and embodiment two, please refer to Figure 8 , Figure 8 The user's use journey map provided for the embodiment two of the present application is specifically:

[0133] The behavior journey of a user includes interaction with the HBase underlying module from the behavior template module. The user first performs identity authentication at the authentication module through registration with the management platform. After successful authentication, the user's behavior is recorded through the behavior template module. The recorded user behavior is collected by the index collection module and passed to the data processing module. The data processing module converts the user-defined data structure into a data structure for operating the HBase database. At the same time, the data source management module is responsible for managing the data sources of single-cluster and multi-cluster in the HBase database, and loading the cluster configuration file pre-configured by the user into the HBase database for link initialization. At this time, the HBase interaction module is responsible for interacting with the HBase database through the HBase client, processing data storage and retrieval.

[0134] In addition, the index collection module can also output the collected index data, calculate and analyze the index data for subsequent analysis and monitoring, to support system operation and data review.

[0135] Further, the output of the behavior template module is passed to the specification detection module, which is used to detect whether the user behavior conforms to the predefined specification. If a behavior that does not conform to the specification is detected, the specification detection module outputs an alarm information.

[0136] By introducing the above modules, when the user operates the HBase database, the user behavior data can be collected and flexibly converted according to the actual situation. Even in the face of complex business logic scenarios, the workload can be reduced, and various use scenarios can be met in management and use, realizing the management and control of HBase interaction.

[0137] Reference Figure 9 The above database interaction system of the present application provides a database interaction method, Figure 9 The flowchart provided by the first embodiment of the database interaction method of the present application. The database interaction system is as described in the above embodiment, and is applied to the HBase database. The system includes an index collection module, a data processing module, a database interaction module, and a data source management module.

[0138] In this embodiment, the database interaction method includes steps S10-S40:

[0139] Step S10: The index collection module collects data from each level node of the system when detecting a data request instruction sent by the user, and obtains index data.

[0140] It should be noted that the index collection module undertakes the task of data collection in the whole system, and is used for operating on each hierarchical node in the system, and can start the data collection process according to a data request instruction sent by a user.

[0141] It can be understood that the data request instruction can be an instruction issued by the user for interacting with some data of the HBase database in the system.

[0142] The HBase database is a distributed column-oriented non-relational database. The HBase database is built on the Hadoop distributed file system, and can use the high reliability and high fault tolerance of the HDFS to store massive data, and is suitable for processing random read-write operations of large-scale data.

[0143] It should be understood that the hierarchical node refers to a node where different modules in the database interaction system are located.

[0144] It should be noted that the index data is quantitative information for measuring the performance of various operations of the user in using the system, including performance index data of a key node when the user uses the module, user information and behavior data, and performance indexes including input data volume, input data capacity, output data volume, output data capacity, module call start time, module call end time, database call start time, database call end time, operation result, etc. The user information and behavior data include instance name, service name, action type, namespace, table name, registration information, link log ID, etc.

[0145] In a specific implementation, the index collection module identifies the user data request instruction, sequentially locates each hierarchical node according to the system module structure, and obtains the index data related to the use of the system by the user from each module.

[0146] Step S20: The data processing module performs data structure conversion on the user behavior data in the index data, and obtains a database operation request corresponding to the HBase database.

[0147] It should be noted that the data processing module is a module for processing data, and involves operations such as querying, adding, updating, and deleting data processing of the HBase database.

[0148] It can be understood that the user behavior data is data related to user operations, such as operation results, time points, affected record numbers, action types, etc.

[0149] It should be understood that the database operation request is a specific instruction to the HBase database for obtaining data. When data is to be obtained from the HBase database, the data processing module needs to convert the user-defined data structure into a data structure for operating the HBase database.

[0150] In a specific implementation, considering that different data storage and systems require different data structures, the original user behavior data can be a user-defined data structure, and the HBase database needs to store and query data in a specific key-value pair. Therefore, when the data processing module receives the user behavior data in the index data, it can convert the user-defined data structure into a data structure for operating the HBase database to obtain the corresponding database operation request of the HBase database.

[0151] In a possible implementation, the step S20 of the embodiment can include the following steps: the data processing module extracts the user behavior data in the index data and a user-defined data structure; the data processing module determines the data type of the user behavior data, the data type including entity mapping, custom logic, and condition constructor; and the data processing module converts the user behavior data from the user-defined data structure into a data structure for operating the HBase database based on the data type to obtain the corresponding database operation request.

[0152] It should be noted that the entity mapping, as a data type, is a way of establishing a corresponding relationship between entities in the real world and data objects in the database. The structure mapping logic of the entity can be divided into: identification of namespace and table name, identification of Rowkey (row key), and identification of column family. These data are used to specify the key metadata of the table.

[0153] It can be understood that the custom logic is a processing logic set by the user, and in the data processing process, the custom logic determines how the user behavior data is processed. For the mapping logic of the custom structure, the user needs to implement specific data processing logic, and the subsequent processing logic will also depend on the user's custom processing logic to process the data into a custom form.

[0154] It should be understood that the condition constructor is a data type for constructing query conditions, which can construct conditions for querying the HBase database according to the attributes or values in the user behavior data. For example, for the mapping logic of the condition constructor, the specified operation condition can be encapsulated into a condition constructor for standardizing the user's usage behavior. When the user constructs the native condition, the corresponding configuration items will be converted into the corresponding operation condition according to the user's configuration of the condition constructor.

[0155] For the data processing module, the user-defined data structure is converted into the logic of the data structure when operating the HBase database, mainly involving the operations of querying, adding, updating and deleting the HBase database.

[0156] In the embodiment, the data processing module can accurately extract and process the user behavior data and perform corresponding conversion, so that the data can accurately adapt to the operation requirements of the HBase database, and the accuracy and efficiency of database query can be improved.

[0157] Step S30: The data source management module loads the cluster configuration file pre-configured by the user into the HBase database for link initialization.

[0158] It should be noted that the data source management module is a functional component specially responsible for managing data source related operations, and undertakes the tasks of configuring, initializing and the like of the HBase database.

[0159] It can be understood that the cluster configuration file contains various setting information related to the HBase database cluster, such as the number of nodes in the cluster, the configuration parameters of the nodes (such as memory size, storage path, etc.), network related settings (such as IP address range, port number, etc.), and some key parameters of the HBase database itself (such as data block size, number of replicas, etc.).

[0160] Step S40: The database interaction module calls the client to interact with the initialized HBase database according to the database operation request.

[0161] In a specific implementation, since the user has pre-configured the cluster configuration file when authenticating the module, after the data source management module obtains the cluster configuration file, it can load it into the HBase database to complete the initialization of the database, including setting parameters, building data structures, etc. When a database operation request is generated, the database interaction module receives the request, then calls the initialized HBase database, and performs data searching, extracting or storing and the like interactive operations according to the request content.

[0162] In another possible implementation, the step S40 of the embodiment can include the steps of: the database interaction module defining the underlying behaviors of interacting with the HBase database, the underlying behaviors including metadata operations, data query operations, data change operations and data buffering operations respectively; the database interaction module parsing the database operation request to obtain the database native operation corresponding to the underlying behaviors; and the database interaction module calling the client to interact with the initialized HBase database according to the database native operation.

[0163] It should be noted that the HBase interaction module defines the underlying behavior of interacting with the HBase database. This module can be divided into four parts: Admin (metadata operations), Query (data query operations), Put (data change operations), and Mutator (data buffering operations).

[0164] For metadata operations, this action defines the user's metadata operation behavior, including obtaining information such as namespaces, tables, column families, and columns. In addition, it also includes operations such as table structure changes. All of these interact with the HBase database through this module.

[0165] For data query operations, this action defines the user's query operation on specific data in the table, including scenarios such as single Rowkey query, multi-Rowkey query, and SCAN query. It is the underlying data entry for query actions on data in the table.

[0166] For data change operations, this action defines the user's real-time change operations on data in the table, including the use of data addition, update, and deletion scenarios. It is the underlying data entry for real-time change operations on data in the table.

[0167] Data buffering defines asynchronous buffering of changes to table data. This includes adding, updating, and deleting data. It serves as the underlying data entry point to reduce network overhead and improve write performance in asynchronous scenarios or when there are a large number of change requests.

[0168] In this implementation, all upper-layer encapsulated user behavior data will be parsed into native operations in the HBase interaction module, and by clearly distinguishing different underlying behaviors (metadata, queries, changes, buffer operations), database interactions can be made more modular and standardized to improve the accuracy and efficiency of interactions.

[0169] The database interaction method provided in this application, applied to the database interaction system in the above-mentioned embodiments, can address the technical issues of traditional HBase interaction components, which are difficult for users to use and require a lot of work when faced with complex business logic, and are difficult to manage and control HBase interactions. Compared with the prior art, the beneficial effects of the database interaction method provided in this application are the same as those of the database interaction system provided in the above-mentioned embodiments, and the other technical features of the database interaction method are the same as those disclosed in the above-mentioned embodiment systems, and are not further described here.

[0170] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the database interaction system and method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0171] The above merely provides part of embodiments of the present application, and does not limit the protection scope of the present application, and any equivalent structure transformation or direct / indirect application in other related technical fields made according to the technical concept of the present application and the content of the specification and drawings of the present application shall be included in the protection scope of the present application.

Claims

1. A database interaction system, characterized in that: The database interaction system is applied to the HBase database, and the system includes: an indicator collection module, a data processing module, a database interaction module and a data source management module; The indicator collection module is used to collect data from nodes at each level of the system to obtain indicator data when a data request instruction sent by a user is detected; The data processing module is used to perform data structure conversion on the user behavior data in the indicator data to obtain a database operation request corresponding to the HBase database; The data source management module is used to load the cluster configuration file pre-configured by the user into the HBase database for link initialization; The database interaction module is used to call the client to perform data interaction with the initialized HBase database according to the database operation request.

2. The system according to claim 1, wherein The database interaction module is further used to define the underlying behaviors of interacting with the HBase database, wherein the underlying behaviors include metadata operations, data query operations, data change operations, and data buffering operations; The database interaction module is further configured to parse the database operation request to obtain the database native operation corresponding to the underlying behavior; The database interaction module is further configured to call the client to perform data interaction with the initialized HBase database according to the database native operation.

3. The system according to claim 2, wherein: The data processing module is further used to extract user behavior data and user-defined data structures from the indicator data; The data processing module is further configured to determine the data type of the user behavior data, wherein the data type includes entity mapping, custom logic, and condition constructors; The data processing module is further configured to convert the user behavior data from the user-defined data structure into a data structure for operating the HBase database based on the data type, and obtain a corresponding database operation request.

4. The system according to claim 3, wherein: The data processing module is further configured to extract original interaction data from the HBase database according to the interaction results; The data processing module is further used to determine the data structure of the HBase database; The data processing module is further configured to convert the original interaction data from the data structure of the HBase database into the user-defined data structure to obtain target interaction data.

5. The system according to claim 1, wherein: The system also includes a behavior template module; The behavior template module is used to define the action templates for the user to operate the HBase database according to the operation type, and the operation type includes adding, deleting, querying and metadata; The indicator collection module is further configured to collect data on the user's use of nodes at each level of the system when a data request instruction sent by the user is detected, to obtain user behavior data, wherein the user behavior data is restricted by the action template; The indicator collection module is also used to record the performance indicators of the data processing module; The indicator collection module is further configured to use the user behavior data and the performance indicator as indicator data.

6. The system according to any one of claims 1 to 5, characterized in that The system also includes an authentication module; The authentication module is configured to configure an authentication mode according to the user's authentication request upon receiving the user's authentication request, wherein the authentication mode includes a registration mode and a local mode; The authentication module is further configured to provide a corresponding authentication interface to the user, and the user performs cluster configuration based on the authentication interface; The authentication module is further configured to obtain a cluster configuration file based on the configuration result.

7. The system according to claim 6, wherein: The system further comprises: a specification detection module; The standard detection module is further configured to perform standard detection on the user behavior data according to the authentication mode and preset detection items to obtain a detection result, wherein the preset detection items corresponding to the registration mode include operation permissions, performance indicators, and data specifications, and the preset detection items corresponding to the registration mode and the local mode also include cell capacity, operation time, and number of batch operation records; The standard detection module is further configured to issue an abnormal behavior warning to the user when there are abnormal indicators in the corresponding preset detection items in the detection results.

8. A database interaction method, characterized in that: The database interaction method is applied to a database interaction system, and the database interaction system is applied to an HBase database. The system includes: an indicator collection module, a data processing module, a database interaction module, and a data source management module; the method includes: When the indicator collection module detects a data request instruction sent by a user, it collects data from nodes at each level of the system to obtain indicator data; The data processing module performs data structure conversion on the user behavior data in the indicator data to obtain a database operation request corresponding to the HBase database; The data source management module loads the cluster configuration file pre-configured by the user into the HBase database to initialize the link; The database interaction module calls the client to perform data interaction with the initialized HBase database according to the database operation request.

9. The method according to claim 8, wherein The database interaction module calls the client to interact with the initialized HBase database for data according to the database operation request, including: The database interaction module defines the underlying behaviors for interacting with the HBase database, and the underlying behaviors include metadata operations, data query operations, data change operations, and data buffering operations. The database interaction module parses the database operation request to obtain the database native operation corresponding to the underlying behavior; The database interaction module calls the client to interact with the initialized HBase database according to the database native operation.

10. The method according to claim 9, wherein The step of converting the user behavior data in the indicator data into a data structure by the data processing module to obtain a database operation request corresponding to the HBase database includes: The data processing module extracts user behavior data and user-defined data structures from the indicator data; The data processing module determines the data type of the user behavior data, wherein the data type includes entity mapping, custom logic, and condition constructor; The data processing module converts the user behavior data from the user-defined data structure into a data structure for operating the HBase database based on the data type, and obtains a corresponding database operation request.

Images