A data platform, data processing method, device, equipment and storage medium

Abstract

The embodiment of the present invention discloses a data middle platform, a data processing method, an apparatus, a device and a storage medium, wherein the data middle platform includes multiple data warehouses and an intermediate layer shared by the multiple data warehouses, wherein: the intermediate layer is used to receive a data processing request sent by a client, determine a target structured query language template method associated with a control identifier carried in the data processing request, and call the target structured query language template method to execute the data processing request, and generate a response signal for response according to the execution result. The data middle platform provided by the embodiment of the present invention replaces the application programming interface in the middle layer with the corresponding structured query language template method, so that the data processing instructions of the middle layer can be applied to different versions of data warehouses, thereby realizing the use of the same middle layer to adapt to different versions of data warehouses, and reducing the difficulty of operation and maintenance of the data middle platform.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular to a data center, a data processing method, an apparatus, a device, and a storage medium. Background Art

[0002] With the continuous development of Internet technology and the continuous increase in data volume, the application of data middle platform is becoming more and more extensive. Based on big data technology, data middle platform collects, processes and stores massive data, which can break through information silos and enhance cooperation among various departments within the enterprise. As the most important part of the big data ecosystem, Hadoop and products derived from Hadoop have covered various fields of big data segmentation. HIVE, as an important part of the HADOOP ecosystem, relies on HDFS storage and builds a clear metadata MetaStore, which has long become the most mainstream data warehouse product on the market. The data middle platform solutions of major manufacturers are also based on HIVE to build the data warehouse of the data middle platform.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following technical problems in the prior art: when building a data middle platform, each business department often has built its own Hadoop cluster, so there are multiple versions of HIVE at the bottom of each business department. Due to the different versions of the underlying HIVE, one solution is to develop different versions of the data middle platform; and another solution is to build a central data warehouse and then migrate the data. The above two solutions not only have security risks, but also increase the difficulty of operation and maintenance of the entire system. Summary of the invention

[0004] The embodiments of the present invention provide a data middle platform, a data processing method, an apparatus, a device and a storage medium to achieve the use of the same middle layer to adapt to different versions of data warehouses, thereby reducing the difficulty of operation and maintenance of the data middle platform.

[0005] In a first aspect, an embodiment of the present invention provides a data middle platform, including multiple data warehouses and an intermediate layer shared by multiple data warehouses, wherein:

[0006] The middle layer is used to receive the data processing request sent by the client, determine the target structured query language template method associated with the control identifier carried in the data processing request, call the target structured query language template method to execute the data processing request, and generate a response signal according to the execution result.

[0007] In a second aspect, an embodiment of the present invention further provides a data processing method, which is executed by an intermediate layer in a data center provided by any embodiment of the present invention, and includes:

[0008] Receiving a data processing request sent by a client, wherein the data processing request is triggered by the client after detecting a click operation on a control;

[0009] Obtaining a control identifier carried in a data processing request, and determining a target structured query language template method associated with the control identifier;

[0010] The target structured query language template method is called to execute the data processing request, and a response signal is generated according to the execution result to respond.

[0011] In a third aspect, an embodiment of the present invention further provides a data processing device, which is configured in the middle layer of the data center provided by any embodiment of the present invention, and includes:

[0012] A request receiving module, used for receiving a data processing request sent by a client, wherein the data processing request is triggered by the client after detecting a click operation of a control;

[0013] A target template method module is used to obtain a control identifier carried in a data processing request and determine a target structured query language template method associated with the control identifier;

[0014] The request response module is used to call the target structured query language template method to execute the data processing request, and generate a response signal according to the execution result to respond.

[0015] In a fourth aspect, an embodiment of the present invention further provides a computer device, the device comprising:

[0016] one or more processors;

[0017] A storage device for storing one or more programs;

[0018] When one or more programs are executed by one or more processors, the one or more processors implement the data processing method provided by any embodiment of the present invention.

[0019] In a fifth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the data processing method provided by any embodiment of the present invention is implemented.

[0020] The data middle platform provided by the embodiment of the present invention includes multiple data warehouses and an intermediate layer shared by multiple data warehouses, wherein: the intermediate layer is used to receive a data processing request sent by a client, determine the target structured query language template method associated with the control identifier carried in the data processing request, and call the target structured query language template method to execute the data processing request, and generate a response signal according to the execution result for response. By replacing the application programming interface in the middle layer with the corresponding structured query language template method, the data processing instructions of the middle layer can be applied to different versions of data warehouses, realizing the use of the same middle layer to adapt to different versions of data warehouses, reducing the difficulty of operation and maintenance of the data middle platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1a It is a structural diagram of the data center in the prior art;

[0022] Figure 1b This is a schematic diagram of the structure of a data center provided in the first embodiment of the present invention;

[0023] Figure 2 It is a flowchart of a data processing method provided by Embodiment 2 of the present invention;

[0024] Figure 3 is a structural schematic diagram of a data processing device provided by Embodiment 3 of the present invention;

[0025] Figure 4 It is a structural diagram of a computer device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0027] Embodiment 1

[0028] Figure 1a This is a schematic diagram of the structure of the data center in the prior art. Figure 1a As shown, the traditional data middle platform architecture needs to develop an intermediate layer corresponding to the data warehouse version when the data warehouse version is not unified, that is, the traditional data middle platform contains multiple intermediate layers, each of which corresponds to a version of the data warehouse, and the development and maintenance costs are high. In response to the above-mentioned technical problems existing in the data middle platform in the prior art, an embodiment of the present invention provides a data middle platform to achieve the use of the same intermediate layer to adapt to different versions of data warehouses, thereby reducing the difficulty of operation and maintenance of the data middle platform.

[0029] Figure 1b 1 is a schematic diagram of the structure of a data middle platform provided in the first embodiment of the present invention. The data middle platform provided in this embodiment can be applied to data processing of multiple versions of data warehouses. Figure 1b As shown, the data middle platform provided by the embodiment of the present invention includes multiple data warehouses 110 and an intermediate layer 120 shared by multiple data warehouses, wherein: the intermediate layer is used to receive a data processing request sent by a client, determine a target structured query language template method associated with a control identifier carried in the data processing request, and call the target structured query language template method to execute the data processing request, and generate a response signal in response according to the execution result.

[0030] In this embodiment, only one middle layer is needed to be compatible with multiple versions of data warehouses, which effectively reduces the difficulty of operation and maintenance of the data middle station and controls the maintenance cost of the data middle station. Specifically, the data processing of the data middle station requires access to the metadata database of the underlying service MetaStore service. In order to access the metadata of multiple versions of data warehouses, the application programming interface (API) of MetaSotre is no longer used, but is implemented by directly accessing the metadata database MYSQL. Based on the above ideas, each method of MetaStore-API can be replaced by the select operation of the relational database management system (MySQL) to create a corresponding structured query language (SQL) template method. It can be understood that the structured query template language and its corresponding MetaStore-API method have the same logic, and the difference lies in the language. It can be understood that each method of MetaStore-API corresponds to the data operation performed by the user, and the processing operation performed by the user can be determined by the control identifier carried in the data processing request. Therefore, it can be understood that each control identifier corresponds to a structured query language method for the corresponding operation. Table 1 schematically shows the MetaStore-API method for data preview and its corresponding structured query language (SQL) template method.

[0031] Table 1

[0032]

[0033]

[0034] For example, in addition to the MetaStore-API related to the data warehouse metadata preview, some MetaStore-APIs perform delete / insert / update operations on the metadata database. For example, to create a database in the data warehouse, the traditional method calls the create_database method of the MetaStore-API, and in the embodiment of the present invention, the create database statement of HIVE-SQL is executed. To delete a database in the data warehouse, the traditional method calls the drop_database method of the Metastore-API, and in the embodiment of the present invention, the drop database method of HIVE-SQL is executed.

[0035] In order to achieve compatibility of multi-version data warehouses, the database management part uses HIVE-SQL to replace MetaStore-API, and creates a corresponding SQL template method. Table 2 schematically shows the MetaStore-API method of data management and its corresponding HIVE-SQL template method.

[0036] Table 2

[0037] MetaStore-API Methods SQL Template Method create_database CREATE DATABASE? drop_database DROP DATABASE? create_role CREATE ROLE? …… ……

[0038] In this embodiment, the data processing request may be any operation request of the database, such as a database creation request, query request, and edit request, etc. When the middle layer receives a data processing request initiated by a user through a client, the data processing request is parsed to obtain a control identifier carried in the data processing request, and the structured query language template method associated with the control identifier is used as the target structured query language template method, and then based on the processing parameters carried in the data processing request, the target structured query language template method is called to execute the operation corresponding to the data processing request, and a response is made according to the execution result.

[0039] Exemplarily, when a user needs to create a database, he enters the information of the database to be created (such as the name) on the client interface and clicks the create control to trigger a data processing request to create the database. The client generates a data processing request based on the information entered by the user and the triggered control, and sends the data processing request to the data center. After receiving the data processing request, the middle layer in the data center parses the data processing request, obtains the create control carried in the data processing request, determines the SQL template method associated with the create control as the target SQL template method, and then calls the target SQL template method to create the database and responds according to the creation result. Optionally, if the creation is successful, the created database is generated and sent to the client for display. If the creation fails, a failure prompt message is generated and sent to the client for display.

[0040] It can be understood that the above example is only a specific example when the data processing request is to create a database. As in the above embodiment, the data processing request can also be a request for querying or editing a database or table. The processing logic can refer to the processing logic for creating a database in the above embodiment, which will not be repeated here.

[0041] The data middle platform provided by the embodiment of the present invention includes multiple data warehouses and an intermediate layer shared by multiple data warehouses, wherein: the intermediate layer is used to receive a data processing request sent by a client, determine the target structured query language template method associated with the control identifier carried in the data processing request, and call the target structured query language template method to execute the data processing request, and generate a response signal according to the execution result for response. By replacing the application programming interface in the middle layer with the corresponding structured query language template method, the data processing instructions of the middle layer can be applied to different versions of data warehouses, realizing the use of the same middle layer to adapt to different versions of data warehouses, reducing the difficulty of operation and maintenance of the data middle platform.

[0042] In one embodiment of the present invention, calling the target structured query language template method to execute the data processing request, and generating a response signal according to the execution result to respond, including: obtaining the processing parameters carried in the data processing request, and using the processing parameters as the execution parameters of the target structured query language template method; executing the target structured query language template method based on the execution parameters, and obtaining the permission verification parameters associated with the data processing request; verifying the execution authority of the data processing request according to the permission verification parameters, and calling the target data warehouse to obtain the processing result of the data processing request after the permission verification passes, and responding with the processing result as the response signal. Optionally, in the above embodiment, the access control of the source data of the data warehouse has been unified in the middle layer. In order to further improve the stability of the data middle platform, the permission control can be unified on the basis of the above embodiment, wherein the unified verification of permissions can be realized in the BASE-API service of the middle layer. Specifically, the processing parameters carried in the data processing request include the operation type and the operation parameters. The processing parameters are executed as the execution parameters of the target structured query language template method, and then the permission verification parameters associated with the data processing request are obtained, and the permission verification is performed based on the permission verification parameters. Considering that there are no parsing tools for SPARK-SQL and HIVE-SQL in the BASE-API service of the middle layer, in order to realize unified control of permissions in the middle layer, it is necessary to introduce the AbstractSqlParser class in the SPARK module into the BASE-API module as the parsing tool for SPARK-SQL, and introduce the ParseDriver class in the HIVE module into the BASE-API module as the parsing tool for HIVE-SQL.

[0043] In one embodiment, the execution permission includes the execution type permission, the permission verification parameter includes the target structured query language type of the target structured query language template method, and the execution permission of the data processing request is verified according to the permission verification parameter, including: matching the target structured query language type with the preset disabled structured query language type; when the target structured query language type successfully matches the preset disabled structured query language type, determining that the execution type permission verification of the data processing request has failed; when the target structured query language type fails to match the preset disabled structured query language type, determining that the execution type permission verification of the data processing request has passed. In order to further enhance the security of the data middle station, at least one disabled structured query language type can be pre-set to prohibit the execution of the set disabled structured query language type. For example, when the SQL type is SET, the configuration files of HIVE and SPARK may be modified; when the SQL type is ADD, a jar package that does not exist in the system can be uploaded; when the SQL type is SHOW_TABLE_PROPERTIES or DESC_FORMATTED_TABLE, the HDFS information of the data will be exposed; when the SQL type is CREATE_TABLE_WITH_LOCATION, an external table can be created by specifying HDFS information to bypass permission verification. Therefore, the above SQL types may cause instability in the data center when executed. Based on this, the disabled structured query language types can be set to SET, ADD, SHOW_TABLE_PROPERTIES, DESC_FORMATTED_TABLE, and CREATE_TABLE_WITH_LOCATION. When the target structured query language type of the target structured query language template method is a disabled structured query language type, it is determined that the execution type permission check of the data processing request has not passed, and a response message of operation failure is generated in response; when the target structured query language type of the target structured query language template method is not a disabled structured query language type, subsequent operations are performed.

[0044] On the basis of the above scheme, the execution authority also includes user authority. After the target structured query language type successfully matches the preset disabled structured query language type, it also includes: obtaining the database table information associated with the data processing operation, calling the authority interface of the database table information to determine whether the user ID has the authority to process the database table information; when the user ID has the authority to process the database table information, it is determined that the user authority verification has passed; when the user ID does not have the authority to process the database table information, it is determined that the user authority verification has not passed. Optionally, in order to further ensure the security of the data middle station, after the execution type authority verification of the data processing request passes, the user authority verification needs to be performed to ensure the executability of the data processing request.

[0045] Optionally, when performing permission division, the permissions of the data warehouse can be determined according to the data operations of the data warehouse, and the corresponding names can be determined. Generally, data warehouse operations include data warehouse data addition, data warehouse data deletion, data warehouse data modification, data warehouse data query, data warehouse metadata creation, data warehouse metadata deletion, and data warehouse metadata modification. Based on this, seven permissions of the data warehouse of the data center can be constructed: INSERT, DELETE, UPDATE, SELECT, ALTER, CREATE, and DROP. The above seven permissions represent data warehouse data addition permissions, data warehouse data deletion permissions, data warehouse data modification permissions, data warehouse data query permissions, data warehouse metadata creation permissions, data warehouse metadata deletion permissions, and data warehouse metadata modification permissions. The seven permissions correspond to different SQL sentences, and the specific user's permission information can be stored in the MetaStore metadata. Among them, the user's permission information can be stored in different tables according to the permission type. Exemplarily, the user's permission information can be divided into library permissions, table permissions, non-partition column permissions, and partition column permissions according to the permission type. Correspondingly, the database table information includes at least one of library information, table information, non-partition column information, and partition column information. After determining the target SQL template method, the BASE-API module can use the library, table, and column information obtained after parsing the target SQL template method as database table information, and call the permission interface related to the database table information for further user permission verification. After the user permission verification passes, it can be determined that the execution permission verification of the data processing request has passed, and the target SQL template method is passed to the underlying HiveServer service or Spark-SQL service for execution to obtain the execution result. If the user permission verification fails, a processing failure message can be generated as a response message.

[0046] Embodiment 2

[0047] Figure 2 : is a flowchart of a data processing method provided in the second embodiment of the present invention. The data processing method provided in this embodiment can be executed by the middle layer in the data middle platform provided in the above embodiment. The middle layer can be implemented in software and / or hardware. For example, the middle layer can be configured in a computer device. Figure 2 As shown, the method includes:

[0048] S210: Receive a data processing request sent by a client, wherein the data processing request is triggered by the client after detecting a click operation on a control.

[0049] Optionally, after the user performs relevant operations on the client, the user clicks the control to trigger the generation of a data processing request and sends it to the middle layer, and the middle layer receives the data processing request sent by the client.

[0050] S220: Acquire a control identifier carried in the data processing request, and determine a target structured query language template method associated with the control identifier.

[0051] The data processing request is parsed to obtain a control identifier, and a target SQL template method associated with the control identifier is determined. The specific implementation method can be found in the above embodiment, which will not be described in detail here.

[0052] S230: Call the target structured query language template method to execute the data processing request, and generate a response signal according to the execution result to respond.

[0053] Based on the determined target structured query language, the relevant operations of the data processing request are executed. The specific implementation method can refer to the above embodiment.

[0054] In one embodiment, the target structured query language template method is called to execute the data processing request, and a response signal is generated according to the execution result to respond, including: obtaining the processing parameters carried in the data processing request, and using the processing parameters as the execution parameters of the target structured query language template method; executing the target structured query language template method based on the execution parameters, and obtaining the permission verification parameters associated with the data processing request; verifying the execution permission of the data processing request according to the permission verification parameters, and obtaining the processing result of the data processing request after the permission verification passes, and responding with the processing result as the response signal. Referring to the above embodiment, in order to unify the permission verification, the AbstractSqlParser class in the SPARK module is introduced into the BASE-API module as a parsing tool for SPARK-SQL, and the ParseDriver class in the HIVE module is introduced into the BASE-API module as a parsing tool for HIVE-SQL, so as to realize the SQL parsing in the middle layer and then control the permission. The specific implementation method can be referred to the above embodiment.

[0055] The embodiment of the present invention receives a data processing request sent by a client, wherein the data processing request is triggered by the client after detecting a click operation on a control; obtains a control identifier carried in the data processing request, and determines a target structured query language template method associated with the control identifier; calls the target structured query language template method to execute the data processing request, and generates a response signal in response to the execution result. By replacing the application programming interface with the corresponding structured query language template method, the data processing instructions of the middle layer can be applied to data warehouses of different versions, thereby realizing the use of the same middle layer to adapt to data warehouses of different versions, reducing the difficulty of operation and maintenance of the data middle platform, and moving the permission verification forward to the middle layer, realizing the unification of permission verification, and improving the security of the data middle platform.

[0056] Embodiment 3

[0057] Figure 3 Schematic diagram of the structure of a data processing device provided in the third embodiment of the present invention. The data processing device can be implemented in software and / or hardware. For example, the data processing device can be configured in the middle layer and implemented in the form of a computer device. Figure 3 As shown, the device includes a request receiving module 310, a target template method module 320 and a request response module 330, wherein:

[0058] The request receiving module 310 is used to receive a data processing request sent by a client, wherein the data processing request is triggered by the client after detecting a click operation on a control;

[0059] The target template method module 320 is used to obtain the control identifier carried in the data processing request and determine the target structured query language template method associated with the control identifier;

[0060] The request response module 330 is used to call the target structured query language template method to execute the data processing request, and generate a response signal according to the execution result to respond.

[0061] The embodiment of the present invention receives a data processing request sent by a client through a request receiving module, wherein the data processing request is triggered by the client after detecting a click operation on a control; a target template method module obtains a control identifier carried in the data processing request, and determines a target structured query language template method associated with the control identifier; a request response module calls the target structured query language template method to execute the data processing request, and generates a response signal according to the execution result to respond, and by replacing the application programming interface with the corresponding structured query language template method, the data processing instructions of the middle layer can be applied to different versions of data warehouses, thereby realizing the use of the same middle layer to adapt to different versions of data warehouses, and reducing the difficulty of operation and maintenance of the data middle platform.

[0062] Optionally, based on the above solution, the request response module 330 is specifically used for:

[0063] Obtaining processing parameters carried in the data processing request, and using the processing parameters as execution parameters of the target structured query language template method;

[0064] Execute the target structured query language template method based on the execution parameters to obtain the permission verification parameters associated with the data processing request;

[0065] The execution authority of the data processing request is verified according to the authority verification parameters, and the processing result of the data processing request is obtained after the authority verification passes, and the processing result is responded as a response signal.

[0066] Optionally, based on the above solution, the execution permission includes the execution type permission, the permission verification parameter includes the target structured query language type of the target structured query language template method, and the request response module 330 is specifically used to:

[0067] Matching the target structured query language type with the preset forbidden structured query language type;

[0068] When the target structured query language type successfully matches the preset disabled structured query language type, it is determined that the execution type permission check of the data processing request fails;

[0069] When the target structured query language type fails to match the preset disabled structured query language type, it is determined that the execution type permission check of the data processing request passes.

[0070] Optionally, based on the above solution, the execution authority also includes user authority, and the request response module 330 is specifically used to:

[0071] Obtain the database table information associated with the data processing operation, and call the permission interface of the database table information to determine whether the user ID has the permission to process the database table information;

[0072] When the user ID has the authority to process the database table information, it is determined that the user authority check has passed;

[0073] When the user ID does not have the authority to process the database table information, it is determined that the user authority check has failed.

[0074] Optionally, based on the above solution, the database table information includes at least one of library information, table information, non-partition column information and partition column information.

[0075] The data processing device provided in the embodiment of the present invention can execute the data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0076] Embodiment 4

[0077] Figure 4 It is a structural diagram of a computer device provided in Embodiment 4 of the present invention. Figure 4 A block diagram of an exemplary computer device 412 suitable for use in implementing embodiments of the present invention is shown. Figure 4 The computer device 412 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0078] like Figure 4 As shown, the computer device 412 is in the form of a general-purpose computing device. The components of the computer device 412 may include, but are not limited to: one or more processors 416, a system memory 428, and a bus 418 that connects different system components (including the system memory 428 and the processor 416).

[0079] Bus 418 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a local bus to processor 416, or a bus structure using any of a variety of bus architectures. These architectures include, by way of example, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0080] The computer device 412 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 412, including volatile and non-volatile media, removable and non-removable media.

[0081] System memory 428 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 430 and / or cache memory 432. Computer device 412 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage device 434 may be used to read and write non-removable, non-volatile magnetic media ( Figure 4 not shown, usually called a "hard drive"). Although Figure 4Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 418 via one or more data medium interfaces. The memory 428 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0082] A program / utility 440 having a set (at least one) of program modules 442 may be stored, for example, in memory 428, such program modules 442 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 442 generally perform the functions and / or methods of the embodiments described herein.

[0083] The computer device 412 may also communicate with one or more external devices 414 (e.g., keyboard, pointing device, display 424, etc.), may also communicate with one or more devices that enable a user to interact with the computer device 412, and / or communicate with any device that enables the computer device 412 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed via input / output (I / O) interface 422. Furthermore, the computer device 412 may also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) via network adapter 420. As shown, the network adapter 420 communicates with other modules of the computer device 412 via bus 418. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 412, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0084] The processor 416 executes various functional applications and data processing by running the programs stored in the system memory 428, for example, implementing the data processing method provided in the embodiment of the present invention, which includes:

[0085] Receiving a data processing request sent by a client, wherein the data processing request is triggered by the client after detecting a click operation on a control;

[0086] Obtaining a control identifier carried in a data processing request, and determining a target structured query language template method associated with the control identifier;

[0087] The target structured query language template method is called to execute the data processing request, and a response signal is generated according to the execution result to respond.

[0088] Of course, those skilled in the art can understand that the processor can also implement the technical solution of the data processing method provided by any embodiment of the present invention.

[0089] Embodiment 5

[0090] Embodiment 5 of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the data processing method provided in the embodiment of the present invention is implemented. The method includes:

[0091] Receiving a data processing request sent by a client, wherein the data processing request is triggered by the client after detecting a click operation on a control;

[0092] Obtaining a control identifier carried in a data processing request, and determining a target structured query language template method associated with the control identifier;

[0093] The target structured query language template method is called to execute the data processing request, and a response signal is generated according to the execution result to respond.

[0094] Of course, the computer-readable storage medium provided by the embodiment of the present invention stores a computer program which is not limited to the above method operations, but can also execute related operations of the data processing method provided by any embodiment of the present invention.

[0095] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0096] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0097] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0098] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0099] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data middle platform, characterized in that: The method comprises a plurality of data warehouses and an intermediate layer shared by the plurality of data warehouses, wherein: The middle layer is used to receive a data processing request sent by a client, determine a target structured query language template method associated with a control identifier carried in the data processing request, call the target structured query language template method to execute the data processing request, and generate a response signal for response according to the execution result; the middle layer is provided with a structured query language template method corresponding to an application programming interface method, and the target structured query language template method is a structured query language template method associated with the control identifier carried in the data processing request; The step of calling the target structured query language template method to execute the data processing request and generating a response signal according to the execution result to respond includes: Acquire processing parameters carried in the data processing request, and use the processing parameters as execution parameters of the target structured query language template method; the processing parameters carried in the data processing request include an operation type and an operation parameter; Executing the target structured query language template method based on the execution parameter to obtain the permission verification parameter associated with the data processing request; Verify the execution authority of the data processing request according to the authority verification parameter, and call the target data warehouse to obtain the processing result of the data processing request after the authority verification passes, and respond with the processing result as a response signal; The execution permission includes an execution type permission, the permission verification parameter includes a target structured query language type of the target structured query language template method, and verifying the execution permission of the data processing request according to the permission verification parameter includes: Matching the target structured query language type with a preset prohibited structured query language type; When the target structured query language type successfully matches the preset forbidden structured query language type, determining that the execution type permission check of the data processing request fails; When the target structured query language type fails to match the preset forbidden structured query language type, it is determined that the execution type permission check of the data processing request passes.

2. The method according to claim 1, characterized in that The execution permission also includes user permission, and after the target structured query language type successfully matches the preset prohibited structured query language type, it also includes: Obtaining database table information associated with the data processing operation, and calling the permission interface of the database table information to determine whether the user identifier has the permission to process the database table information; When the user identifier has the processing authority of the database table information, it is determined that the user authority check has passed; When the user identifier does not have the authority to process the database table information, it is determined that the user authority check has failed.

3. The method according to claim 2, characterized in that The database table information includes at least one of library information, table information, non-partition column information and partition column information.

4. A data processing method, executed by the middle layer in the data center according to any one of claims 1 to 3, characterized in that: include: Receiving a data processing request sent by a client, wherein the data processing request is triggered by the client after detecting a click operation on a control; Acquire the control identifier carried in the data processing request, and determine the target structured query language template method associated with the control identifier; the intermediate layer is provided with a structured query language template method corresponding to the application programming interface method, and the target structured query language template method is the structured query language template method associated with the control identifier carried in the data processing request; Calling the target structured query language template method to execute the data processing request, and generating a response signal for response according to the execution result; The step of calling the target structured query language template method to execute the data processing request and generating a response signal according to the execution result to respond includes: Acquire processing parameters carried in the data processing request, and use the processing parameters as execution parameters of the target structured query language template method; the processing parameters carried in the data processing request include an operation type and an operation parameter; Executing the target structured query language template method based on the execution parameter to obtain the permission verification parameter associated with the data processing request; Verifying the execution authority of the data processing request according to the authority verification parameter, and obtaining the processing result of the data processing request after the authority verification passes, and responding with the processing result as a response signal; The execution permission includes an execution type permission, the permission verification parameter includes a target structured query language type of the target structured query language template method, and verifying the execution permission of the data processing request according to the permission verification parameter includes: Matching the target structured query language type with a preset prohibited structured query language type; When the target structured query language type successfully matches the preset forbidden structured query language type, determining that the execution type permission check of the data processing request fails; When the target structured query language type fails to match the preset forbidden structured query language type, it is determined that the execution type permission check of the data processing request passes.

5. A data processing device, characterized in that: The middle layer configured in the data middle platform according to any one of claims 1 to 3 includes: A request receiving module, used to receive a data processing request sent by a client, wherein the data processing request is triggered by the client after detecting a click operation on a control; A target template method module is used to obtain the control identifier carried in the data processing request and determine the target structured query language template method associated with the control identifier; the middle layer is provided with a structured query language template method corresponding to the application programming interface method, and the target structured query language template method is the structured query language template method associated with the control identifier carried in the data processing request; A request response module, used to call the target structured query language template method to execute the data processing request, and generate a response signal for response according to the execution result; Wherein, the request response module is specifically used for: Acquire processing parameters carried in the data processing request, and use the processing parameters as execution parameters of the target structured query language template method; the processing parameters carried in the data processing request include an operation type and an operation parameter; Executing the target structured query language template method based on the execution parameter to obtain the permission verification parameter associated with the data processing request; Verifying the execution authority of the data processing request according to the authority verification parameter, and obtaining the processing result of the data processing request after the authority verification passes, and responding with the processing result as a response signal; The execution permission includes an execution type permission, and the permission verification parameter includes a target structured query language type of the target structured query language template method: The request response module is specifically used for: Matching the target structured query language type with a preset prohibited structured query language type; When the target structured query language type successfully matches the preset forbidden structured query language type, determining that the execution type permission check of the data processing request fails; When the target structured query language type fails to match the preset forbidden structured query language type, it is determined that the execution type permission check of the data processing request passes.

6. A computer device, characterized in that: The device comprises: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method as claimed in claim 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data processing method as claimed in claim 4 is realized.

Images